Amazon S3
The AWS S3 Bucket connector lists, reads and writes objects in one Amazon S3 bucket. It signs in with an access key.
When to use it
- You read or write files in an S3 bucket.
- Another system lands its exports in S3 and you collect them.
- You deliver generated CSV, JSON or XLSX files to a bucket.
What you need
- An IAM access key ID and secret access key.
- The bucket name and its region.
- An IAM policy that lets the key list the bucket and read, write and delete its objects.
Provider-side setup for this connector, including the app registration, the redirect URI and the permissions, is in the File hub setup guide. Connection basics and environments are on the Connections page.
Connect
The connection form lists its steps on the left. Open any step from that list. The Environments step holds the values that differ per environment.
Test connection is in the form footer. Beside it, Test against picks Base, the connection as saved, or an environment, whose values are applied first.
| Field | Step | What to enter |
|---|---|---|
| Access Key ID | Authentication | The IAM access key ID, such as AKIA.... |
| Secret Access Key | Authentication | The matching secret. |
| Region | Connection | The bucket's region, such as us-east-1. Required. |
| Bucket | Connection | The bucket name. Required. |
| Default Prefix | Connection | Optional. A folder prefix that paths resolve under, such as finance/inbound. |
| Notes | Options | Free text for your team. |
Instance roles and temporary credentials are not supported.
Supported operations
| Node | Operations |
|---|---|
| Source | List Files, Get File |
| Load | Upload File(s), Write Rows To File(s), Move/Rename, Delete, Create Folder |
Reading files. Recursive and Files only shape a listing.
Include (glob), Exclude (glob) and Regex (advanced) choose files by
name. Parse content into rows turns each CSV file into rows, with CSV has
header row and CSV Delimiter settings. Format Auto (by extension)
reads a .csv file as CSV.
Writing rows to files. Under Format & Behavior:
- Format is CSV, JSON or XLSX. Compress (.gz) compresses the file.
- File Name Template accepts the tokens
{runId},{YYYYMMDD},{HHmmss},{chunkIndex},{partIndex},{groupKey}and{field:MyField}. Append timestamp adds{YYYYMMDD}_{HHmmss}_{SSS}when the template has none. - Split is Single file, One per row or Group by. Group by takes one or more field names, separated by commas.
- Overwrite decides what happens when the file exists: Overwrite, Skip or Fail.
Choosing a path
The remote path field in Configure Action has a Browse button that lists the folder.
- Open enters a folder, and Up goes back. Use picks a file, and Use this folder picks the folder you are in.
- An operation that names a folder shows folders only.
- The field stays editable. Type a wildcard such as
/inbound/*.csv. - Browsing reads only. Nothing is written.
Limits
- One connection reaches one bucket. Use a second connection for a second bucket.
Troubleshooting
- Test connection passes and a Source lists nothing. Check the path. A folder that does not exist lists as empty, not as an error. Pick the folder with Browse.
- A file is written but does not appear where you expect. Check the target path on the Load node. A path that resolves to another folder still succeeds.
- Large files. Staging space is limited per run, and the run log names any node that spilled to disk. See Files and staging.
- Access Denied on a listing or a write. The IAM policy does not grant that action on the bucket or prefix. Add it to the policy attached to the key's user.
Related
- Source node and Load node.
- Azure Blob Storage and SFTP.