Azure Blob Storage
The Azure Blob Storage connector lists, reads and writes blobs in one container of an Azure storage account. It signs in with a SAS token or a service principal.
When to use it
- You read or write files in Azure Blob Storage.
- Another system lands its exports in a container and you collect them.
- You deliver generated CSV, JSON or XLSX files to a container.
What you need
- The storage account name and the container name.
- One of these:
- A SAS token with read, write, delete, list and create permissions on the container.
- A service principal: an app registration in Microsoft Entra ID with a client secret, and the Storage Blob Data Contributor role on the container or the storage account.
Provider-side setup for this connector, including the app registration, the redirect URI and the permissions, is in the File hub setup guide. Connection basics and environments are on the Connections page.
Connect
The connection form lists its steps on the left. Open any step from that list. The Environments step holds the values that differ per environment.
Test connection is in the form footer. Beside it, Test against picks Base, the connection as saved, or an environment, whose values are applied first.
| Field | Step | What to enter |
|---|---|---|
| Auth Method | Authentication | SAS Token or Service Principal. The step shows only that method's fields. |
| SAS Token | Authentication | For SAS Token. |
| Tenant ID | Authentication | For Service Principal. The directory (tenant) ID. |
| Client ID | Authentication | For Service Principal. The application (client) ID. |
| Client Secret | Authentication | For Service Principal. |
| Storage Account | Connection | The storage account name. Required. |
| Container | Connection | The container name. Required. |
| Default Path Prefix | Connection | Optional. A folder prefix that paths resolve under. |
| Notes | Options | Free text for your team. |
Supported operations
| Node | Operations |
|---|---|
| Source | List Files, Get File |
| Load | Upload File(s), Write Rows To File(s), Move/Rename, Delete, Create Folder |
Reading files. Recursive and Files only shape a listing.
Include (glob), Exclude (glob) and Regex (advanced) choose files by
name. Parse content into rows turns each CSV file into rows, with CSV has
header row and CSV Delimiter settings. Format Auto (by extension)
reads a .csv file as CSV.
Writing rows to files. Under Format & Behavior:
- Format is CSV, JSON or XLSX. Compress (.gz) compresses the file.
- File Name Template accepts the tokens
{runId},{YYYYMMDD},{HHmmss},{chunkIndex},{partIndex},{groupKey}and{field:MyField}. Append timestamp adds{YYYYMMDD}_{HHmmss}_{SSS}when the template has none. - Split is Single file, One per row or Group by. Group by takes one or more field names, separated by commas.
- Overwrite decides what happens when the file exists: Overwrite, Skip or Fail.
Create Folder writes an empty .datalug-folder blob in the new folder.
Listings hide that blob.
Choosing a path
The remote path field in Configure Action has a Browse button that lists the folder.
- Open enters a folder, and Up goes back. Use picks a file, and Use this folder picks the folder you are in.
- An operation that names a folder shows folders only.
- The field stays editable. Type a wildcard such as
/inbound/*.csv. - Browsing reads only. Nothing is written.
Limits
- One connection reaches one container. Use a second connection for a second container.
Troubleshooting
- Test connection passes and a Source lists nothing. Check the path. A folder that does not exist lists as empty, not as an error. Pick the folder with Browse.
- A file is written but does not appear where you expect. Check the target path on the Load node. A path that resolves to another folder still succeeds.
- Large files. Staging space is limited per run, and the run log names any node that spilled to disk. See Files and staging.
- AuthorizationPermissionMismatch with a service principal. The principal has no data role on the container. Assign Storage Blob Data Contributor, then wait a few minutes for the assignment to apply.
- AuthenticationFailed with a SAS token. The token has expired or lacks a permission the operation needs. Generate a new token with the permissions listed above.
Related
- Source node and Load node.
- Amazon S3 and SharePoint Files.