Reading and writing files
This page covers how a file data pipeline reads a file, resumes a read, and writes an export. File connectors include SFTP, S3, Azure Blob, Google Drive, OneDrive, SharePoint Files and Dropbox.
Reading a file
CSV is parsed as it is read, not loaded whole.
- Quoted fields with commas and line breaks inside them are handled. A newline inside quotes does not end the row.
""inside a quoted field is one literal quote.- A large file is read in chunks. Rows are passed downstream as the file is read.
| Setting | What it does |
|---|---|
| Has header | Use the first row as column names. |
| Columns | Name the columns yourself when there is no header row. |
| Delimiter | The character between fields. Comma by default. |
Resuming a file read
A file read resumes where it stopped, including part way through a single file. A file already downloaded in the run is not downloaded again, so a run that failed at file 400 of 500 does not fetch the first 399 again.
Writing a file
| Format | How it is written |
|---|---|
| CSV, JSON lines | Written a row at a time where the destination supports it. |
| XLSX, Parquet | Built whole, then uploaded. |
| Any format with compression on | Built whole, then compressed with gzip. |
Compression adds .gz to the file name.
Large file runs
- The run viewer's Written to disk tile shows whether a run set data aside while it ran. See Runtime Executions.
- For a run over many files, narrow the file filter or read fewer files per run.
- Group By, Join and Pivot over a large input also make a run slower; filter rows before them where you can.
See also Run performance.