Source node
What it does
The Source node reads rows from a connected system and sends them downstream. Most data pipelines start with at least one Source. Rows are read in chunks, so a large table or file moves through the flow without being read into one piece first.
When to use
- The flow reads from a database (PostgreSQL, SQL Server, MySQL, Oracle, Snowflake and others), a SaaS application such as Salesforce, a file store (S3, Google Drive, SharePoint, Dropbox, SFTP) or an HTTP API.
- The same read is used in several flows: save it as a Saved Action and bind each Source to it, so one change updates every flow.
Configure
The node panel
- Header: the connector's logo, the node's kind and its connector.
- Status: says the node is ready to run, or lists what needs fixing.
- Node name: the name shown on the card.
- Connection: the connection, with Edit to open it.
- Action: the action this node runs, with Configure and the saved-action picker.
- Run limits: Max rows per run (empty reads every row) and the cache-write switch.
- Advanced: execution hooks, execution policy and notifications. It starts closed, and its one visible line says whether anything inside is set. See Execution hooks and policies.
Configure Action
For a database connection, Configure Action is a three-step wizard. The step bar names the step you are on and what it still needs.
- Object: the database, the object type (tables, views, procedures, functions) and the object itself. On a connection whose objects have not been read, press Get Metadata to fill the list. Press it again to pick up objects added since. Or choose Write a query to replace the object list with a SQL editor.
- Operation and fields: the operation, the fields and the Action name. The action name is required.
- Review: every value in one list, with an Edit link back to its step. Save attaches the action to the node.
Other connection types (Salesforce, file transfer, Generic API, CRM) have their own steps, with the same step bar and a final Save.
Save Action saves the action and keeps the dialog open. Done saves it and closes the dialog, and stays open if the save fails. Close closes it without saving.
- Operations use the connector's own names. Google Drive, OneDrive, SharePoint, Dropbox, Azure Blob and S3 read with List Files and Get File. SFTP, FTP and FTPS read with List Directory and Download (Get File).
- The first operation the connector offers is selected for you. If you pick an object that does not support it, the first supported operation is selected instead.
- File paths: a path that names a file reads that file. A path that names a folder reads the files in it that match the filters. A path containing
*or?is a pattern under its parent folder. Google Sheets, Docs and Slides are not files: read a Sheet with the Google Sheets connector.
Connecting an edge into a Source that has no saved action opens Configure Action. A configured node stays closed when you rewire it.
Caching
Turn on the cache-write switch to save the rows this Source reads into a named cache that later nodes read with CACHE_VALUE. With Scope Persistent, the cache is kept between runs until its TTL expires.
Ports
| Port | Direction | Kind | Notes |
|---|---|---|---|
in | in | control | The edge from Start, or from a node that runs before this Source. |
out | out | data | Rows read from the system, one chunk at a time. |
error | out | control | Receives a row with the error when the read fails, for example on a connection, sign-in or query error. |
How it reads large data
Database tables are read in chunks, and the read depends on the table's primary key:
- A single-column primary key of an integer or text type: read by seeking on the key. This is the fastest read and stays correct while the table is being written to.
- A composite key, or a key that is a date, decimal or uuid: read with an ordered scan. Correct, and slower on a large table. Oracle tables keyed on an unconstrained NUMBER, and all Snowflake tables, use this read.
- No primary key: read in an order the database does not guarantee, so rows written during a long read can be missed or repeated. Add a primary key, or set Max rows per run so the read is short.
A field list without the key column works: the key is read and removed before rows reach the next node.
CSV files are read in chunks, so a large file makes a run slower, not impossible.
- Max rows per run stops reading at that row.
- A CSV with no header row and no column names set is read whole, because column names come from the widest row. Set column names to read it in chunks.
Examples
Copy a table. Start > Source (SQL Server, table dbo.Orders, operation select) > Transform (map the columns) > Load (PostgreSQL, table orders, insert). A Load is always preceded by a Transform.
Sample a large table. Set Max rows per run to 1000 while you build the flow, then clear it before scheduling the job.
Read every CSV in a folder. Source on an SFTP connection with operation Download (Get File) and path /inbound/*.csv.
Troubleshooting
| Symptom | Cause | What to do |
|---|---|---|
Connection not found for source node "X". | The node has no connection, or the connection was deleted. | Pick a connection in the node panel. |
| A DNS, TLS or sign-in error | The connection settings are wrong or the credentials expired. | Fix the connection on the Connections page and test it. |
| A database error | The query or object is invalid. | Correct the query in Configure Action. The message is the database's own. |
| Fewer rows than expected | Max rows per run is set. | Clear it. |
| The object list is empty | The connection's objects have not been read. | Press Get Metadata on step 1. |