Skip to main content

Source node

Source

The Source node reads rows from an external system and emits a stream downstream. Every workflow that pulls data from anywhere starts with at least one Source. Under the hood the runtime chunks reads so large result sets stream through the pipeline without buffering the whole set in memory.

When to use​

  • The workflow needs to pull data from a real system (Postgres, SQL Server, MySQL, Oracle, Snowflake, Salesforce, S3, Google Drive, SharePoint, Dropbox, SFTP, Generic API, Google Sheets).
  • You want to reuse the same query across workflows — bind the Source to a Saved Action so a change to the query updates every flow.
  • The result set is larger than memory — the chunked pump keeps only one chunk resident at a time and gates on maxRows.

Ports​

PortDirectionKindNotes
inindataOnly fires when Start feeds the Source directly (chained multi-source) or a file-transfer chained-download shape upstream.
outoutdataRows read from the system, one chunk at a time.
erroroutcontrolFires on connection / auth / query failures.
notifyoutcontrolNotification policy port.

Configuration​

  • Connection. Pick a connection from the account's connection list. The connector family (SQL Server, PostgreSQL, Salesforce, S3, Generic API, etc.) determines which Action shapes are available.
  • Action. A bound reference to a Saved Action, or an in-node action config. Saved Actions are portable — the same action powers Source and Load nodes across workflows.
  • Action operation. select (SQL / Salesforce) or a connector-specific read op (Download, List objects, GET, etc.).
  • Object name / SOQL / SQL / path. The specific target: table, view, procedure name, SOQL query, remote path, or URL suffix.
  • maxRows. Optional hard cap on the number of rows this Source emits. Combined with a manifest-level cap for run-wide limits.
  • Chunk size. Optional override; defaults to the runtime's memory-adaptive chunk sizer.

Configure Action walkthrough​

The Configure modal opens directly on the action. Order:

  1. System family — pick the connector (SQL Server / Salesforce / File Transfer / etc.). Filters the action list.
  2. Connection — pick the connection. Presence of the connection unlocks Test.
  3. Action — choose a Saved Action for the flow, or author an ad-hoc one. Saved Actions include a Preview button that runs against the live connection with the current row shape.
  4. Advanced — chunk size, max rows, retry policy, and pre/post hook actions.

Runtime behaviour​

  • The runtime opens a per-chunk read cursor. Each returned chunk fires the downstream subgraph once; the pump breaks when the returned chunk is smaller than the requested size (end-of-stream) or when totalRows >= maxRows.
  • SQL Server / PostgreSQL / MySQL / Oracle / Snowflake reads route through the same connector adapters that DbScript and Load nodes share.
  • Salesforce SOQL is bulk-API friendly; the runtime opts into the bulk path when the query size crosses DLR_SF_BULK_SPLIT_JOBS threshold.
  • File-transfer Downloads emit one row per file with contentRef set to a local staged path so downstream nodes can read the bytes without re-fetching.

Failure modes​

  • Connection missing. Connection not found for source node "X". Wire the connection in the Configure modal.
  • Connection unreachable. The connector's connect() throws with the underlying diagnostic (DNS, TLS, auth). Fix the connection config on the Connections page.
  • Query error. The DB error is surfaced verbatim on the error port so an operator can act.
  • maxRows exceeded. The pump stops mid-stream when the cap is reached. Downstream sees exactly maxRows rows.

Transform

Reshape rows before they land

Filter

Drop rows early to save downstream cost

Load

Write the rows to a target system