Your first data pipeline

Tutorial

This walkthrough builds an end-to-end data pipeline: read rows from PostgreSQL, transform them, and load them into Snowflake. Most production integrations have the same shape.

Prerequisites

Finish the Quickstart first. You also need a Snowflake account. Without one, a Library Project lets you run the same flow against sample data.

Step 1: Create the project

On the Projects page, select New Project, choose Empty, and name the project CRM Lead Sync. Under First data pipeline, enter a name or leave it to default to the project name.

Step 2: Register the source connection (PostgreSQL)

1. 1
   Connections page

   Select + New Connection and pick PostgreSQL.
2. 2
   Fill in the fields

   Host, port (5432), database, user and password.
3. 3
   Add an environment setting

   Add an Environment Setting so DEV, QA and Production can point at different databases without changing the connection.
4. 4
   Test Connection

   Checks the host, the credentials and that data can be read.
5. 5
   Save

   The connection appears in the list and is available to every project.

Step 3: Register the target connection (Snowflake)

Use the same wizard and pick Snowflake. Snowflake has two authentication modes:

• Password: username, password, account, warehouse, database, schema and role.
• Keypair: username, account, warehouse, database, schema, role and private key, with an optional passphrase. Recommended for service accounts.

Select Test Connection, then Save.

Step 4: Wire the flow

On the canvas:

1. 1
   Source

   Pick the PostgreSQL connection. Choose a table, or write a SELECT in script mode.
2. 2
   Transform

   Open the Map Fields modal. Drag source columns to target columns. Use expressions for derived fields, lookups or coalesce.
3. 3
   Load

   Pick the Snowflake connection and the destination table. Choose a mode: Append, Replace or Merge by key.
4. 4
   Wire them

   Drag from each node's output port to the next node's input port: Source, then Transform, then Load.

Step 5: Quick Debug Run

Select Quick Debug Run (10 rows) on the canvas. The runtime reads 10 rows from PostgreSQL, passes them through the Transform, and previews the Load without writing. The Runs tab in the right panel shows per-node row counts and payload samples. Adjust the mapping until the Load preview is right.

Step 6: Deploy as a Job

Open Jobs and select Add Job. Pick the project, the data pipeline and the environment. Under Scheduling, choose a mode such as Hourly window or Daily, then save. Runtime Executions shows the next scheduled run.

Step 7: Watch it run

Open Monitor > Runtime Executions. The latest run shows status, duration, per-node row counts and logs. Expand a run for per-node detail; the payload sample shows the rows that moved.

Where to go next

• Source node and the Configure Action reference for source, target, script and notification settings.
• Transform node for mapping with expressions, lookups and type conversion.
• Load node for write modes and pre- and post-run database hooks.
• Runtime Executions for troubleshooting failed runs.