Skip to main content

Core concepts

Foundations

Data Pipelines in DataLug are graphs of nodes that move and transform data between systems. This page defines the terms the rest of the docs use.

Glossary​

  • Account is a company or tenant. Most data is scoped to an account.
  • User is a person within an account.
  • Role is the authorization level for a user (Viewer, Builder, Auditor, Admin, Owner). Roles determine which pages and actions the user can access within their account.
  • Project is a workspace containing data pipelines and saved artifacts.
  • Data Pipeline is an execution graph built on the design canvas.
  • Sub-workflow is a node that calls another data pipeline as a step. Use it to build one data pipeline from others.
  • Connection is connector configuration for an external system (Salesforce, SQL Server, S3 and others).
  • Saved Action is a saved query, operation, procedure or script definition attached to a connection.
  • Environment is a named deployment context, for example DEV, QA or Production.
  • Runtime is the execution service or agent that runs data pipelines for an environment.
  • Deployed Job is a deployment definition: project, data pipeline, environment and schedule.
  • Runtime Execution is a single run with status, logs and per-node metrics.

How they fit together​

  1. An Account owns Projects
    Each project is a workspace. An account can have many.
  2. A Project has Data Pipelines, Connections and Saved Actions
    Data Pipelines are the visual flow graphs. Connections describe how to reach external systems. Saved Actions are reusable queries and operations against a connection.
  3. An Environment ties a runtime to a named context
    For example DEV, QA and Production. Connections have per-environment settings, so a run in DEV reads your test database and a run in Production reads the live one.
  4. A Deployed Job runs a data pipeline in an environment on a schedule
    When the schedule fires, or you run the job by hand, the runtime executes the flow and writes a Runtime Execution record.
  5. Runtime Executions are the audit trail
    Logs, row counts, payload samples and per-node duration. Searchable, filterable and exportable.

Two execution modes​

  • Design-time runs (Quick Debug Run and Preview) happen on the canvas while you build. They are capped at 10 rows. Use them to check a step, such as a Filter rule.
  • Deployed runs happen through Jobs, scheduled or manual. They use the environment's runtime and the full row volume, and they write Runtime Execution records.

Library projects​

A Library Project is a ready-made project with sample data and shared connections. Clone one into your account to start from a working baseline. The database library projects hold 100 sample records in SQL Server, PostgreSQL, MySQL/MariaDB, Oracle and Snowflake.

Finding a component​

The canvas's left sidebar has two tabs: Components, the catalogue of node types you can add, and Explorer, a tree view of the project.

The Components tab finds a node by what you want to do as well as by its name. Press / to jump to the search box. It matches a node's name, its description, its type and a set of synonyms, so "combine two datasets" finds Join. Nodes are grouped into ten colour-coded categories you can filter by, from the pills or the filter dropdown beside them, and you can star the ones you use often. Hovering a node, or tabbing to its +, shows what it does and which inputs and outputs it has before you place it.

Drag a node onto the canvas, or press the + on its row. The + adds a node without a mouse.

Turning a node off​

Select a node and press Skip in its toolbar. The node is skipped on the next run and the edges stay connected, so you can try a flow without one step and rewire nothing. The same button then reads Turn on.

The skip carries downstream only where there is nothing left to run on: a node is skipped when every one of its inputs is off. A node with one input still live runs on that input, so a Join fed by one lane that is off and one lane that is on still runs.

A node that is off is not checked before a run, so an unfinished step does not block the run.

The card shows which case applies. Off is a node you turned off, and the same button turns it back on. Off: no live input is a node that inherited the skip, and it comes back when one of its inputs does.

In the run record a node that is off reports DISABLED. SKIPPED means a node never got the chance to run, for example an unmet dependency or a branch not taken.

Naming conventions​

  • Connection names use a per-system prefix, for example (Snowflake) - Prod or (Salesforce) - Sandbox. The wizard supplies the prefix; you type the rest.
  • Environment names are free-form, for example DEV, QA and Production.
  • Project and data pipeline names describe the integration rather than the systems, for example CRM Lead Sync rather than Salesforce to Snowflake.