Skip to main content

Core concepts

Foundations

Workflows in DataLug are graphs of nodes that move and transform data between systems. This page defines the core terms so the rest of the docs makes sense.

Glossary​

  • Account is a company or tenant. Most data is scoped to an account.
  • User is a person within an account.
  • Role is the authorization level for a user (Viewer, Builder, Auditor, Admin, Owner). Roles determine what pages and actions the user can access within their account.
  • Project is a workspace containing workflows, pipelines, and saved artifacts.
  • Workflow is an execution graph built on the design canvas.
  • Pipeline is a higher-level container that chains workflows together.
  • Connection is connector configuration to an external system (Salesforce, SQL Server, S3, etc.).
  • Saved Action is a saved query, operation, procedure, or script definition attached to a connection.
  • Environment is a named deployment context (for example, DEV, Docker, Prod).
  • Runtime is the execution service or agent that runs workflows for an environment.
  • Deployed Job is a deployment definition (project + workflow or pipeline + environment + schedule).
  • Runtime Execution is a single run instance with status, logs, and per-node metrics.

How they fit together​

A typical mental model:

  1. An Account owns Projects
    Each project is a workspace. You can have many.
  2. A Project has Workflows, Connections, and Saved Actions
    Workflows are the visual flow graphs. Connections describe how to reach external systems. Saved Actions are reusable queries/operations against a connection.
  3. An Environment ties a runtime to a named context
    DEV, Docker, Prod, etc. Connections have per-environment settings so you don't run prod queries against your test database.
  4. A Deployed Job ships a workflow to an environment on a schedule
    When the schedule fires (or you click Run Now), the runtime executes the flow and emits a Runtime Execution record.
  5. Runtime Executions are the audit trail
    Logs, row counts, payload samples, per-node duration. Searchable, filterable, exportable.

Two execution modes​

  • Design-time runs (Quick Debug Run / Preview) happen on the canvas while you're building. Capped at 10 rows for safety. Good for "does my Filter rule work?" checks.
  • Deployed runs happen via Jobs (scheduled or manual). They use the environment's runtime, full row volume, and write Runtime Execution records.

Library projects​

A Library Project is a curated project the platform ships with sample data and shared connections. Customer admins can clone library projects into their own account to start from a known-working baseline. The library DB-family labs build then populate 100 sample records across SQL Server, PostgreSQL, MySQL/MariaDB, Oracle, and Snowflake.

Naming conventions​

  • Connection names use a per-system prefix, e.g. (Snowflake) - Prod, (Salesforce) - Sandbox. The wizard shows the prefix automatically; you supply the suffix.
  • Environment names are free-form but typically DEV, QA, Prod, Docker.
  • Project and workflow names should describe the integration, not the system, e.g. CRM Lead Sync not Salesforce -> Snowflake.