Skip to main content

Compute node

Transform

The Compute node adds derived columns to each row (derive mode) or aggregates rows into groups (group_by mode). It's the "spreadsheet formula bar" of the flow.

When to use​

  • You need a derived column: full_name = CONCAT(first, ' ', last), is_active = EQ(status, 'active'), age_bucket = IF(GT(age, 65), 'senior', IF(GT(age, 40), 'adult', 'young')).
  • You need aggregates over groups: total, count, min, max, avg per group.
  • You need window-style behaviours across a source stream (rolling total, cumulative count).

Ports​

PortDirectionKindNotes
inindataRows.
outoutdataRows with derived / aggregated columns.
erroroutcontrolFires on expression eval errors when strict mode is on.
notifyoutcontrolNotification policy port.

Configuration​

  • Mode.
    • derive — per-row derived columns via expressions.
    • group_by — aggregate into groups.
  • Derived columns (derive). List of { name, expr }. Each expression runs once per row via evalTransformExpr.
  • Group-by keys (group_by). SchemaFieldMultiSelect over upstream columns.
  • Aggregates (group_by). List of { op, field, as }. Supported ops: count, sum, avg, min, max, first, last, count_distinct.
  • Group-by scope.
    • global (default) — aggregate across every chunk from every source.
    • chunk — aggregate within each chunk.

Configure Action walkthrough​

  1. Pick a mode (derive or group_by).
  2. Derive mode: add { name, expr } rows; the expression editor uses the same MappingExpressionModal as Transform.
  3. Group-by mode: pick the group-by keys with the SchemaFieldMultiSelect chips; add aggregate rows.

Runtime behaviour​

  • Derive runs per row, streaming.
  • Group-by with global scope buffers reducer state across chunks and emits ONCE per source stream at the finalize pass (after every source has drained).
  • Group-by with chunk scope emits per-chunk.
  • Group-by state is per-node-per-source; two Sources feeding the same Compute node produce two separate group-by outputs (finalize pass runs per source).

Failure modes​

  • Expression error on a row. Derived column emits null for that row; the run continues.
  • Empty group. Emitted with the group key values and null / 0 for the aggregate columns (per op).

Transform

Per-field mapping + expressions on a projection

Reshape

Pivot / unpivot the aggregate output

SQL

Same group-by shape in a SQL statement