Profile node
The Profile node consumes the upstream stream and emits a data profile: row counts, per-column null rates, distinct value counts, min / max / avg for numeric columns, and top-N values for categorical columns. It's the observe-what-flowed-through node.
When to use
- You want a data-quality snapshot per run — how many rows, how many nulls per column, distribution of values.
- You want to drive an alert on data drift (fewer rows than usual, more nulls on a specific column).
- You want a sanity-check step before a Load that's expensive to undo.
Ports
| Port | Direction | Kind | Notes |
|---|---|---|---|
in | in | data | Rows to profile. |
out | out | data | One row per profiled column with the stats. |
error | out | control | Fires on validation errors. |
notify | out | control | Notification policy port. |
Configuration
- Row presets. Quick-pick from
all,first 100,first 1000, orcustom. The custom option lets you cap the sample size explicitly. - Custom rows. When preset is
custom, the row cap. - Numeric ops. Toggle per numeric summary: min, max, avg, stddev.
- Top-N. For categorical columns, how many top values to include in the profile row.
Runtime behaviour
- Buffers the sample (up to the row cap) across chunks and emits at the finalize pass.
- Per-column stats are computed in one pass; the sample memory footprint is bounded by the row cap.
Failure modes
- Very high-cardinality columns. Top-N truncates; if the column is UUID-like, the top-N is meaningless.