Data Profile node
What it does
The Data Profile node summarises the rows that reach it, column by column: counts, empty values, distinct values, ranges and the most frequent values. Rows pass through unchanged.
In a run, the profile is recorded in the run's Log Events. In the builder, the node's panel shows a Field Analysis table when you run the profile there.
When to use
- You want a data-quality snapshot of each run: row counts, empty values, value spread.
- You want to check data before an expensive Load that is hard to undo.
- You are exploring a new source and want to see what each column holds.
Configure
Select the node to edit it in the panel.
- Rows to Profile: the number of rows to sample. Choose 100, 250, 500, 1000, 2500, 5000 or 10000, or Custom... to enter a number. The default is 500.
- Input Node: the node whose rows are profiled.
Every column is profiled. There is nothing else to set.
What is reported for each column
| Measure | Applies to |
|---|---|
| Inferred type and a suggested field type | Every column |
| Row, null, blank, populated and distinct counts | Every column |
| Minimum, maximum and average | Numbers |
| Earliest and latest | Dates |
| Shortest and longest length | Text |
| The 50 most frequent values | Every column |
Ports
| Port | Direction | Kind | Notes |
|---|---|---|---|
in | in | data | Rows to profile. |
out | out | data | The input rows, unchanged. |
error | out | control | Receives the input rows when the node fails. |
A source read in chunks is profiled chunk by chunk: each chunk that reaches the node gives its own profile.
Examples
Check a new source. Start > Source > Data Profile, Rows to Profile 1000. Run it in the builder and read the Field Analysis table.
Watch quality on every run. Source > Data Profile > Transform > Load. Open the run's Log Events to compare null counts between runs.
Troubleshooting
| Symptom | Cause | What to do |
|---|---|---|
| A code column is reported as a number | Values such as years or zero-padded codes look numeric. | Read the suggested type with care for such columns. |
| The most frequent values tell you nothing | The column holds unique ids. | No action needed. The list is capped at 50 values. |
| The profile covers fewer rows than the source | Rows to Profile limits the sample, and chunks are profiled one at a time. | Raise Rows to Profile. |
| The panel shows no Field Analysis after a run | The table is filled by running the profile in the builder. | Run it in the builder, or read the run's Log Events. |