Sub-workflow node
What it does
The Sub-workflow node runs another data pipeline from the same project as one step of this one. The rows from the child's end node come back on out. Build a sequence once, such as address clean-up or a compliance check, and call it from any pipeline in the project.
The child runs inside the parent run, with the same environment and the same per-run record allowance. It has its own run id, derived from the parent's.
When to use
- Several pipelines share the same sequence of nodes.
- A long canvas is easier to review with a block moved into its own pipeline.
- A step must run with the parent's environment and limits.
Configure
Select the node and press Configure in the side panel to open Configure Sub-workflow. Save keeps the dialog open, Done saves and closes it, and Cancel discards the changes since the last save. A message confirms each save.
- Child workflow: any other data pipeline in this project. The pipeline the node sits on is not listed.
- Terminal node: the id of the child node whose rows are returned, such as
n_final_load. Leave it blank when the child has one end node (a node with no outgoing edge). Set it when the child branches. - Input binding:
Pass parent rows(default): the parent's rows reach the child as the run variable__sub_input_rows. The child's Sources still read their own data, so only a child built to read that variable uses them.Ignore parent rows: the child runs from its own Sources only.
- Recursion guard: Max depth, default 3, allows parent, child and grandchild. A deeper chain fails.
Child variables
Name and value pairs passed to the child. The child reads each as {{var.exec.NAME}}, so it also runs on its own with a default.
- A value is literal text, or a reference to the parent's values:
{{var.exec.x}},{{var.project.x}},{{var.global.x}}or{{sys.project.id}}. It is read once, when the node starts. - Row values such as
{{row.field}}are not available, because a variable holds one value for the run. - A name starts with a letter or
_and holds letters, digits and_. A row with no name is dropped.
On failure is under Execution Policy in the Advanced section of the node panel. End run is the default. With Continue (Skip Node), a failed child skips the branch below this node, the other branches carry on, and the run reports Completed with errors.
Ports
| Port | Direction | Kind | Notes |
|---|---|---|---|
in | in | data | Parent rows for the child. |
out | out | data | Rows from the child's terminal node. |
error | out | control | Receives the input rows, with dl_error, dl_failed_node_id and dl_failed_node_name, when the child fails or the depth limit is reached. The run still ends Failed. |
How the child run behaves
- A child that fails makes this node fail. The error names the child run id.
- A child set to continue past its own failed node reports Completed with errors, and the parent reports the same.
- A child that pauses for a Human Approval fails this node: a nested run cannot wait for a person.
- The child's node row counts are recorded on the child run.
Examples
Reuse an address clean-up.
The child pipeline holds the clean-up nodes and ends in a Transform. Its node id is the Terminal node. The parent receives only the rows that Transform emits.
Pass a batch id. Child variable batch_id with value {{var.exec.batch_id}}. The child writes {{var.exec.batch_id}} into a file path.
Troubleshooting
| Message or symptom | Cause | What to do |
|---|---|---|
Sub-workflow "X": terminal node "Y" is not present in the child manifest. | The terminal node id is wrong. | Copy the node id from the child's canvas. Nothing in the child ran. |
Sub-workflow "X": the child data pipeline has 2 possible end nodes (a, b). Set the terminal node on this node to pick one. | The child branches and no terminal node is set. | Set the terminal node. |
| The depth limit is reached | Pipelines call each other in a chain deeper than Max depth, or a pipeline calls itself. | Remove the loop, or raise Max depth. |
| The child ignores the parent's rows | The child's Sources read their own data. | Read __sub_input_rows in the child, or pass values as child variables. |
| The run fails at an approval in the child | A nested run cannot wait for a person. | Move the Human Approval node into the parent. |