Run performance
Most runs need no tuning. This guide covers what to check when a run is slow, the settings you control on a connection, and the workloads that behave differently.
Start with the run viewer
Open the run in Runtime Executions and expand What this run cost. These tiles answer most questions:
- Rows per statement. How many rows each database write carried. A badge reading "One per row" means a target makes one round trip per row, which is slow whatever else you change.
- Waiting to start. Time source chunks spent queued. When this is high, the run is competing with other runs on the same runtime. Move a schedule so the runs do not overlap.
- Rate limit wait. Time requests waited for a connection's pace. Shown only when a request waited. When this is high, other runs share the connection's pace, or the pace is lower than your plan allows.
- Slowest measured node. The node with the lowest rows per second. Start there.
See Runtime Executions for every tile.
When a provider rate limits you
Set the pace on the connection. Every connector retries a 429 response; pacing is what stops you causing one.
Open the connection and expand API limits.
| Setting | What it does |
|---|---|
| Requests per period | The rate your provider plan allows. Set it together with Period. |
| Period | The time window for that rate. |
| Page size | Rows per request, up to the provider's maximum. |
| Max attempts | Attempts per request, including the first try. |
| Cool-off (ms) | The first wait after a 429. |
| Requests at once | How many requests the connection may have open at the same time within one run. Blank for no limit. |
Requests per period and Period work only as a pair. Leave a field blank to use the default shown under it. Two connections never slow each other down.
One pace for every run on a connection
The pace is shared by every run that uses the connection, including runs on your Private Agents. Two runs at the same time split the pace between them; they do not each get the full pace.
When the connection sets no pace, DataLug uses a default for the connector where one is set, and otherwise the rate the provider publishes for every account. Copper (180 requests a minute) and Insightly (10 requests a second) publish one. Where the provider's limit depends on your plan, set the pace on the connection.
A Retry-After header from the provider always takes precedence over Max attempts and Cool-off. On a paced connection it holds every request on that connection, in every run, until the time it names.
When the pace is used up
A request waits for its turn for up to 60 seconds. If its turn is further away, the step fails with "The connection's rate limit is reached" and the time until the next request could be sent. A provider's daily allowance that is used up fails the step at once.
A scheduled pipeline can suspend instead of failing. Open the pipeline's Start node and set When a connection's rate limit is reached to Suspend and resume later. The run is marked Rescheduled and runs again once the pace allows, continuing from where its source stopped. This applies when:
- the run was started by a schedule or by Run now on a workflow job,
- every source in the run is a database or SQL Server table read in a stable order, and
- nothing after the sources would apply an effect twice. The rules are the same as for resuming an interrupted run.
Otherwise the run fails as described above, and the run log says why it was not suspended. A run that has been rescheduled 24 times fails.
Reading a large table
How a source pages through a table decides whether a large read stays fast.
- A table with a single-column primary key is read by seeking that key. Each chunk starts where the last one ended, so the hundredth chunk costs the same as the first.
- Any other table is read in a fixed order with an offset. Each chunk reads past every row before it, so the read slows down as it goes.
- A composite primary key is read with an offset, not by seeking.
Resuming an interrupted run
A run that was interrupted resumes from where it stopped only when its source was read in a stable order and nothing downstream of the source writes outside the run.
A step that inserts rows, runs SQL or a script, sends a message or a notification, calls an AI model, creates an approval or writes a document prevents a resume, because running it twice would apply its effect twice. A step that works over every row of the run also prevents a resume: a Group By, a Pivot, a Join, a Dedupe, or a Fork set to Buffer All. Transform, Filter, Router, Variable, Cache, Lookup, Union and Log steps do not. When a resume is not possible, the run log names the step and the reason, and the run starts again from the beginning.
Workloads that run differently
| Workload | What to expect |
|---|---|
| Reading an object with no unique key | Ordered paging. Slower on large tables, and the run cannot resume part way. |
| Snowflake sources | Ordered paging. Snowflake tables are not read by seeking a key. |
| Group By, Join or Pivot over a very large input | These read their whole input before producing output, so the run takes longer. Filter rows before them where you can. |
| A data pipeline with one slow node | A slow node holds up the nodes that feed it. |
| Oracle Merge with "delete when matched" | Not supported. Use a separate Delete step. |
| SQL Server Append into a table the driver refuses | The load retries with every column as text, which is much slower. The run log says when this happens. |
See also Reading and writing files.
Throughput figures
DataLug does not publish a rows-per-second figure per plan. Throughput depends on your database, your network and your data. The run viewer reports what your own runs did.