Webhook Trigger Node
What it does
The Webhook Trigger node starts a run when an external system sends a POST to a webhook URL, signed or with an unsigned URL token. The body of the request becomes the node's rows. JSON, form, XML and plain text bodies are accepted; Receive a webhook from any system covers each format, other signatures and splitting an order into line items.
The node is in the Source group of the component browser. Connect Start to it, as you would a Source.
When to use
- An external system sends events, such as Stripe, Shopify, GitHub or your own application, and each event should start a run.
- The sender can sign requests with a shared secret, so unsigned or altered requests are rejected.
- You want a record of every request, including the rejected ones.
Set up a webhook
- Open Manage Account > Webhooks. Owners and Admins manage webhooks.
- Select New webhook, enter a name, optionally choose a Project, and press Create webhook.
- Copy the Secret. It is shown once. Store it in the sender's settings.
- Copy the Inbound URL and set it as the sender's webhook address.
- On the canvas, add a Webhook Trigger node, connect Start to it, and bind it to the webhook. Saving the data pipeline links the webhook to its project.
Project
A webhook runs one project. The list under Webhooks has a Project column, and the webhook shows Project and the project name, or Not linked to a project. Deliveries to a webhook with no project fail until a saved Webhook Trigger node uses it.
- Saving a Webhook Trigger node bound to a webhook with no project links it to the node's project.
- A webhook linked to another project is listed under Bound webhook as Linked to that project, and cannot be chosen.
- Settings on the webhook changes or clears the project. A project cannot be cleared or changed while a saved Webhook Trigger node in it still uses the webhook: bind that node to another webhook, or remove it, and save first.
- A hosted form's webhook keeps its project.
Each request runs every data pipeline in the project whose Webhook Trigger node is bound to that webhook, one run per data pipeline. Only a saved node counts. When no data pipeline binds it, no run starts: the delivery is recorded as Not used by any pipeline, and the sender still gets its normal reply. Add a Webhook Trigger node that uses the webhook, save the data pipeline, then replay the delivery.
For a sender with its own signature, such as HubSpot, Calendly or Typeform, choose it under Signature before step 2. Receive a webhook from any system lists each sender and its settings. Some senders issue their own secret, such as a HubSpot client secret or a Calendly OAuth app's signing key: paste it when you create the webhook. A pasted secret is stored encrypted and never shown.
Configure
Select the node and press Configure in the side panel. Configure Webhook Trigger opens. Save keeps the dialog open, Done saves and closes it, and Cancel discards the changes since the last save. A message confirms each save.
- Webhook: one of the account's webhooks. The dialog shows its sender, its inbound URL with a copy button, and the setup steps for that sender. An unsigned webhook is marked Unsigned; its URL is shown once, on create and rotate. To change the sender, open Manage Account > Webhooks. With no webhook bound, choose a Sender for a new webhook and press Create webhook: the webhook settings open in a new tab with that sender chosen.
- Output: Raw payload gives the rows below. Split array at path gives one row per element of the list at a field path, such as
order.line_items, with the other fields of its parent added with the prefixparent_. Meta leads reads each lead from Meta through the chosen Meta Lead Ads connection and gives one row per lead; see Meta Lead Ads. Lead gives one row per lead with the same columns from every sender; see Lead output below. In a split path,[]takes every element part way along, as inentry[].changes[].value; it cannot end the path. A split that would copy more than 1,000,000 parent fields in total (rows times parent fields) fails the node. - Input schema (optional): a JSON Schema that describes the body, so downstream nodes can complete field names. It does not validate requests. Only valid JSON is saved.
Lead output
With Rows from each delivery set to Lead, every lead becomes one row with the same columns, whichever form or sender it came from. A column with no value is empty.
| Column | What it holds |
|---|---|
source | The sender: typeform, webflow, jotform, calendly, hubspot, meta, hosted_form or generic. |
sourceEventId | The sender's own id for the submission. |
formId | The sender's form id. For Calendly, the event type. |
submittedAt | When the sender says the form was submitted. Numeric times become ISO 8601; text is kept as sent. |
receivedAt | When DataLug accepted the delivery. |
email, phone | Contact details. An email needs an @. |
firstName, lastName, fullName | The name. A full name is split into first and last when those are missing, and built from them when it is missing. |
company | The company name. |
consent, consentText | true, false or empty, and the consent question or text. |
utmSource, utmMedium, utmCampaign | Campaign tags, when the sender passes them. |
extra | Every other answer, under the name the sender used. Nothing is dropped. |
raw | The delivery as received. |
DataLug fills the columns from the fields each sender names, then by field type, then by field name or label. For any other sender it looks at the top-level fields first and then up to three levels down, such as an order's customer.email.
To choose where a column comes from, press Map a column and enter a field path, such as contact.email or answers[2].text. A mapping wins whenever its path holds a value. A path that is not valid fails the node with a message naming the column. source, receivedAt, extra and raw cannot be mapped.
For a Meta Lead Ads webhook, pick the Page's connection under Meta Lead Ads connection (Optional): each lead is read from Meta first.
Ports
| Port | Direction | Kind | Notes |
|---|---|---|---|
in | in | control | Connect Start here. Carries no rows. |
out | out | data | The rows built from the request body. |
error | out | control | Receives a row with the error when the node fails. |
How a body becomes rows
| Body | Rows |
|---|---|
| A JSON array | One row per element. Ten orders give ten rows. |
| A JSON object | One row. An empty object is still one row. |
| Any other JSON value, such as a string or number | One row with the value in a value column. |
| An empty array | No rows. |
Running the flow by hand produces no rows, because there is no request to read. The run log says so.
Signing requests
Each POST must carry this header:
X-DataLug-Signature: t=<unix-seconds>,v1=<hex-hmac>
v1 is HMAC_SHA256(secret, "<t>." + raw-body-bytes). Sign the raw body bytes as sent. Tools that produce Stripe-style signature headers work with small changes.
The timestamp t must be within 5 minutes of the server's time. Older requests are rejected as replays.
Each request that verifies starts one run. A request whose body is identical, byte for byte, to a delivery for the same webhook in the last 24 hours is a duplicate: it is answered 202 with the first delivery's id and "duplicate": true, and no second run starts. The same body after 24 hours is a new delivery, and so is a retry with a changed body, such as a new timestamp inside it. If the work must not repeat, key the downstream Load on an id the sender supplies and use upsert, or set Event id for duplicates on the webhook so a repeat of the same event never runs twice.
Responses
| Code | When |
|---|---|
202 Accepted | The signature verified and the delivery was recorded, or it is a duplicate of an earlier one. |
400 Bad Request | The signature verified but the body cannot be read: JSON that does not parse, XML with a DOCTYPE, or a file upload. |
401 Unauthorized | The signature did not verify: wrong secret, changed body or an old timestamp. |
404 Not Found | No webhook has this id. Nothing is recorded. |
410 Gone | The webhook is disabled. |
415 Unsupported Media Type | The Content-Type is not one DataLug reads. |
429 Too Many Requests | Over the rate limit. The Retry-After header says when to retry. |
Rate limits: 100 requests per minute per webhook, and 500 per minute across all of an account's webhooks.
Deliveries
Every request is recorded as a delivery, including rejected ones. Press Deliveries on a webhook to see them, with the time received, status, reason, run id and payload. Hover over a reason to see its code, which support may ask for. Select a run id to open that run on Runtime Executions.
| Status | Meaning | Replay |
|---|---|---|
Accepted | The signature verified and the delivery is queued. The run has not started yet. | Yes |
Dispatched | The run started. Open the run to see how it ended. | No |
Failed | The run cannot start and retrying will not help, for example because no project is bound or the account is disabled. The reason is recorded. | Yes |
Dead letter | Every automatic retry failed. The last reason is recorded. | Yes |
Not used by any pipeline | The signature verified, but no saved data pipeline in the project has a Webhook Trigger node that uses this webhook. No run started. | Yes |
Rejected | The signature or body was invalid. The reason is recorded. | No |
Automatic retries. When the run cannot start because the runtime is busy, restarting or unreachable, DataLug retries the delivery on its own for about 75 minutes, waiting longer between attempts and honouring a Retry-After the runtime sends. The delivery stays Accepted while it waits, with the last reason shown.
Dead letter. A delivery whose retries all fail moves to Dead letter. The webhook shows how many deliveries are in dead letter, and Owners and Admins are told by email, at most once an hour per webhook. The Operational alerts recipient in Manage Account > Notifications gets the same email while that card is on, and accounts with log destinations receive the event there.
Replay. Press Replay on a delivery to queue it again with a fresh set of retries, or Replay all in dead letter in the deliveries list to queue every dead-lettered delivery of that webhook at once. A replay sends the stored payload.
A rejected delivery cannot be replayed, because its sender was never verified. A delivery whose payload was removed after the retention period cannot be replayed either.
The run uses the project's default environment.
Payload: press Payload on a delivery to see the body exactly as sent. It may hold personal data, so it stays closed until you open it, and only Owners and Admins can see it.
Retention: a delivery's payload and headers are kept for 30 days, then removed. The delivery record, with its status, reason and run id, stays.
A delivery still Accepted after 24 hours with no retry queued is marked Failed with the reason Not started within 24 hours instead of starting late. Replay it if the run is still wanted.
A delivery whose run could not load its workflow is marked Failed with the reason Workflow could not be loaded. No step of the workflow ran, so it can be replayed.
When the project runs on a Private Agent that is offline, deliveries wait for it and run once it reconnects. If the agent takes a delivery and stops before starting it, the delivery is offered again. After three such attempts it is marked Failed with the reason Private Agent never started it, and can be replayed.
Test a webhook
Press Test on a webhook, paste a JSON body (up to 64 KB) and press Send test payload. DataLug signs the body with the webhook's secret and sends it to the inbound URL.
A test is a real delivery and starts a real run. If the flow writes to a production system, the test writes there too.
The result shows the response code, the delivery id and the signature header that was sent.
Rotate the secret
Press Rotate to replace the secret. The new secret is shown once. The old secret keeps working for 24 hours, so the sender can be updated without rejected deliveries.
Press Revoke old secret to stop the old secret working before the 24 hours are up, for example once the sender uses the new one, or if the old one may have leaked.
Secrets are stored encrypted and are never shown again after they are created or rotated.
Examples
Record payments as they happen.
The payment provider sends each completed payment to the inbound URL. The Transform maps the payload fields to the ledger columns, and the Load upserts on the payment id, so a retried event updates the same row.
Route by event type. Webhook Trigger > Router on type > payment.succeeded to one Transform and Load, payment.refunded to another.
Troubleshooting
| Symptom | Cause | What to do |
|---|---|---|
Deliveries show Rejected with a signature reason | The sender signs with the wrong secret or changes the body before signing. | Check the secret, and sign the raw body bytes. |
Deliveries show Rejected with a timestamp reason | The sender's clock is off by more than 5 minutes. | Correct the sender's clock. |
| No deliveries appear | The sender uses the wrong URL. | Copy the inbound URL again from Webhooks. |
| A manual run has no rows | There is no request in a manual run. | Send a test payload instead. |
Deliveries show Dead letter | The runtime was unavailable for longer than the retry window. | Check the runtime, then press Replay all in dead letter. |
Deliveries show Not used by any pipeline | No saved data pipeline has a Webhook Trigger node that uses the webhook. | Add the node, save the data pipeline, then press Replay on each delivery. |
| A test payload is not run | It is identical to a test sent in the last 24 hours, so it is a duplicate. | Change the payload and send it again. |
| The same event ran twice | The sender retried with a changed body. | Upsert on an id the sender supplies. |