Generic REST API

Other

The Generic REST API connector calls any HTTP endpoint. The connection holds the base URL, the authentication and default request values. Each action holds the method, path, body and the shape of the response.

When to use it

• The system you need has an HTTP API and no dedicated connector.
• You call a webhook or an internal service as a step in the data pipeline.
• You send a payload that a Message node has built.

A dedicated connector, where one exists, knows the system's objects and operations. This one sends what you configure.

What you need

• The API's base URL, reachable from the runtime.
• The credential the API expects: an API key, a bearer token, a username and password, or an OAuth 2.0 client.
• The API's documentation for paths, bodies and rate limits.

Connect

The form has five sections: Endpoint, Authentication, Request defaults, Rate and limits and Environments.

Field	Section	What to enter
Base URL	Endpoint	The address every request starts from, such as https://api.company.com/v1.
Timeout (ms)	Endpoint	Default 15000.
Auth method	Authentication	None, API key, Bearer, Basic or OAuth 2.0.
Send in	Authentication	API key only. Header or Query parameter.
Header name or Parameter name	Authentication	API key only. Such as x-api-key.
API key	Authentication	API key only.
Bearer token	Authentication	Bearer only. Sent as Authorization: Bearer <token>.
Username, Password	Authentication	Basic only. Sent as Authorization: Basic <credentials>.
Grant type	Authentication	OAuth 2.0 only. Client credentials, Password grant or Authorization code.
Token URL, Client ID, Scope, Audience	Authentication	OAuth 2.0 only. From the API provider.
Headers, Query params, Path params, Body params	Request defaults	Values sent with every request. Nodes can add to or override them.
Retries	Request defaults	Retries after the first try.

Authorization code also takes an Authorization URL and a Client secret. Register the Redirect URI shown in the form with the API provider, then select Connect and sign in to the provider in the window that opens. The status reads Account connected once the provider returns a token. The sign-in uses PKCE. When the access token expires, the runtime refreshes it with the refresh token. Reconnect signs in again.

OAuth 2.0 can refresh tokens automatically and store them on the connection.

Stored secrets are never sent back to the browser.

Supported operations

Action	Node	Methods
Request (Source)	Source	GET, POST, HEAD, OPTIONS
Request (Target)	Load	POST, PATCH, PUT, DELETE

GET, DELETE, HEAD and OPTIONS requests are sent without a body.

A Lookup node can also use the connection: it sends one GET or POST per distinct key and adds fields from the response to each row. See REST API lookups.

An action names the method, the path and the body template. The response schema comes from one of three sources:

• Preview from API sends the request, stores the response as an example and infers the columns. Preview Variables (JSON) fills templates in the path and body for that request.
• Import response example (JSON) infers the columns from a pasted response, without calling the API.
• Import schema (columns) takes the columns as you define them, without calling the API.

Preview from API and Test on a Generic API connection need a role that can edit connections: Builder, Admin or Owner.

Import a spec

A saved connection can read the API's description and create its actions. The Import a spec card is in the Endpoint section of the connection form.

These formats are read:

• OpenAPI 3.0, 3.1 and 3.2, in JSON or YAML.
• Swagger 2.0.
• Postman collection 2.1 (JSON) and 3.0 (YAML).

1. Choose Upload file, Paste or URL, and give the spec.
2. Select Preview. The spec is read and nothing is saved yet.
3. When the spec lists several servers, pick the Base URL.
4. Leave Set authentication on to take the auth method from the spec.
5. Select Import.

A spec URL must be a public address on one of the ports listed under Troubleshooting.

The import sets:

Setting	Taken from
Base URL	The spec's servers, its host and base path, or the collection's baseUrl variable.
Auth method	The spec's security scheme: API key in a header or query, bearer, HTTP Basic, or OAuth 2.0 client credentials or authorization code.
Actions	One per operation, with the method, path, parameters, request body and response fields.
Pagination	Offset and limit, page number, cursor token, Link header or @odata.nextLink, read from the operation's parameters and responses.

Secrets are never part of an import. Enter the API key, token, username and password, or client secret under Authentication afterwards. A spec that uses HTTP Basic sets the Basic method.

Importing again compares the new spec with the saved one. When operations are changed or removed, the card lists what is added, changed and removed, and Replace import saves the new spec.

Importing needs a role that can edit connections: Builder, Admin or Owner.

Use an imported operation

On a Source or Load node that uses the connection, Imported operation lists the operations grouped by tag. Choosing one fills the method, the path, the required parameters, the pagination and the fields, so Map Fields has the columns without a preview call.

Responses

JSON and XML responses are both read. A response is read as XML when its content type is application/xml, text/xml or ends in +xml, or when it has no content type and starts with <?xml. In XML, attributes appear as fields named @name and element text as #text. The rows are picked from the parsed response by the action's response path, as for JSON.

XML responses that declare a DOCTYPE or entities are refused, as are documents nested more than 64 levels deep.

Request bodies

A Load action sends one of these bodies:

Body	What is sent
Row JSON (per row)	Each incoming row as a JSON object.
Batch JSON (all rows)	The whole chunk as a JSON array.
Static JSON	The JSON you enter. Templates are allowed.
XML template	The XML you enter. Templates are allowed. Sent as application/xml unless a content type header is set.
Row XML (per row)	Each row as one XML element per field, under the root element you name. The default root is row.
Form (URL encoded)	The form fields you list, or the row's fields when none are listed.
Multipart form	The same fields as text parts of a multipart/form-data body.
None	No body.

Form field values accept templates. A multipart body replaces any configured content type header with its own.

Pagination

A Source action can read more than one page. Choose a style under Pagination. Only the fields for that style are shown.

Style	How the next page is requested	Fields
None	One request.	None.
Offset and limit	The offset advances by the page size.	Page size, Offset param, Limit param.
Page number	The page number advances by one.	Page size, Page param, Size param, First page.
Cursor token	A token read from each response goes into the next request.	Cursor response path, Cursor param.
Link header	Follows the Link response header with rel="next".	None.
Next URL in body	Follows the next URL found in each response.	Next URL response path.

With no Next URL response path, the source looks for @odata.nextLink, @nextLink and nextLink in the response.

Reading stops at the first of these:

• a page with no rows;
• a page with fewer rows than the page size;
• a response with no next token, link or URL;
• the maximum number of pages.

Max pages (optional) lowers the platform limit for one action. It cannot raise it. A next URL must keep the scheme, host and port of the first request, or the run fails before the request is sent.

Each page is one request, so pages count toward the connection's API limits.

A paged source that stops part way resumes from the last page it committed.

OData

A Source action can read an OData v4 service, such as Dataverse, Dynamics 365 or SAP S/4HANA. Set Request mode to OData. The connection's base URL is the service root, and the service's $metadata lists the entity sets.

Field	What to enter
Entity set	The entity set to read.
Fields	The fields to read. With none selected, every field is read.
Filters	Field, comparison and value. Match all or Match any combines them.
Order by, Direction	One field, ascending or descending.
Page size (optional)	Rows per page requested from the service. The service may send fewer.
Related records	Navigation properties to expand, each with the related fields to read. With no fields selected, every related field is read.
Related fields	Keep nested keeps each related record as an object. Flatten with a prefix writes its fields as Navigation_field columns.
Search (optional)	Free text sent as $search. AND, OR, NOT, parentheses and "quoted phrases" are kept.

A filter compares with equals, not equals, greater than, greater or equal, less than or less or equal. Text fields also offer contains, starts with and ends with. Each value is checked and written for the field's type, so quotes and query text inside a value stay part of the value.

A related collection stays a list in both Related fields settings. Search works only on services that support $search.

OData mode pages through @odata.nextLink automatically and reads the rows from value. Max pages (optional) still applies. Reload metadata reads the service's $metadata again after the service changes.

OData V2 services are not supported.

Limits

The Rate and limits section holds the API limits card. Set it from the provider's documented limit.

The API limits card on the connection form is collapsed by default. Every field on it is optional, and a blank field uses the default shown below it.

Setting	What it does	Default
Requests per period, Period	Paces outgoing calls. Set both to throttle.	No throttle, or the rate shown on the card.
Page size	Rows per request. A larger value is reduced to the provider's maximum.	The maximum shown on the card, or set by the provider.
Max attempts	Tries per call, including the first.	3
Cool-off (ms)	The first wait after a 429. Later waits grow from it.	500 ms
Requests at once	Requests open at the same time within one run.	No limit

A Retry-After header from the provider overrides the cool-off. The pace applies per connection and is shared by every run on it. See Run performance.

When both Retries and Max attempts are set, Max attempts wins. Retries counts retries only, so 2 there means three tries in total.

The OAuth 2.0 token request is paced with the data calls.

DataLug retries a call that the provider refused with 429, and a call that could not connect. A read is also retried after a 408, a 5xx or a dropped connection. A write is retried only after a 429 or a failed connection.

A Retry-After delay of up to one minute is honoured. A longer one fails the call with the 429.

Troubleshooting

• The call returns 401 or 403. The connector sends exactly the headers you configured. Check the header name as well as the value: an API expecting Authorization: Bearer <token> rejects Bearer: <token>.
• The response arrives but the Source returns no rows. The rows are inside the response body. Point the action at the array, not at the root.
• It works from your computer and not in DataLug. The runtime calls the endpoint from where it runs. A service reachable only on your network needs a Private Agent inside that network.
• The request is refused before it is sent. Requests go only to public addresses, on ports 80, 443, 3100, 8080, 8088, 8443, 9200 or 9243. Use one of those ports, or a Private Agent for a service on your network.
• Repeated 429 responses. The API is rate limiting the connection. Set Requests per period and Period to the provider's limit.

Related

• Source node, Load node and Message node.
• Connections for environments and sharing.