Grafana Loki
The Grafana Loki log destination sends this account's data pipeline logs to Grafana Cloud Logs or your own Loki server.
When to use it
- You run Grafana and want DataLug logs beside your other sources.
- You want to query run and connector failures with LogQL.
What you need
- The Loki URL and the numeric user ID. In Grafana Cloud, find both under Loki > Details.
- An access policy token with the
logs:writescope, created in Grafana Cloud. - For a self-hosted Loki behind a private certificate authority, that authority's certificate in PEM form.
Connect
A log destination is set up under Manage Account > Log Destinations, not on the canvas. Press New destination, choose the provider, and fill in its fields. Log Destinations is available on the Teams and Business plans.
Each destination sends the streams you choose:
| Stream | Selected by default |
|---|---|
| Data pipeline runs | Yes |
| Node execution | Yes |
| Connector errors | Yes |
| Platform errors | No |
| Audit trail | No |
| Milo usage | No |
Minimum severity is debug, info, warn, error or fatal. Events below
it are not sent.
The audit trail does not include the request IP address. Milo usage carries token counts and cost, never prompt or response text.
Each event carries a unique eventId. Delivery is at least once, so use
eventId to remove duplicates.
Secrets are stored encrypted and are never sent back to the browser. To change one, enter a new value.
Send test event sends one event to the provider. A destination saves only after a test event succeeds, and a changed setting needs a new test.
DataLug sends new events about once a minute. While the provider is unreachable, events wait in DataLug and are sent when it recovers.
The destination list shows each destination's Status, Last delivery or last error, and two counts:
- Dropped counts events in a batch the provider rejected outright, such as
with a
400. Those events are not sent again. - Refused counts events the provider rejected one by one inside an accepted batch.
| Field | What to enter |
|---|---|
| Name | A name for this destination. |
| Loki URL | An https address, such as https://logs-prod-012.grafana.net. |
| User ID | The numeric Loki user ID. It is not a secret and stays visible. |
| Access policy token | The token with logs:write. |
| Certificate authority | Optional. The PEM certificate for a private authority. Blank for a public one. |
What is sent
Each event carries four labels: account, environment, eventType and
severity. Everything else, including the run and node identifiers, is in the
log line as JSON, where LogQL can still filter on it.
Limits
- At most 4 MB per request.
- A log line is capped at 256 KB. A longer line is shortened to fit.
- Loki refuses a line older than one hour. After a long outage, events older than that are counted under Refused and not sent again.
Grafana Cloud limits ingestion per tenant, by default 5 MB per second and 5,000 active streams.
Troubleshooting
- The test passes but older events are missing. Loki refuses a line older than one hour. After an outage, the backlog beyond that window is counted under Refused.
- The destination shows Failing. Loki answered 401 or 403. The token was
revoked, or lacks the
logs:writescope. - 429 responses. The tenant is over its ingestion rate. DataLug sends the batch again on the next delivery.
- The tenant hit its stream limit. DataLug sends four labels only, so check whether another source writes to the same tenant.
Related
- Datadog, Elasticsearch and Splunk.
- Account for the Manage Account tabs.