Elasticsearch
The Elasticsearch log destination sends this account's data pipeline logs to a data stream in Elastic Cloud or your own Elasticsearch cluster.
When to use it
- You run Elastic and want DataLug logs in the same cluster.
- You build Kibana dashboards over run and connector failures.
What you need
- The Elasticsearch URL, as an
httpsaddress. - A data stream name, such as
logs-datalug-default. - An API key that can write to that data stream. In Kibana, go to Stack
Management > Security > API keys, create the key, and copy the
encodedvalue Elastic shows. - For a self-hosted cluster behind a private certificate authority, that authority's certificate in PEM form.
Connect
A log destination is set up under Manage Account > Log Destinations, not on the canvas. Press New destination, choose the provider, and fill in its fields. Log Destinations is available on the Teams and Business plans.
Each destination sends the streams you choose:
| Stream | Selected by default |
|---|---|
| Data pipeline runs | Yes |
| Node execution | Yes |
| Connector errors | Yes |
| Platform errors | No |
| Audit trail | No |
| Milo usage | No |
Minimum severity is debug, info, warn, error or fatal. Events below
it are not sent.
The audit trail does not include the request IP address. Milo usage carries token counts and cost, never prompt or response text.
Each event carries a unique eventId. Delivery is at least once, so use
eventId to remove duplicates.
Secrets are stored encrypted and are never sent back to the browser. To change one, enter a new value.
Send test event sends one event to the provider. A destination saves only after a test event succeeds, and a changed setting needs a new test.
DataLug sends new events about once a minute. While the provider is unreachable, events wait in DataLug and are sent when it recovers.
The destination list shows each destination's Status, Last delivery or last error, and two counts:
- Dropped counts events in a batch the provider rejected outright, such as
with a
400. Those events are not sent again. - Refused counts events the provider rejected one by one inside an accepted batch.
| Field | What to enter |
|---|---|
| Name | A name for this destination. |
| Elasticsearch URL | An https address, such as https://my-deployment.es.us-central1.gcp.cloud.es.io. |
| Data stream | Lower case, no spaces. |
| API key | The encoded API key value. |
| Certificate authority | Optional. The PEM certificate for a private authority. Blank for a public one. |
The URL and data stream are not secret and stay visible.
What is sent
Each event is a document with the ECS fields @timestamp, message,
log.level and event.dataset set to datalug.logs. Fields specific to
DataLug are under datalug, such as datalug.runId and datalug.fields.
Documents are written with the create action, which a data stream requires.
Limits
- At most 1000 documents or 4 MB per request.
- Elasticsearch can accept a batch and refuse single documents, usually when a field arrives as a string in one event and an object in another. Those documents are counted under Refused and not sent again.
Troubleshooting
- Some events arrive and some do not, with no error. Elasticsearch accepted the batch and refused single documents, usually for a mapping conflict. Check the Refused count and the data stream's mapping.
- Nothing arrives and the destination shows Failing. The API key cannot write to the data stream, or the data stream does not exist and no index template creates it.
- The data stream name is refused on save. It must be lower case with no
spaces, must not contain
/ * ? " < > | , # :, and must not start with-,_,+or.. - A certificate error against a self-hosted cluster. Paste the private authority's PEM into Certificate authority.
Related
- Datadog, Grafana Loki and Splunk.
- Account for the Manage Account tabs.