Skip to main content

Splunk

Observability

The Splunk log destination sends this account's data pipeline logs to Splunk Cloud or Splunk Enterprise through the HTTP Event Collector (HEC).

When to use it​

  • Splunk is your system of record for operational logs.
  • You want DataLug run failures searchable beside your other sources.

What you need​

  • The HEC address, as an https URL. Splunk Cloud uses port 443, at an address such as https://http-inputs-yourstack.splunkcloud.com. Splunk Enterprise and Splunk Cloud trials use port 8088, so include it.
  • A HEC token. In Splunk, go to Settings > Data inputs > HTTP Event Collector. Set indexer acknowledgement on the token to match Wait for indexer acknowledgement on the destination.
  • For a self-hosted Splunk behind a private certificate authority, that authority's certificate in PEM form.

Connect​

A log destination is set up under Manage Account > Log Destinations, not on the canvas. Press New destination, choose the provider, and fill in its fields. Log Destinations is available on the Teams and Business plans.

Each destination sends the streams you choose:

StreamSelected by default
Data pipeline runsYes
Node executionYes
Connector errorsYes
Platform errorsNo
Audit trailNo
Milo usageNo

Minimum severity is debug, info, warn, error or fatal. Events below it are not sent.

The audit trail does not include the request IP address. Milo usage carries token counts and cost, never prompt or response text.

Each event carries a unique eventId. Delivery is at least once, so use eventId to remove duplicates.

Secrets are stored encrypted and are never sent back to the browser. To change one, enter a new value.

Send test event sends one event to the provider. A destination saves only after a test event succeeds, and a changed setting needs a new test.

DataLug sends new events about once a minute. While the provider is unreachable, events wait in DataLug and are sent when it recovers.

The destination list shows each destination's Status, Last delivery or last error, and two counts:

  • Dropped counts events in a batch the provider rejected outright, such as with a 400. Those events are not sent again.
  • Refused counts events the provider rejected one by one inside an accepted batch.
FieldWhat to enter
NameA name for this destination.
Splunk URLThe HEC address, with the port when it is 8088.
HEC tokenThe token.
IndexOptional. Blank uses the token's default index.
SourcetypeOptional. Blank uses datalug:log.
Certificate authorityOptional. The PEM certificate for a private authority. Blank for a public one.
Wait for indexer acknowledgementOptional. Off by default. See below.

The URL, index and sourcetype are not secret and stay visible.

Indexer acknowledgement​

With Wait for indexer acknowledgement off, a successful response from HEC counts as sent. That response means the collector accepted the events, not that an index stored them.

With it on, DataLug asks Splunk whether each batch reached an index, and moves on only after Splunk confirms. Delivery takes about a minute longer per batch. If Splunk does not confirm within five minutes, the batch is sent again, so remove duplicates on eventId. Confirmation needs indexer acknowledgement on the HEC token as well. With it off on the token, a batch counts as sent when HEC accepts it.

What is sent​

Each event carries source set to datalug, the sourcetype, and an event with the message and severity. Identity fields such as accountId, runId and workflowId are in the HEC fields object. Nested values are flattened into dotted keys, since HEC accepts flat fields only.

Limits​

  • At most 1000 events or 4 MB per request.

Troubleshooting​

  • The test passes but events are not in the index you expected. A blank Index uses the token's default index. Set it explicitly.
  • Nothing arrives and the destination shows Failing. Splunk answers 403 for a disabled or wrong token. Check that the token is enabled under Data inputs and allows the index you named.
  • Connection refused or timing out. The port is wrong. Splunk Cloud uses 443. Splunk Enterprise and Splunk Cloud trials use 8088, which must be in the URL.
  • HEC answers that a data channel is missing. Indexer acknowledgement is on for the token and off on the destination. Turn on Wait for indexer acknowledgement, or turn acknowledgement off on the token.
  • A field is missing from search. Nested values are flattened. Search for parent.child, not a nested object.
  • A certificate error against a self-hosted Splunk. Paste the private authority's PEM into Certificate authority.