Splunk Observability Cloud

Observability

Synopsis

Creates a Splunk Observability Cloud target that sends metrics and traces to Splunk Observability Cloud over OTLP.

The target is the OTLP target with Splunk Observability Cloud's sign-in built in. To send logs to the Splunk platform, use the Splunk target.

Schema

- name: <string>
  description: <string>
  type: splunkobservability
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    token: <string>
    cumulative_to_delta: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `splunkobservability`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Connection

Field	Required	Default	Description
`url`	Y	-	The ingest URL of the organization's realm, such as `https://ingest.us1.observability.splunkcloud.com`. The legacy `https://ingest.<realm>.signalfx.com` works too. It must use https. The target sends metrics to `/v2/datapoint/otlp` and traces to `/v2/trace/otlp` under it. In the console: Ingest URL
`token`	Y	-	An access token that can ingest data. It can be plain text, an environment variable or a secret. In the console: Access token
`cumulative_to_delta`	N	`true`	Sends counters as deltas, because Splunk Observability Cloud asks for delta histograms. While it is on, the target uses one worker. `false` sends counters unchanged. In the console: Counters as deltas

Performance

Field	Required	Default	Description
`compression`	N	`true`	Compresses each request with gzip. YAML only
`timeout`	N	`30`	How long a request may take, in seconds
`batch_size`	N	`1000`	The most events sent in one request
`max_retries`	N	`3`	How many times a failed request is retried
`retry_delay`	N	`1`	How long to wait before a retry, in seconds
`disable_retries`	N	`false`	`true` sends each batch once, as on the OTLP page. In the console: Disable retries

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf, and it checks Splunk Observability Cloud's certificate against the system's trust store. So it ignores protocol, format, path, grpc_passthrough, address, port and tls.

It always sends the readings of metric devices as metrics, so it ignores send_metrics.

It signs in itself, so it replaces an X-SF-Token header set under headers.

The Console Form

The form has one step, Splunk Observability Cloud settings:

a notice, This target sends only metrics and traces, which says to use the Splunk target to send logs;
Ingest URL;
Access token, with the Manual / Environment variable / Vault picker;
Counters as deltas, on by default;
the Performance fields.

Details

Where to Find the Values

The ingest URL: the one for the organization's realm, such as https://ingest.us1.observability.splunkcloud.com or https://ingest.eu0.observability.splunkcloud.com. The legacy https://ingest.<realm>.signalfx.com form works too.
The access token: an access token with the Ingest authorization scope. Splunk Observability Cloud lists the organization's tokens under Settings → Access Tokens.

How the Target Signs In

The target sends the access token in the X-SF-Token header of every request.

Metrics and Traces Only

The target sends metrics to /v2/datapoint/otlp and traces to /v2/trace/otlp under the URL. Splunk Observability Cloud's ingest URL takes no logs, so logs go to the Splunk platform through the Splunk target. This target drops a log record, counts it as dropped, and logs a warning at most once a minute.

Readings as Metrics

The target always sends the readings of metric devices as metrics. It ignores send_metrics.

Counters as Deltas

Splunk Observability Cloud asks for delta histograms, so cumulative_to_delta is on by default for this target. It works as described in the OTLP page's Counters as deltas section:

The first point of each series is kept and not sent, so a restart of Director loses one interval of each series.
The target uses one worker while the setting is on.
One Director must see all the points of a series.

Set cumulative_to_delta to false to send counters unchanged.

Size Limit

The target sets no limit on the size of a request.

Everything else works as on the OTLP page.

Examples

The Ingest URL of a Realm

The ingest URL of a realm, with the token from a secret...

targets:
  - name: splunkobservability
    type: splunkobservability
    properties:
      url: "https://ingest.us1.observability.splunkcloud.com"
      token: "$secret{store=corp-vault,ref=splunk/o11y-ingest-token}"

The Legacy URL, with Counters Sent Unchanged

The legacy URL, with the token from an environment variable, and counters sent unchanged...

targets:
  - name: splunkobservability_legacy
    type: splunkobservability
    properties:
      url: "https://ingest.us1.signalfx.com"
      token: "${SPLUNK_O11Y_TOKEN}"
      cumulative_to_delta: false

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

url is required for Splunk Observability Cloud
url for Splunk Observability Cloud must be the https ingest URL of the realm, such as https://ingest.us1.observability.splunkcloud.com
token is required for Splunk Observability Cloud
the token of the Splunk Observability Cloud target resolved to nothing: check the environment variable or the secret it names: token is set, but its environment variable or secret is empty.
failed to resolve the token of the Splunk Observability Cloud target: ...: the environment variable or the secret could not be read.

Warnings While It Runs

These appear in the Director log:

[OTLP] dropped N log records in the last minute: Splunk Observability Cloud takes only metrics and traces: the target dropped log records, because it sends only metrics and traces. Director logs this warning at most once a minute.
[OTLP] dropped N points of new series: cumulative_to_delta keeps at most 100000 series: the target already follows 100,000 series. This warning can appear only while cumulative_to_delta is on. See the OTLP page's Counters sent as deltas troubleshooting entry.

The OTLP page's Troubleshooting entries apply too.

Synopsis​

Schema​

Configuration​

Connection​

Performance​

Scheduling​

Debug Options​

Settings That Do Not Apply​

The Console Form​

Details​

Where to Find the Values​

How the Target Signs In​

Metrics and Traces Only​

Readings as Metrics​

Counters as Deltas​

Size Limit​

Examples​

The Ingest URL of a Realm​

The Legacy URL, with Counters Sent Unchanged​

Troubleshooting​

Configuration Errors​

Warnings While It Runs​