Datadog (Metrics and Traces)

Observability

Synopsis

Creates a Datadog (metrics and traces) target that sends metrics and traces to Datadog over OTLP.

The target is the OTLP target with Datadog's sign-in and limits built in. To send logs to Datadog, use the Datadog target.

Schema

- name: <string>
  description: <string>
  type: datadogotlp
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    api_key: <string>
    otlp_source: <string>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `datadogotlp`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Connection

Field	Required	Default	Description
`url`	Y	-	The OTLP intake of the organization's Datadog site, such as `https://otlp.datadoghq.com`. It must use https. The target sends metrics to `/v1/metrics` and traces to `/v1/traces` under it. In the console: OTLP endpoint
`api_key`	Y	-	A Datadog API key. It can be plain text, an environment variable or a secret. In the console: API key
`otlp_source`	N	-	The value that Datadog gives an organization that it allows to send traces through the intake. The target sends it in the `dd-otlp-source` header. In the console: OTLP source (Optional)

Performance

Field	Required	Default	Description
`compression`	N	`true`	Compresses each request with gzip. YAML only
`timeout`	N	`30`	How long a request may take, in seconds
`batch_size`	N	`1000`	The most events sent in one request
`max_retries`	N	`3`	How many times a failed request is retried
`retry_delay`	N	`1`	How long to wait before a retry, in seconds
`disable_retries`	N	`false`	`true` sends each batch once, as on the OTLP page. In the console: Disable retries

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf, and it checks Datadog's certificate against the system's trust store. So it ignores protocol, format, path, grpc_passthrough, address, port and tls.

It always sends the readings of metric devices as metrics, so it ignores send_metrics. It always sends counters as deltas, so cumulative_to_delta has no effect.

It signs in itself, so it replaces a dd-api-key header set under headers. It also replaces a dd-otlp-source header when otlp_source has a value.

The Console Form

The form has one step, Datadog (metrics and traces) settings:

a notice, This target sends only metrics and traces. It says that Datadog takes counters only as deltas, so Director sends each counter as its change since the last reading, and to use the Datadog target to send logs;
OTLP endpoint;
API key, with the Manual / Environment variable / Vault picker;
OTLP source (Optional);
the Performance fields.

Details

Where to Find the Values

The URL: the OTLP intake of the organization's Datadog site, such as https://otlp.datadoghq.com. Datadog's OTLP documentation lists the endpoint of each site.
The API key: a Datadog API key. It must belong to the same Datadog site as the URL.
The OTLP source: Datadog gives this value when it allows the organization to send traces through the intake. Leave it out when you send only metrics.

How the Target Signs In

The target sends the API key in the dd-api-key header of every request. When otlp_source has a value, it also sends the value in the dd-otlp-source header. An empty otlp_source sends no header.

Metrics and Traces Only

The target sends metrics to /v1/metrics and traces to /v1/traces under the URL. It drops a log record, counts it as dropped, and logs a warning at most once a minute. Send logs with the Datadog target.

Readings as Metrics

The target always sends the readings of metric devices as metrics. It ignores send_metrics.

Counters as Deltas

Datadog's intake takes only deltas, so the target always sends counters as deltas. cumulative_to_delta has no effect on this target. The conversion works as described in the OTLP page's Counters as deltas section:

The first point of a cumulative series is the baseline. The target keeps it and does not send it, so a restart of Director loses one interval of each series.
A counter that is already a delta goes out unchanged, with its first point.
The target uses one worker.
One Director must see all the points of a series.

Size Limit

Datadog refuses a metrics request over 512 KiB after compression. The target keeps a metrics request at or under 512,000 bytes before compression, and a traces request at or under 3,200,000 bytes. It splits a batch that would be larger. A single record over a limit is dropped and counted as dropped, with a warning in the Director log.

Everything else works as on the OTLP page.

Examples

The OTLP Intake of a Datadog Site

The OTLP intake of a Datadog site, with the API key from a secret...

targets:
  - name: datadogotlp
    type: datadogotlp
    properties:
      url: "https://otlp.datadoghq.com"
      api_key: "$secret{store=corp-vault,ref=datadog/api-key}"

An Organization Allowed to Send Traces

An organization that Datadog allows to send traces, with the API key from an environment variable...

targets:
  - name: datadogotlp_traces
    type: datadogotlp
    properties:
      url: "https://otlp.datadoghq.com"
      api_key: "${DATADOG_API_KEY}"
      otlp_source: "<the value that Datadog gives>"

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

url is required for Datadog
url for Datadog must be the https OTLP intake of the Datadog site, such as https://otlp.datadoghq.com
api_key is required for Datadog
the api_key of the Datadog target resolved to nothing: check the environment variable or the secret it names: api_key is set, but its environment variable or secret is empty.
failed to resolve the api_key of the Datadog target: ...: the environment variable or the secret could not be read.

Warnings While It Runs

These appear in the Director log:

[OTLP] dropped N log records in the last minute: Datadog takes only metrics and traces: the target dropped log records, because it sends only metrics and traces. Director logs this warning at most once a minute.
[OTLP] dropped a record of up to N bytes: Datadog refuses an OTLP metrics request over 512000 bytes: a single metrics record was too large to send.
[OTLP] dropped a record of N bytes: Datadog refuses an OTLP traces request over 3200000 bytes: a single traces record was too large to send.
[OTLP] dropped N points of new series: cumulative_to_delta keeps at most 100000 series: the target already follows 100,000 series. See the OTLP page's Counters sent as deltas troubleshooting entry.

Errors from Datadog

The OTLP page's Troubleshooting entries apply. For Datadog in particular:

status 403: Datadog refused the request. One cause is that the URL and the API key belong to different Datadog sites.

Synopsis​

Schema​

Configuration​

Connection​

Performance​

Scheduling​

Debug Options​

Settings That Do Not Apply​

The Console Form​

Details​

Where to Find the Values​

How the Target Signs In​

Metrics and Traces Only​

Readings as Metrics​

Counters as Deltas​

Size Limit​

Examples​

The OTLP Intake of a Datadog Site​

An Organization Allowed to Send Traces​

Troubleshooting​

Configuration Errors​

Warnings While It Runs​

Errors from Datadog​