Dynatrace

Observability

Synopsis

Creates a Dynatrace target that sends logs, metrics and traces to Dynatrace over OTLP.

The target is the OTLP target with Dynatrace's sign-in and limits built in, and it always sends counters as deltas.

Schema

- name: <string>
  description: <string>
  type: dynatrace
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    token_type: <string>
    token: <string>
    send_metrics: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    tls:
      status: <boolean>
      verify: <boolean>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be dynatrace
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y	-	The OTLP endpoint. For a SaaS environment: https://<environment-id>.live.dynatrace.com/api/v2/otlp. For an ActiveGate: https://<activegate>:9999/e/<environment-id>/api/v2/otlp. It must use https. In the console: OTLP endpoint
token_type	N	access	The kind of token: access for an access token, or platform for a platform token. In the console: Token type
token	Y	-	The token. It can be plain text, an environment variable or a secret. In the console: Token
send_metrics	N	true	Sends the readings of metric devices as metrics. false sends them as log records. In the console: Readings as metrics, on by default

Performance

Field	Required	Default	Description
compression	N	true	Compresses each request with gzip. YAML only
timeout	N	30	How long a request may take, in seconds
batch_size	N	1000	The most events sent in one request
max_retries	N	3	How many times a failed request is retried
retry_delay	N	1	How long to wait before a retry, in seconds
disable_retries	N	false	true sends each batch once, as on the OTLP page. In the console: Disable retries

TLS

The tls settings work as on the OTLP page. See An ActiveGate's Certificate.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf, so it ignores protocol, format, path, grpc_passthrough, address and port.

It signs in itself, so it replaces an Authorization header set under headers.

It always sends counters as deltas, so cumulative_to_delta has no effect on it.

The Console Form

The form has two steps:

• Dynatrace settings: OTLP endpoint; Token type and Token, the token with the Manual / Environment variable / Vault picker; Readings as metrics; the Performance fields.
• TLS settings: the OTLP form's TLS step.

Details

The Endpoint

For a SaaS environment, the endpoint is the environment's address followed by /api/v2/otlp. Use the live address: an apps address answers 404.

To send through an ActiveGate, use the ActiveGate's address, port 9999, and the environment's ID in the path.

The Token

Two kinds of token work:

• An access token (token_type: access, the default) works on every Dynatrace environment. It needs the scopes metrics.ingest, logs.ingest and openTelemetryTrace.ingest. The target sends it as Authorization: Api-Token <token>.
• A platform token (token_type: platform) works on the latest Dynatrace platform. It needs the scopes openpipeline:metrics:ingest, openpipeline:logs:ingest and openpipeline:traces:ingest. The target sends it as a bearer token.

Counters as Deltas

Dynatrace takes counters only as deltas, so this target always converts them, as described in the OTLP page's Counters as deltas section:

• The first point of each series is not sent.
• A restart of Director loses one interval of each series.
• All the points of a series must reach the same Director.

A delta counter from a metric device goes as a delta sum, which Dynatrace takes.

What Dynatrace Does with the Metrics

• A cumulative sum that can also fall arrives as a gauge.
• Dynatrace does not take cumulative exponential histograms, and the target does not convert them.
• Dynatrace does not take summaries.

Size Limits

Dynatrace limits each request. The target keeps:

• a logs request to 8,000,000 bytes. Dynatrace's limit is 16 MB, counted after it adds the resource attributes to each record;
• a metrics request to 4,000,000 bytes and 15,000 data points. Dynatrace drops a request over either limit whole;
• a traces request to 8,000,000 bytes.

The target splits a batch that would be larger. A single record over a limit by itself is dropped and counted as dropped, with a warning in the Director log.

Partial Success

When only some metrics are invalid, Dynatrace answers 200 and says how many points it refused. For example, it refuses a point with more than 50 dimensions, a dimension key over 100 characters, or a dimension value over 255 characters. The OTLP page's partial_success troubleshooting entry applies.

An ActiveGate's Certificate

An ActiveGate's certificate is self-signed by default. Set tls.status: true and tls.ca_name to the certificate authority that signed it, or set tls.verify: false.

With the TLS settings off, the target still uses https and checks the certificate against the system's trust store.

Examples

A SaaS Environment

A SaaS environment, with an access token from an environment variable...

targets: - name: dynatrace type: dynatrace properties: url: "https://abc12345.live.dynatrace.com/api/v2/otlp" token: "${DYNATRACE_TOKEN}"

An ActiveGate

An ActiveGate with a self-signed certificate, and a platform token from a secret...

targets: - name: dynatrace_activegate type: dynatrace properties: url: "https://activegate.example.com:9999/e/abc12345/api/v2/otlp" token_type: platform token: "$secret{dynatrace-platform-token}" tls: status: true ca_name: "activegate-ca"

Device Readings as Log Records

Device readings as log records instead of metrics...

targets: - name: dynatrace_logs type: dynatrace properties: url: "https://abc12345.live.dynatrace.com/api/v2/otlp" token: "$secret{dynatrace-token}" send_metrics: false

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

• url for Dynatrace must be the https OTLP endpoint, such as https://<environment-id>.live.dynatrace.com/api/v2/otlp
• token is required for Dynatrace
• token_type for Dynatrace must be access or platform
• failed to resolve the token of the Dynatrace target: ...: the environment variable or the secret named in token could not be read.

Errors from Dynatrace

The OTLP page's Troubleshooting entries apply. For Dynatrace in particular:

• status 401 or status 403: the token is wrong, token_type does not match the kind of token, or the token lacks a scope for the signal.
• status 404: the url is not the OTLP endpoint. It must end with /api/v2/otlp, and a SaaS environment's address must be the live one.
• status 429: Dynatrace is limiting the environment. The target retries.
• [OTLP] partial_success ... rejected=N: Dynatrace refused some points, and the message says why.
• [OTLP] dropped a record of N bytes: Dynatrace refuses an OTLP <signal> request over <limit> bytes, or [OTLP] dropped a record of N data points: Dynatrace refuses an OTLP metrics request of more than 15000 data points: a single record was too large to send. <signal> is logs, metrics or traces. In the log, the warning ends with target= and the name of the target.
• The warnings in the OTLP page's Counters sent as deltas troubleshooting entry apply too.