OpenTelemetry Protocol (OTLP)

Network

Synopsis

Creates a target that forwards telemetry data to any OpenTelemetry Protocol (OTLP)-compatible receiver. Supports HTTP and gRPC transports, protobuf and JSON encoding, gzip compression, custom headers, TLS, and configurable retry logic.

Schema

- name: <string>
  description: <string>
  type: otlp
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    address: <string>
    port: <numeric>
    protocol: <string>
    path: <string>
    format: <string>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    grpc_passthrough: <boolean>
    send_metrics: <boolean>
    cumulative_to_delta: <boolean>
    headers:
      <key>: <value>
    tls:
      status: <boolean>
      verify: <boolean>
      cert_name: <string>
      key_name: <string>
      server_name: <string>
      min_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be otlp
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y*	-	Full endpoint URL for HTTP (e.g. http://collector:4318) or host:port for gRPC
address	Y*	-	Remote host; alternative to url
port	Y*	-	Remote port; alternative to url

* = Provide url or both address and port. Exactly one form is required.

Transport

Field	Required	Default	Description
protocol	N	http	Wire transport: http or grpc
path	N	/v1/logs	HTTP URL path for the logs signal; leading / is added automatically. Metrics and traces are routed to /v1/metrics and /v1/traces automatically, regardless of this value. HTTP only.
format	N	protobuf	Encoding: protobuf or json. json is invalid with protocol: grpc.
compression	N	true	Enable gzip compression
send_metrics	N	false	true sends the readings of metric devices as OTLP metrics instead of log records. Records without readings still go as log records. When the field is absent or false, nothing changes. In the console this is the Readings as metrics switch. See Device readings as metrics
cumulative_to_delta	N	false	Sends cumulative counters and histograms as deltas: the change since the previous point of the same series. In the console: Counters as deltas, in the OTLP settings step under Transport. See Counters as deltas

Performance

Field	Required	Default	Description
timeout	N	30	Request or RPC timeout in seconds
batch_size	N	1000	Maximum events per batch; must be >= 1
max_retries	N	3	Maximum delivery attempts on failure; must be >= 0
retry_delay	N	1	Seconds between retry attempts
disable_retries	N	false	true makes one attempt per batch and never waits to retry it, whatever max_retries holds. The payload that attempt fails is still handed back and redelivered by the sender, as any failed send is. Absent or false, nothing changes. In the console this is the Disable retries switch

gRPC

Field	Required	Default	Description
grpc_passthrough	N	true	When true, sends raw OTLP bytes using content-subtype: raw-otlp (VirtualMetric-to-VirtualMetric only). When false, re-encodes as standard typed OTLP/gRPC for interoperability with any spec-compliant receiver.

Headers

Field	Required	Default	Description
headers	N	-	Map of string key-value pairs added to every HTTP request or gRPC metadata (e.g. authorization tokens, tenant IDs)

TLS

Field	Required	Default	Description
tls.status	N	false	Enable TLS

TLS material is resolved through the shared client builder, so these keys mean the same thing on every target that uses it. They are nested under a tls: block.

Field	Required	Default	Description
tls.verify	N	true	Verify the server certificate. On by default — set it to false only to accept an untrusted certificate, and only where you control the network path
tls.server_name	N	-	SNI hostname override for the handshake. Use it when the certificate's name does not match the address you connect to
tls.ca_name	N	-	CA bundle used to verify the server certificate. When unset the host trust store is used; when set it replaces the host trust store rather than adding to it
tls.cert_name	N*	-	Client certificate, for mutual TLS
tls.key_name	N*	-	Client private key, for mutual TLS
tls.passphrase	N	-	Passphrase for an encrypted client private key
tls.min_tls_version	N	tls1.2	Lowest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)
tls.max_tls_version	N	tls1.3	Highest protocol version accepted (tls1.0, tls1.1, tls1.2, tls1.3)

* cert_name and key_name are individually optional but must be supplied together — a certificate without its key, or a key without its certificate, is a configuration error.

note

tls.max_tls_version is accepted but not applied on this target — the sender deliberately leaves the maximum version unset, so only tls.min_tls_version takes effect here.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The OTLP target delivers telemetry to any OpenTelemetry-compatible receiver — otel-collector, rotel, Grafana Alloy, vendor ingest endpoints, or another VirtualMetric Director.

To send metrics to Prometheus, Grafana Mimir, VictoriaMetrics, Thanos or Cortex with the Prometheus remote-write protocol, use the Prometheus remote-write target.

To send to a Grafana Cloud stack, use the Grafana Cloud target. To send to a Grafana stack you run yourself, see A Self-Managed Grafana Stack.

To send to a Dynatrace environment, use the Dynatrace target. It signs in with a Dynatrace token, keeps to Dynatrace's request limits and always sends counters as deltas.

Record translation

Records originating from this Director's own OTEL listener carry a JSON envelope with full Resource and Scope context. The target parses the envelope and reassembles each record as a one-record ExportLogsServiceRequest, preserving the original Resource and Scope end-to-end. This path is lossless.

Records from non-OTEL sources (TCP listener, syslog, etc.) are wrapped as a synthetic LogRecord with the body set to the raw message bytes and minimal resource attributes (service.name = the device name). This path is lossy by definition — non-OTEL sources have no OTLP context — but produces valid OTLP that any downstream collector accepts.

With send_metrics: true, the readings of the devices listed under Device readings as metrics go as OTLP metrics instead of log records.

HTTP transport

HTTP is the default transport. Each worker maintains its own http.Client to avoid idle-connection contention. Batches are POSTed to endpoint + path. The Content-Type header is set to application/x-protobuf for format: protobuf and application/json for format: json. When compression: true, the body is gzip-compressed and Content-Encoding: gzip is set.

The signal is decided by the record, not by path. Metrics and traces from this Director's OTEL listener always go to /v1/metrics and /v1/traces on the same endpoint, whatever path is set to. The path field only overrides the route used for the logs signal, which is where a receiver with a non-standard logs route is accommodated. With send_metrics: true, device readings go to /v1/metrics on their own, so path still only changes the logs route. Every other record from a non-OTEL source is sent as logs, so setting path to /v1/metrics or /v1/traces posts a logs body to a route that expects something else, and the receiver rejects it.

gRPC transport

When protocol: grpc, each worker maintains its own connection. The gRPC method is chosen per signal, so logs, metrics and traces are all carried over this transport: /opentelemetry.proto.collector.logs.v1.LogsService/Export, /opentelemetry.proto.collector.metrics.v1.MetricsService/Export, and /opentelemetry.proto.collector.traces.v1.TracesService/Export. The receiver must serve the service for every signal you send it.

grpc_passthrough: true (default): sends raw OTLP bytes using grpcraw.Codec with content-subtype: raw-otlp. This is the fast path from one Director to another, with no re-serialization. A standard OTLP receiver typically refuses this subtype, so set it to false for anything else.

grpc_passthrough: false: re-encodes each batch into the typed proto message (ExportLogsServiceRequest, etc.) using the standard gRPC codec and marshals with UnmarshalVT. This produces spec-compliant OTLP/gRPC that any receiver accepts, at the cost of one additional unmarshal+marshal per outbound batch. Use false for cross-vendor deployments.

Device readings as metrics

By default every record from a source other than the OTEL listener goes as a log record. With send_metrics: true, the target sends the readings of metric devices as OTLP metrics instead. The setting is off by default.

What changes when the setting is on

• A record from a device in the table below goes to the metrics signal: /v1/metrics over HTTP, whatever path says, or the metrics service over gRPC.
• Each reading in the record becomes one data point.
• Every other record goes as before. A record from the OTEL listener keeps its own signal. The rest go as log records, including records from log sources, records from a device that is not in the table, and records that a pipeline has changed so that the target no longer finds the readings in them.
• There is no metrics-only mode. Routing decides which records reach the target.

What a data point carries

• A name: the device's own name for the reading. Examples: modbus.energy.voltage, a Prometheus sample's own name, or system.cpu.user.norm.pct on a Linux host.
• A kind: a gauge, or a monotonic sum for a running total. A sum is cumulative, or delta where the device reports a count for each interval.
• A unit, where the device states one. Units use OpenTelemetry's notation, such as 1 for a fraction, %, By, By/s, s, ms, us, ns, Cel, W, V, J, MHz and {rpm}.
• Attributes: the device's labels, plus the fields that tell one series from another series of the same name, such as a Modbus slave id and register type, an OPC UA node, an interface, a disk, or a virtual machine's name. A reading's value, a time and a changing status are never attributes.
• A time: the record's time.

Text values are not readings, so they are left out.

The resource: service.name is the device's name, unless the record carries a service.name of its own. The record's resource attributes come along, as they do for log records.

Delivery counts: a record counts once, as delivered or as dropped, however many data points it became.

Devices whose readings go as metrics. In the names, <type> is the device's type as its YAML names it, such as opcua.

Devices	Metric names	Kinds
Modbus, OPC UA, BACnet, Siemens S7, EtherNet/IP, KNX, ctrlX CORE, DNP3, IEC 104	<type>.<table>.<field>	gauges
Prometheus	the sample's own name	counters, the buckets, sums and counts of histograms, and the sums and counts of summaries, as sums; the rest as gauges
StatsD	the series' own name; a timer's statistics as <series>.<statistic>	counters as delta sums; the rest as gauges
InfluxDB	<measurement>.<field>	gauges
SNMP	snmp.<path>	gauges (the device already turns counters into deltas and rates)
gNMI, gNMI dial-out, Cisco MDT, Juniper JTI	<type>.<table>.<path>; gNMI dial-out names start with gnmi.	readings under a counters container as sums; the rest as gauges
Redfish	redfish.<kind>.<field>	gauges, with units
Amazon CloudWatch	cloudwatch.<namespace>.<metric>.<statistic>	Sum and SampleCount as delta sums; the rest as gauges
Elasticsearch query	elasticsearch_query.<measurement>.<metric>	document counts and sums as delta sums; the rest as gauges
libvirt, VMware vSphere	<type>.<dataset>.<field>	running totals as sums, counts for each interval as delta sums, the rest as gauges
Website, HTTP check, TCP check, DNS check, ICMP check, TLS check	<type>.<field>, and <type>.up (1 or 0)	gauges
Linux and Windows hosts	the field's own path, such as system.cpu.user.norm.pct	running totals as sums; the rest as gauges

Known limits

• Modbus, Siemens S7, BACnet and InfluxDB records do not say whether a number is whole, so their whole numbers go as doubles. A true or false value goes as 1 or 0.
• DNP3 counters and IEC 104 integrated totals go as gauges, because their records do not say which point is a running total.
• BACnet and KNX points carry no unit. Their records state units in their own notation, which is not translated yet.
• A Prometheus sample keeps its own name, except that a colon becomes _, because OTLP metric names do not allow colons.
• Receivers built on Prometheus, such as Prometheus itself, Grafana Mimir and Grafana Cloud, refuse delta sums unless delta ingestion is turned on. With such a receiver, the readings that go as delta sums in the table above are lost, or the receiver refuses the request. Send them through a collector that turns delta sums into running totals first, such as the OpenTelemetry Collector's deltatocumulative processor or Grafana Alloy's otelcol.processor.deltatocumulative.

Counters as deltas

Some receivers take only deltas: the change of a counter over one interval. Dynatrace is one of them, and Datadog's and New Relic's OTLP intakes prefer deltas. With the Counters as deltas switch on (cumulative_to_delta: true), the target sends cumulative counters and histograms as deltas: the change since the previous point of the same series. The setting is off by default, and the target then sends counters as they come.

What is converted

• Each cumulative sum that only grows (a counter), and each cumulative histogram with explicit buckets.
• A counter's delta is the change in its value. A histogram's delta is the change in its count, its sum and each bucket.
• This includes the counters of metric devices when send_metrics is on. See Device readings as metrics.
• Counters and histograms are converted as the OpenTelemetry Collector's cumulativetodelta processor converts them. Unlike that processor, the target leaves exponential histograms unchanged and always holds back the first point of a series, as described below.

What goes out unchanged: gauges, sums that can also fall, sums that are already deltas, exponential histograms and summaries.

A series is one metric, by name, unit and kind, with one set of resource attributes, one scope (name, version and attributes) and one set of data point attributes.

How the deltas are worked out

• The first point of a series is its baseline. It has nothing to subtract from, so it is not sent, and the first interval of each series is missing. After each restart of Director, every series loses one interval again.
• A counter that starts over is a reset: its value drops, or its start time changes. A new start time counts only when both points have one. For a reset, the target sends the new value as the delta.
• An old point. A point that is not newer than the last point of its series is dropped. Its interval is already covered.

Delivery

• The target remembers a point only after the receiver has taken the request. If a request fails, the same deltas are sent again.
• A target with request limits, such as Dynatrace, splits a batch that is too large into parts. If one part is sent and another fails, the deltas of the part that was sent go a second time.

One worker. While the setting is on, the target uses one worker, even when queue.parallelism asks for more, and logs a warning. One worker keeps the points of a series in order.

One Director per series. Each Director keeps its own memory of the series. All the points of a series must reach the same Director. Do not balance one OTLP source across several Directors. A polled device meets this rule.

Limits. The target keeps up to 100,000 series, and forgets a series that sends nothing for an hour. It drops the points of new series over that limit and logs a warning. A record that loses all its points this way counts as dropped.

Configuration validation

The following rules are enforced at config load:

• url or (address + port) must be provided; providing neither is rejected.
• format must be protobuf or json.
• format: json combined with protocol: grpc is rejected — OTLP/gRPC is protobuf-only on the wire.
• batch_size must be >= 1.
• max_retries must be >= 0.
• tls.cert_name and tls.key_name must both be present or both be absent.

Examples

Basic HTTP

Forwarding logs to an OTLP/HTTP collector with default settings...

targets: - name: otel_collector type: otlp properties: url: "http://otel-collector.example.com:4318"

JSON Format over HTTP

Sending telemetry as JSON-encoded OTLP to an HTTP endpoint...

targets: - name: otel_json type: otlp properties: url: "http://otel-collector.example.com:4318" format: json path: /v1/logs

gRPC Spec-Compliant

Delivering logs over gRPC to a third-party collector using typed OTLP encoding...

targets: - name: grpc_collector type: otlp properties: address: "otel-collector.example.com" port: 4317 protocol: grpc grpc_passthrough: false

TLS-Secured

Sending logs over HTTPS with mutual TLS client certificate authentication...

targets: - name: otel_secure type: otlp properties: url: "https://otel-collector.example.com:4318" tls: status: true verify: true cert_name: "client-cert.pem" key_name: "client-key.pem" min_tls_version: "tls1.2"

Performance Tuning

High-volume configuration with large batches, retry logic, and auth header...

targets: - name: otel_high_volume type: otlp properties: url: "https://ingest.example.com:4318" format: protobuf compression: true batch_size: 5000 timeout: 60 max_retries: 5 retry_delay: 2 headers: Authorization: "Bearer ${OTLP_TOKEN}" X-Tenant-ID: "tenant-prod"

Device Readings as OTLP Metrics

Sending the readings of metric devices as OTLP metrics, while every other record goes as before...

targets: - name: otel_metrics type: otlp properties: url: "http://otel-collector.example.com:4318" send_metrics: true

Counters as Deltas for a Delta-Only Receiver

Sending counters as deltas, for a receiver that takes only deltas. Exponential histograms still go out unchanged, so the receiver must accept them...

targets: - name: otlp_deltas type: otlp properties: url: "https://otlp.example.com:4318" send_metrics: true cumulative_to_delta: true

Send to other observability vendors

Some observability vendors accept OTLP but have no target of their own. You can send to them with the OTLP target. Each recipe below gives the URL to enter, the Protocol and Format, the headers, the signals the vendor accepts, and any limit it sets.

What applies to every recipe:

• URL. Over HTTP, URL is the vendor's address with its scheme (https://) and without a path. Logs go to Logs HTTP path, which defaults to /v1/logs. Metrics and traces always go to /v1/metrics and /v1/traces, and you can't change those two paths. Over gRPC, URL is host:port.
• Headers. Headers are one set for every signal. Over gRPC they go as metadata.
• gRPC passthrough. Turn off gRPC passthrough for every vendor. When it is on, the Director sends bytes that only another VirtualMetric Director accepts.
• TLS over gRPC. gRPC uses TLS only when TLS encryption is on. Turn it on when you send to a vendor over gRPC. With it off, the target connects without TLS, and an endpoint that takes only TLS refuses the connection.
• Compression. Compression is on by default, and it uses gzip.
• Signals. Every record goes as a log record, except records that already are metrics or traces, such as those from the OTEL listener. The readings of metric devices go as metrics only when Readings as metrics is on, and it is off by default. See Device readings as metrics.
• One target sends every signal. The target can't turn off one signal. A vendor that refuses a signal makes the requests for that signal fail, so route to its target only the records it takes.
• Size. Batch size is 1000 events by default. The OTLP target sets no limit on the size of a request, so lower Batch size for a vendor that limits it.

Axiom

Axiom takes logs, traces and metrics over HTTP. See Axiom's OpenTelemetry guide.

• URL: https://<your Axiom domain>. Protocol: HTTP. Format: protobuf or json.
• Headers: Authorization: Bearer <API token>, plus the dataset: X-Axiom-Dataset: <dataset> for logs and traces, or x-axiom-metrics-dataset: <dataset> for metrics.

Axiom takes the dataset for metrics in a header of its own, and the OTLP target sends the same headers with every signal. So use two targets:

• one for logs and traces, with X-Axiom-Dataset;
• one for metrics, with x-axiom-metrics-dataset and Readings as metrics on.

Route to each target only the records that go as its signals:

• The readings of metric devices go as metrics, so they belong on the metrics target. A metric device's record without readings still goes as a log record, and the metrics target sends it without X-Axiom-Dataset, which Axiom needs for logs.
• Records from the OTEL listener keep their own signal. The listener's metrics belong on the metrics target, and its logs and traces on the other one.

Better Stack

Better Stack takes logs, metrics and traces. See Better Stack's OpenTelemetry guide.

• URL: https://<ingesting host>, the ingesting host of your Better Stack source. Protocol: HTTP. Format: protobuf.
• Better Stack also takes gRPC, at <ingesting host>:443.
• Headers: Authorization: Bearer <source token>.
• Better Stack takes gzip, so Compression can stay on.

SolarWinds Observability

See SolarWinds' endpoints and OpenTelemetry pages.

• URL: https://otel.collector.<data center>.cloud.solarwinds.com, where the data center is na-01, na-02, eu-01 or ap-01. Protocol: HTTP. Format: protobuf.
• Port 443 of the same host also takes gRPC.
• Headers: authorization: Bearer <API token>.
• Signals: SolarWinds names logs and metrics.

IBM Instana

See Instana's backend endpoints.

• Instana names its SaaS hosts after your region, such as blue or orange, and lists two forms. otlp-<region>-saas.instana.io takes gRPC on port 4317 and HTTP on port 4318. otlp-grpc-<region>-saas.instana.io and otlp-http-<region>-saas.instana.io take gRPC and HTTP on port 443. Use the host that Instana gives for your region.
• Over HTTP: URL https://otlp-<region>-saas.instana.io:4318 or https://otlp-http-<region>-saas.instana.io, Protocol HTTP, Format protobuf.
• Over gRPC: URL otlp-<region>-saas.instana.io:4317 or otlp-grpc-<region>-saas.instana.io:443, Protocol gRPC.
• Headers: x-instana-key: <agent key>, and x-instana-host: <host> when the data carries no host ID.
• Before you send logs, check that your Instana takes OTLP logs. If it refuses them, every log request fails.

Logz.io

This recipe sends logs only. See Logz.io's OTLP guide and its regions.

• URL: https://otlp-listener.logz.io, or the listener of your account's region: https://otlp-listener-au.logz.io, https://otlp-listener-ca.logz.io, https://otlp-listener-eu.logz.io or https://otlp-listener-uk.logz.io.
• Protocol: HTTP. Format: protobuf. Logs HTTP path: /v1/logs.
• Headers: Authorization: Bearer <logs token>. Logz.io's example also sends user-agent: logzio-protobuf-logs.
• Keep Readings as metrics off, and route only log records to this target.

SigNoz Cloud

SigNoz takes logs, metrics and traces. See SigNoz's ingestion overview.

• URL: https://ingest.<region>.signoz.cloud:443 over HTTP, or ingest.<region>.signoz.cloud:443 over gRPC. Your region is under Settings > Ingestion in SigNoz.
• Headers: signoz-ingestion-key: <key>. The older authorization: Bearer header is deprecated.
• Every SigNoz endpoint takes gzip, so Compression can stay on.
• A request can be at most 16 MB. Lower Batch size if requests come close to that.

Troubleshooting

This section covers the errors you are most likely to see with the otlp target, what causes each one, and how to fix it. Most of them come from the receiver rather than from Director, so the status code or the text after desc = is the receiver's own verdict on what you sent.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause. Lines that begin with [OTLP] also name the URL or the gRPC method that failed, which tells you exactly which endpoint to check.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

What the collector needs

This target has no identity model of its own. Everything it depends on lives on the receiver. Match a failure against this table before changing anything else.

The receiver must	When it applies	What you see when it is missing
Listen for the configured protocol on the port you set	Always	HTTP: connect: connection refused. gRPC: code = Unavailable with a dial error, or code = Unimplemented desc = unknown service ... when the port serves something other than OTLP
Serve the signal being sent	Always. With send_metrics: true the receiver must serve metrics as well as logs	HTTP status 404. gRPC code = Unimplemented desc = unknown service .... A receiver that takes logs only answers status 404 on /v1/metrics, or code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService over gRPC. Turn on metrics on the receiver, or set send_metrics to false
Accept the credential in headers	When the receiver authenticates	HTTP status 401 or status 403. gRPC code = Unauthenticated or code = PermissionDenied
Accept gzip-compressed payloads	compression: true, which is the default	gRPC code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding "gzip". Over HTTP a receiver typically answers status 415 or status 400
Accept application/json bodies	format: json	Over HTTP a receiver typically answers status 415 or status 400
Accept the size of one batch	Always	HTTP status 413. gRPC code = ResourceExhausted desc = grpc: received message larger than max (...)

There is no size limit on the Director side, so an oversized batch is only ever refused by the receiver. Lower batch_size when you see status 413 or ResourceExhausted.

"status 401" or "status 403" over HTTP

[Error] [director] [target-<target id>] [otel_high_volume] [OTLP] http raw send-error url=https://collector.example.com:4318/v1/logs file=<payload> target=otel_high_volume msglen=48213 err=status 401
[Error] [director] [target-<target id>] [otel_high_volume] Sender worker 3 execute() failed for <payload>: target broken: status 401

Cause: the receiver refused the request. 401 means it found no usable credential, 403 means it found one and would not accept it. The only credentials this target sends are the entries under headers, plus a client certificate when tls.cert_name and tls.key_name are set.

Fix: set the header the receiver expects, such as Authorization: "Bearer ${OTLP_TOKEN}" or a vendor API-key header. Two mistakes produce an empty header set with no error of their own, so the 401 is all you get:

• headers must be written as a YAML mapping. Written as a list of - key: value items it is read as empty, and no headers are sent at all.
• A ${...} or $secret{...} reference that resolves to nothing yields an empty header value.

Over gRPC the same problem arrives as rpc error: code = Unauthenticated desc = ... or code = PermissionDenied desc = ..., with wording the receiver chooses. Header names are lower-cased when they are sent as gRPC metadata, which matters for a receiver that compares them case-sensitively.

The payload is retried until the credential is accepted, and the queue grows in the meantime. Retries continue while Director runs and while the records are inside the queue retention limit, which is queue.limit seconds, 48 hours by default. With the In-memory storage tier they do not survive a Director restart.

"status 404" over HTTP, or the signal path appears twice

[Error] [director] [target-<target id>] [otel_collector] [OTLP] send-failed url=http://collector.example.com:4318/v1/logs/v1/logs file=<payload> target=otel_collector batch=1000 attempts=1 budget=4 lastStatus=404 lastErr=status 404

Cause: the request went to a route the receiver does not serve. Read the url= value in the log line before anything else. The target appends path to the endpoint you configured, so a url that already ends in /v1/logs produces /v1/logs/v1/logs, exactly as above. The other causes are a receiver that publishes OTLP on a different route, and a receiver that has the signal disabled.

Fix: set url to the base address only, for example http://collector.example.com:4318, and let path supply the route. When the receiver uses a non-standard route for logs, put that route in path and still keep url at the base. Retried until fixed, and nothing is dropped while you correct it.

"connection refused" or "no such host"

[Error] [director] [target-<target id>] [otel_collector] [OTLP] send-error url=http://collector.example.com:4318/v1/logs file=<payload> target=otel_collector batch=1000 attempt=0 err=Post "http://collector.example.com:4318/v1/logs": dial tcp <collector ip>:4318: connect: connection refused
[Error] [director] [target-<target id>] [otel_collector] Sender worker 5 execute() failed for <payload>: target broken: failed to finalize target cache: failed to send after 4 of 4 attempts: Post "...": connect: connection refused

Cause: nothing is listening where the target points, or the route to it is blocked. Starting the target opens no connection, so a wrong host or port never shows up as a startup failure, and the connection status stays green until the first delivery.

Fix: confirm the receiver is up and the port is reachable from the Director host. 4318 is the convention for OTLP/HTTP and 4317 for OTLP/gRPC, and swapping the two is the most common cause. For HTTP, url must carry a scheme: a value such as collector.example.com:4318 is accepted when the configuration loads and then fails on every send with unsupported protocol scheme. Check HTTP_PROXY, HTTPS_PROXY and NO_PROXY on the Director service if the traffic should not go through a proxy.

Over gRPC the same causes arrive as rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: ...". On that transport the endpoint is a host and port with no scheme, so an http:// prefix in url becomes part of the name being looked up, and the failure is typically reported as a name resolver error instead. Use address and port, or a url of the form collector.example.com:4317.

Retried until fixed. Data stays queued and is delivered once the receiver is reachable.

"x509: certificate signed by unknown authority" or "certificate is valid for ..."

[Error] [director] [target-<target id>] [otel_secure] [OTLP] send-error url=https://collector.example.com:4318/v1/logs file=<payload> target=otel_secure batch=1000 attempt=0 err=Post "https://collector.example.com:4318/v1/logs": tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: Director does not trust the certificate the receiver presented.

• certificate signed by unknown authority: the issuing CA is not trusted. Note that tls.ca_name replaces the system trust store rather than adding to it, so a publicly issued certificate stops verifying the moment you point the target at a private CA.
• certificate is valid for ..., not ...: the name being connected to is not in the certificate. This is normal when you connect by IP address.
• certificate has expired or is not yet valid: renew it, or check the clock on the Director host.

Fix: set tls.ca_name to the CA that issued the receiver's certificate, and include the public CA bundle in it if the same Director also needs public endpoints. Set tls.server_name to a name the certificate carries when you connect by IP. Turning tls.verify off removes the check, and the protection with it.

A TLS setting that does not match the port looks different. Over HTTP, https:// against a plaintext port reports tls: first record does not look like a TLS handshake. Over gRPC, plaintext against a TLS port reports code = Unavailable with an error reading the server preface.

Certificate material is read when the target starts, so a bad file stops it before anything is sent:

[Error] [director] [target-<target id>] [otel_secure] Failed to reinitialize target "otel_secure" (attempt 1). Reason: failed to build TLS config: client certificate: decrypt PKCS#8 private key (wrong passphrase?): ...

Other reasons in that position name the material that could not be used, for example ca_name "..." contains no valid PEM certificate(s). PEM files must sit under the service root, or be supplied inline or through an environment or secret reference. One setting fails quietly: an unrecognized tls.min_tls_version falls back to tls1.2 with no warning, so use tls1.0 through tls1.3 exactly.

"Unimplemented ... unknown service ..." over gRPC

[Error] [director] [target-<target id>] [grpc_collector] [OTLP] grpc raw send-error method=/opentelemetry.proto.collector.logs.v1.LogsService/Export file=<payload> target=grpc_collector msglen=48213 err=rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.logs.v1.LogsService

Cause: the connection succeeded, but the server on that port does not offer the OTLP service named after method=. In practice the port is wrong. Pointing a protocol: grpc target at 4318, the OTLP/HTTP port, is the usual way to produce this. The other cause is a receiver that has the signal named in method= turned off.

Fix: set port to the receiver's OTLP/gRPC port, conventionally 4317, or enable that signal on the receiver. A close variant, desc = unknown method Export for service ..., means the service is there but the method is not, which points at a server that is not an OTLP receiver at all.

Retried until fixed, so the queue grows until the port is corrected. Nothing is dropped.

"Decompressor is not installed for grpc-encoding ..."

[Error] [director] [target-<target id>] [grpc_collector] [OTLP] grpc raw send-error method=/opentelemetry.proto.collector.logs.v1.LogsService/Export file=<payload> target=grpc_collector msglen=48213 err=rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding "gzip"

Cause: compression defaults to true, so every call is gzipped and asks the receiver to decompress it. This receiver has no gzip decompressor enabled. The status code is Unimplemented, the same as the previous entry, so read the text after desc = to tell the two apart.

Fix: set compression: false on the target, or enable gzip on the receiver. Prefer enabling it on the receiver, since compression is the cheaper of the two on a busy link.

Over HTTP there is no dedicated error for the same situation. A receiver that will not take a gzipped body, or the application/json body produced by format: json, typically answers status 415 or status 400. When those appear and the endpoint is otherwise correct, try compression: false and format: protobuf in turn. Do not set Content-Type or Content-Encoding under headers to work around it. A custom value there replaces the one the target computed, and the receiver then sees a body that does not match its declared type.

Retried until fixed.

"context deadline exceeded" or "code = DeadlineExceeded"

[Error] [director] [target-<target id>] [otel_high_volume] [OTLP] send-error url=https://collector.example.com:4318/v1/logs file=<payload> target=otel_high_volume batch=5000 attempt=0 err=Post "https://collector.example.com:4318/v1/logs": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Cause: the request did not finish within timeout, which defaults to 30 seconds. Every attempt gets the full timeout, so one flush can take several times that before it gives up. A large batch, a slow or throttled receiver, and a firewall that drops packets silently all look the same here.

Fix: raise timeout, or lower batch_size so each request carries less. If every attempt times out at the same point and the receiver logs nothing at all, treat the route as blocked rather than slow and check the path from the Director host to the receiver. status 429 in the same place means the receiver is throttling instead, which Director retries on its own.

Setting timeout: 0 does not remove the limit. A zero is read as the default, and the same is true of batch_size, max_retries and retry_delay. For a single attempt per batch set disable_retries: true, which never waits to retry, whatever max_retries holds.

The same rejection keeps coming back

Symptom: one payload fails with the same answer on every attempt, for example status 400, status 413 or status 422 over HTTP, or rpc error: code = InvalidArgument desc = ... over gRPC. The target reconnects between attempts, the delivered counter does not move, and the queue keeps growing. The connection status stays green, because the connection itself is fine.

Cause: no answer from a receiver is treated as final. A payload the receiver will never accept is redelivered rather than discarded, so it returns at a steady pace until you change something.

Fix: change whatever the receiver is objecting to.

• status 413, or ResourceExhausted over gRPC: lower batch_size. The gRPC ceiling on the receiving side is 4 MB by default.
• status 400 or status 422: check path and format against what the receiver accepts, then look at the records the pipeline produces. Sending a logs body to /v1/metrics is a common cause, because path only sets the logs route. Device readings go to /v1/metrics by themselves when send_metrics is true, so do not set path to /v1/metrics for them.
• InvalidArgument from a third-party receiver: set grpc_passthrough: false. The default of true uses an encoding that only another Director reads.

warning

Nothing is lost, but nothing queued behind the refused payload moves either. Queued data is retained only for the configured queue limit, so correct the cause rather than waiting it out. See Persistent Storage.

The target is healthy but records are missing

Check these in order.

1. debug.dont_send_logs is enabled. Events are processed and then discarded before any request is made. With debug.status: true, startup logs Log sending is disabled for this target once, naming the target. With debug.status: false nothing is logged at all. Remove the flag.

2. The receiver accepted the call but refused some records. Look for [OTLP] partial_success in the log:

   [Warning] [director] [target-<target id>] [otel_collector] [OTLP] partial_success signal=logs target=otel_collector rejected=2 message="<receiver message>"

   The text after message= is the receiver's own explanation, usually an attribute or schema limit. Those records are counted as dropped and are never retried, because the call itself succeeded. Fix the records upstream, or raise the limit on the receiver.

3. The receiver's answer could not be read. Only the first 16 KB of a response is examined. A receiver that reports its rejections in a longer body, or in a shape this target does not recognize, leaves the whole batch counted as delivered. When the Director counters say everything arrived and the receiver disagrees, the receiver's own logs are the only place left to look.

4. A pipeline rewrote the OTLP envelope. Records from this Director's OTEL listener carry a JSON envelope. A processor that renames or removes its signal, resource, scope or logRecord keys makes the record unusable. It is skipped with a warning, and it is not counted as a drop:

   [Warning] [director] [target-<target id>] [otel_collector] Skipping malformed record: unsupported signal ""

   Leave those keys intact in any pipeline attached to this target.

5. The records went out as a different signal than you expect. Only records from the OTEL listener carry a signal of their own. Everything else, syslog and TCP sources included, is sent as logs whatever path says. With send_metrics: true, the readings of the devices listed under Device readings as metrics arrive among the receiver's metrics, not its logs. A device reading that arrives as a log record comes from a device that is not in that table, or from a record a pipeline has changed.

Counters sent as deltas

These apply when the Counters as deltas switch (cumulative_to_delta) is on. See Counters as deltas.

• The first interval of each counter is missing. This is expected: the first point is the baseline. The same happens once after each restart of Director.
• [OTLP] dropped N points of new series: cumulative_to_delta keeps at most 100000 series target=<name>: more than 100,000 series reached the target. Series that stop sending are forgotten after an hour. Reduce the attributes that make new series, or spread the devices over more targets.
• [OTLP] queue.parallelism N is set to 1: a target that keeps the state of its series runs one worker target=<name>: a warning that needs no action. The target ignores the larger worker count.
• [OTLP] dropped an OTEL request that could not be decoded: counters as deltas need its points target=<name>: Director could not read an OTLP request from an OpenTelemetry device, so it could not convert it. The request is counted as dropped.
• Counters jump or count twice. The points of one series reach more than one Director. Send each source to one Director.