Elastic Observability

Observability

Synopsis

Creates an Elastic Observability target that sends logs, metrics and traces to Elastic Observability over OTLP.

The target is the OTLP target with Elastic Cloud's sign-in and limits built in. For the other Elastic targets, see Elasticsearch and Elastic Security.

Schema

- name: <string>
  description: <string>
  type: elasticobservability
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    api_key: <string>
    namespace: <string>
    send_metrics: <boolean>
    cumulative_to_delta: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `elasticobservability`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Connection

Field	Required	Default	Description
`url`	Y	-	The managed OTLP endpoint that the Elastic Cloud console shows for the project or deployment, such as `https://my-project.ingest.us-east-1.aws.elastic.cloud`. It must use https. In the console: OTLP endpoint
`api_key`	Y	-	An Elastic API key that can write data. It can be plain text, an environment variable or a secret. In the console: API key
`namespace`	N	-	The namespace of the data streams that Elastic writes the data to. See Namespace. In the console: Namespace (Optional)
`send_metrics`	N	`true`	Sends the readings of metric devices as metrics. `false` sends them as log records. In the console: Readings as metrics
`cumulative_to_delta`	N	`false`	`true` sends counters as deltas, as described in the OTLP page's Counters as deltas section. YAML only

Performance

Field	Required	Default	Description
`compression`	N	`true`	Compresses each request with gzip. YAML only
`timeout`	N	`30`	How long a request may take, in seconds
`batch_size`	N	`1000`	The most events sent in one request
`max_retries`	N	`3`	How many times a failed request is retried
`retry_delay`	N	`1`	How long to wait before a retry, in seconds
`disable_retries`	N	`false`	`true` sends each batch once, as on the OTLP page. In the console: Disable retries

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf, and it checks Elastic Cloud's certificate against the system's trust store. So it ignores protocol, format, path, grpc_passthrough, address, port and tls.

It signs in itself, so it replaces an Authorization header set under headers.

The Console Form

The form has one step, Elastic Observability settings:

OTLP endpoint;
API key, with the Manual / Environment variable / Vault picker;
Namespace (Optional);
Readings as metrics, on by default;
the Performance fields.

Details

Where to Find the Values

The URL: the managed OTLP endpoint that the Elastic Cloud console shows for the project or deployment. Elastic Cloud offers it on Elastic Cloud Serverless and Elastic Cloud Hosted.
The API key: an Elastic API key that can write data. Elastic creates API keys in Kibana.

How the Target Signs In

The target sends the API key in the Authorization header of every request, as ApiKey <key>.

Namespace

Set namespace to write the data to the data streams of that namespace. The target removes spaces around the value, and adds it to the resource of all data as the attribute data_stream.namespace. When a resource already has that attribute, the target keeps the resource's own value. Without namespace, Elastic's defaults apply.

Readings as Metrics

send_metrics is on by default, so the target sends the readings of metric devices as metrics. These devices include Modbus, OPC UA, Prometheus and SNMP. false sends the readings as log records. The logs, metrics and traces that the OpenTelemetry device receives keep their kind. Any other record goes as a log record.

Counters

By default, the target sends counters unchanged, because Elastic takes both delta and cumulative counters. Set cumulative_to_delta to true to send them as deltas, as described in the OTLP page's Counters as deltas section.

Size Limit

Elastic's own examples keep a request to 4 MB. The target keeps each request at or under 4,000,000 bytes, and splits a batch that would be larger. A single record over the limit is dropped and counted as dropped, with a warning in the Director log.

Everything else works as on the OTLP page.

Examples

A Project's Managed OTLP Endpoint

The managed OTLP endpoint of a project, with the API key from a secret...

targets:
  - name: elasticobservability
    type: elasticobservability
    properties:
      url: "https://my-project.ingest.us-east-1.aws.elastic.cloud"
      api_key: "$secret{store=corp-vault,ref=elastic/otlp-api-key}"

Data in the Production Namespace

The same endpoint, with the data in the production namespace and the API key from an environment variable...

targets:
  - name: elasticobservability_production
    type: elasticobservability
    properties:
      url: "https://my-project.ingest.us-east-1.aws.elastic.cloud"
      api_key: "${ELASTIC_API_KEY}"
      namespace: "production"

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

url is required for Elastic Observability
url for Elastic Observability must be the https managed OTLP endpoint, such as https://my-project.ingest.us-east-1.aws.elastic.cloud
api_key is required for Elastic Observability
the api_key of the Elastic Observability target resolved to nothing: check the environment variable or the secret it names: api_key is set, but its environment variable or secret is empty.
failed to resolve the api_key of the Elastic Observability target: ...: the environment variable or the secret could not be read.

Warnings While It Runs

These appear in the Director log:

[OTLP] dropped a record of N bytes: Elastic Observability refuses an OTLP logs request over 4000000 bytes, and the same for metrics and traces: a single record was too large to send. When cumulative_to_delta is true, the warning for a metrics record reads dropped a record of up to N bytes.
[OTLP] dropped N points of new series: cumulative_to_delta keeps at most 100000 series: the target already follows 100,000 series. This warning can appear only when cumulative_to_delta is true. See the OTLP page's Counters sent as deltas troubleshooting entry.

The OTLP page's Troubleshooting entries apply too.

Synopsis​

Schema​

Configuration​

Connection​

Performance​

Scheduling​

Debug Options​

Settings That Do Not Apply​

The Console Form​

Details​

Where to Find the Values​

How the Target Signs In​

Namespace​

Readings as Metrics​

Counters​

Size Limit​

Examples​

A Project's Managed OTLP Endpoint​

Data in the Production Namespace​

Troubleshooting​

Configuration Errors​

Warnings While It Runs​