Skip to main content

Prometheus Remote-Write

Observability

Synopsis​

Creates a Prometheus remote-write target that sends metrics to Prometheus, Grafana Mimir, VictoriaMetrics, Thanos or Cortex.

The target speaks Prometheus remote-write 1.0. Prometheus, Grafana Mimir, VictoriaMetrics, Thanos Receive and Cortex all take it. The Amazon Managed Service for Prometheus and Azure Monitor managed service for Prometheus targets work the same way, with their own sign-in. This page describes what the three have in common.

See also: to receive the metrics that Prometheus senders push to Director, use the Prometheus remote-write device.

Schema​

- name: <string>
  description: <string>
  type: prometheus
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    username: <string>
    password: <string>
    token: <string>
    headers: <map>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    tls:
      status: <boolean>
      verify: <boolean>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration​

The following fields are used to define the target:

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be prometheus
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

Connection​

FieldRequiredDefaultDescription
urlY-The receiver's remote-write endpoint, http or https. Prometheus and VictoriaMetrics take /api/v1/write, Grafana Mimir and Cortex /api/v1/push, and Thanos Receive /api/v1/receive. The target uses the URL as it is set, with its path and query. In the console: Remote-write URL

Authentication​

FieldRequiredDefaultDescription
usernameN-The username for basic authentication. In the console: Username, when Authentication is Basic
passwordN-The password for basic authentication. It needs username. In the console: Password
tokenN-A bearer token. Set it instead of username and password, not with them. In the console: Token, when Authentication is Bearer token
headersN-Other headers, sent as they are, such as X-Scope-OrgID for a Grafana Mimir tenant. YAML only

username, password and token can each be plain text, an environment variable or a secret.

Performance​

FieldRequiredDefaultDescription
timeoutN30How long a request may take, in seconds
batch_sizeN1000The most records in one batch. A batch goes out in requests of about 2,000 samples each (see Requests)
max_retriesN3How many times a failed request is retried
retry_delayN1How long to wait before a retry, in seconds
disable_retriesNfalsetrue sends each batch once, as on the OTLP page. In the console: Disable retries

TLS​

The tls settings work as on the OTLP page.

Scheduling​

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options​

The debug options work as on the OTLP page.

Settings That Do Not Apply​

The target always sends Prometheus remote-write over HTTP, compressed with Snappy. So it ignores protocol, format, path, compression, grpc_passthrough, address and port. It always sends device readings as metrics, so send_metrics has no effect. It always sends a count over an interval as a running total, so cumulative_to_delta has no effect.

The Console Form​

The form has two steps:

  • Prometheus remote-write settings: Remote-write URL; Authentication, which is None (the default), Basic or Bearer token. Basic shows Username and Password, and Bearer token shows Token, each with the Manual / Environment variable / Vault picker and each required. Then the Performance fields.
  • TLS settings: the OTLP form's TLS step.

The Authentication choice belongs to the form, not to the YAML: the form sends only the fields of the chosen option. A target's detail page opens on the option that its stored fields match.

Details​

What the Target Sends​

Only metrics:

  • the readings of metric devices, such as Modbus, OPC UA and SNMP;
  • the metrics that the OpenTelemetry device receives.

A record that is not a metric, such as a log record or a span, is not sent. It counts as dropped, and Director logs a warning at most once a minute. So route only metrics to this target.

Metric Types​

  • Gauges and counters go as they are. A sum that can fall goes as a gauge.
  • A histogram with explicit buckets becomes _bucket series with an le label, and _sum and _count series.
  • A summary becomes a series per quantile, with _sum and _count.
  • An exponential histogram needs native histograms, which remote-write 1.0 does not have. It is not sent, and it counts as dropped.

Names and Labels​

They follow Prometheus's own rules for OpenTelemetry metrics, so a query written for Prometheus's OTLP receiver works here too:

  • a counter's name ends in _total, and a unit becomes a suffix, such as _bytes;
  • the device's name becomes the job label, and the host name becomes the instance label;
  • the attributes of a reading become labels, with their names made safe for Prometheus;
  • a metric of the OpenTelemetry device with a named instrumentation scope also gets otel_scope_name, otel_scope_version and an otel_scope_<attribute> label for each scope attribute.

Counts over an Interval​

Some devices report a counter as the change over one interval, such as a StatsD count or a CloudWatch Sum. Remote-write has no type for that. So the target adds each change to a running total and sends the total, and rate() and increase() work as usual.

  • If a request fails, the next sample of the series still carries the full total. No change is lost, as long as the series sends a point again within an hour.
  • When Director restarts, the totals start again. Prometheus reads this as a counter reset, which rate() and increase() handle.
  • One Director must see all the points of a series. A device that one Director polls meets this.
  • The target keeps up to 100,000 such series. It forgets a series that sends nothing for an hour.
  • The target uses one worker, even when queue.parallelism asks for more. This keeps the points of a series in order.

Requests​

A request holds about 2,000 samples. A record never spans two requests, so a request can hold a few more.

Answers from the Receiver​

  • 400, 413 or 422: the receiver refused the data for good, for example samples that are out of order or too old. The records of that request count as dropped, and Director logs the start of the answer. The request is not sent again.
  • 429 or a 5xx: the target retries, within the retry settings.
  • Any other failure, such as 401, 403 or 404: Director delivers the batch again later. A credential or an address that someone fixes loses no data.

A Prometheus Server​

Prometheus takes remote-write only when it runs with the --web.enable-remote-write-receiver flag.

Not Supported​

Remote-write 2.0, native histograms, exemplars, metric metadata and target_info.

Examples​

Prometheus Server​

A Prometheus server, with no sign-in...

targets:
  - name: prometheus
    type: prometheus
    properties:
      url: "http://prometheus:9090/api/v1/write"

Grafana Mimir​

Grafana Mimir, with basic authentication and a tenant header...

targets:
  - name: mimir
    type: prometheus
    properties:
      url: "https://mimir.example.com/api/v1/push"
      username: "tenant-1"
      password: "$secret{mimir-password}"
      headers:
        X-Scope-OrgID: "tenant-1"

VictoriaMetrics​

VictoriaMetrics, with a bearer token from an environment variable...

targets:
  - name: victoriametrics
    type: prometheus
    properties:
      url: "https://victoriametrics.example.com/api/v1/write"
      token: "${VICTORIAMETRICS_TOKEN}"

Troubleshooting​

Configuration Errors​

Director refuses a configuration it cannot use, with one of these messages:

  • url is required for Prometheus remote-write
  • url for Prometheus remote-write must be an http or https URL with a host, such as https://prometheus.example.com/api/v1/write
  • set username and password, or token, not both
  • password needs a username: set username and password, or token
  • failed to resolve the password of the Prometheus target: ...: the environment variable or the secret that the setting names could not be read. The same message names the username or the token.
  • the token of the Prometheus target resolved to nothing: check the environment variable or the secret it names: the setting is set, but its environment variable or secret is empty. The target does not start, so it never sends without the sign-in that was asked for.

Warnings While It Runs​

  • [OTLP] dropped N records that are not metrics in the last minute: Prometheus remote-write takes only metrics: the route sends log records or spans to this target.
  • [OTLP] dropped N records: Prometheus remote-write refused the request for good, status 400: ...: the receiver refused the samples. The rest of the message is the receiver's reason, such as out-of-order samples, a sample too old, too many labels or a tenant limit.
  • [OTLP] dropped N points of new series: Prometheus remote-write keeps at most 100000 series: more than 100,000 series report counts over an interval.
  • status 401 or status 403: the credentials are wrong. The batches are delivered again once they are fixed.
  • status 404: remote write receiver needs to be enabled with --web.enable-remote-write-receiver: start Prometheus with that flag. Any other 404 means that the path of url is wrong.