New Relic

Observability

Synopsis

Creates a New Relic target that sends logs, metrics and traces to New Relic over OTLP.

The target is the OTLP target with New Relic's sign-in and limits built in.

Schema

- name: <string>
  description: <string>
  type: newrelic
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    api_key: <string>
    send_metrics: <boolean>
    cumulative_to_delta: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be newrelic
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y	-	New Relic's OTLP endpoint for the account's region: https://otlp.nr-data.net (US), https://otlp.eu01.nr-data.net (EU), https://otlp.jp.nr-data.net (Japan) or https://gov-otlp.nr-data.net (US FedRAMP). It must use https. In the console: OTLP endpoint
api_key	Y	-	The account's license key. It can be plain text, an environment variable or a secret. In the console: License key
send_metrics	N	true	Sends the readings of metric devices as metrics. false sends them as log records. In the console: Readings as metrics
cumulative_to_delta	N	true	Sends counters as deltas, which New Relic prefers. While it is on, the target uses one worker. false sends counters unchanged. In the console: Counters as deltas

Performance

Field	Required	Default	Description
compression	N	true	Compresses each request with gzip. YAML only
timeout	N	30	How long a request may take, in seconds
batch_size	N	1000	The most events sent in one request
max_retries	N	3	How many times a failed request is retried
retry_delay	N	1	How long to wait before a retry, in seconds
disable_retries	N	false	true sends each batch once, as on the OTLP page. In the console: Disable retries

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf, and it checks New Relic's certificate against the system's trust store. So it ignores protocol, format, path, grpc_passthrough, address, port and tls.

It signs in itself, so it replaces an api-key header set under headers.

The Console Form

The form has one step, New Relic settings:

• OTLP endpoint;
• License key, with the Manual / Environment variable / Vault picker;
• Readings as metrics and Counters as deltas, both on by default;
• the Performance fields.

Details

Where to Find the Values

• The license key: in New Relic, open the API keys page and copy a key of the type INGEST - LICENSE.
• The endpoint: the one for the account's data center region. An EU account uses the EU endpoint, and a FedRAMP account uses the FedRAMP one.

How the Target Signs In

The target sends the license key in the api-key header of every request.

Counters as Deltas

New Relic's metrics are delta-based, so cumulative_to_delta is on by default for this target. It works as described in the OTLP page's Counters as deltas section:

• The first point of each series is kept and not sent, so a restart of Director loses one interval of each series.
• The target uses one worker while the setting is on.
• One Director must see all the points of a series.

Set cumulative_to_delta to false to send counters unchanged.

Size Limit

New Relic refuses a request over 1 MB. The target keeps each request at or under 1,000,000 bytes, and splits a batch that would be larger. A single record over the limit is dropped and counted as dropped, with a warning in the Director log.

Attribute Limits

New Relic removes the attributes of a record past its 128th, and shortens a string value that is longer than 4,095 characters. It reports each change as an NrIntegrationError event in the account.

Everything else works as on the OTLP page.

Examples

A US Account

A US account, with the license key from a secret...

targets: - name: newrelic type: newrelic properties: url: "https://otlp.nr-data.net" api_key: "$secret{newrelic-license-key}"

An EU Account That Receives Counters Unchanged

An EU account, with the license key from an environment variable, that receives counters unchanged...

targets: - name: newrelic_eu type: newrelic properties: url: "https://otlp.eu01.nr-data.net" api_key: "${NEW_RELIC_LICENSE_KEY}" cumulative_to_delta: false

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

• url is required for New Relic
• url for New Relic must be the https OTLP endpoint of the account's region, such as https://otlp.nr-data.net
• api_key is required for New Relic
• the api_key of the New Relic target resolved to nothing: check the environment variable or the secret it names: api_key is set, but its environment variable or secret is empty.
• failed to resolve the api_key of the New Relic target: ...: the environment variable or the secret could not be read.

Warnings While It Runs

These appear in the Director log:

• [OTLP] dropped a record of N bytes: New Relic refuses an OTLP logs request over 1000000 bytes, and the same for metrics and traces: a single record was too large to send.
• [OTLP] dropped N points of new series: cumulative_to_delta keeps at most 100000 series: the target already follows 100,000 series. See the OTLP page's Counters sent as deltas troubleshooting entry.

Errors from New Relic

The OTLP page's Troubleshooting entries apply. For New Relic in particular:

• status 403: the license key is wrong, or it belongs to an account in another region than the endpoint.
• status 429: the account is over a New Relic rate limit. The target retries.