Amazon CloudWatch (Metrics and Traces)

Amazon AWS Observability

Synopsis

Creates an Amazon CloudWatch (metrics and traces) target that sends metrics to Amazon CloudWatch and traces to AWS X-Ray over OTLP.

The target is the OTLP target with AWS's sign-in and limits built in. To send logs to CloudWatch, use the Amazon CloudWatch Logs target.

Schema

- name: <string>
  description: <string>
  type: amazoncloudwatchotlp
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    metrics_url: <string>
    traces_url: <string>
    region: <string>
    key: <string>
    secret: <string>
    session: <string>
    cumulative_to_delta: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `amazoncloudwatchotlp`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Connection

Field	Required	Default	Description
`metrics_url`	Y	-	The CloudWatch OTLP endpoint for metrics, such as `https://monitoring.us-east-1.amazonaws.com/v1/metrics`. It must be an https URL with a host. A URL with no path gets `/v1/metrics`. In the console: Metrics URL
`traces_url`	Y	-	The X-Ray OTLP endpoint for traces, such as `https://xray.us-east-1.amazonaws.com/v1/traces`. It must be an https URL with a host. A URL with no path gets `/v1/traces`. In the console: Traces URL
`region`	N	-	The AWS Region that the requests are signed for. The target reads the Region from the host of each URL. Set `region` when a URL does not name its Region, such as for a private endpoint. It then applies to both URLs, so both endpoints must be in that Region. The target refuses a `region` that differs from the Region that a host names. YAML only
`cumulative_to_delta`	N	`false`	Sends counters as deltas in place of cumulative totals. While it is on, the target uses one worker. YAML only

Authentication

Field	Required	Default	Description
`key`	N	-	The access key ID. Set it with `secret`, or leave both out. It can be plain text, an environment variable or a secret. In the console: Access key ID, when Credentials is Access key
`secret`	N	-	The secret access key. Set it with `key`, or leave both out. It can be plain text, an environment variable or a secret. In the console: Secret access key
`session`	N	-	The session token of temporary credentials. The target uses it only with `key` and `secret`. It can be plain text, an environment variable or a secret. In the console: Session token (Optional)

Performance

Field	Required	Default	Description
`compression`	N	`true`	Compresses each request with gzip. YAML only
`timeout`	N	`30`	How long a request may take, in seconds
`batch_size`	N	`1000`	The most events sent in one request
`max_retries`	N	`3`	How many times a failed request is retried
`retry_delay`	N	`1`	How long to wait before a retry, in seconds
`disable_retries`	N	`false`	`true` sends each batch once, as on the OTLP page. In the console: Disable retries

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf. So it ignores url, protocol, format, path and tls.

It always sends the readings of metric devices as metrics, so it ignores send_metrics.

The Console Form

The form has one step, Amazon CloudWatch (metrics and traces) settings:

a notice, This target sends only metrics and traces, which says to use the Amazon CloudWatch Logs target to send logs to Amazon CloudWatch;
under Connection settings: Metrics URL and Traces URL;
under Authentication: Credentials, which is Host's role (the default) or Access key. Host's role asks for nothing more, and the target finds the credentials itself (see How the Target Signs In). Access key shows Access key ID and Secret access key, which are both required, and Session token (Optional). Secret access key has the Manual / Environment variable / Vault picker;
the Performance fields.

The Credentials choice belongs to the form, not to the YAML. The form sends the access key fields only under Access key. A target's detail page opens on Access key when the target has an access key, and on Host's role when it has none. The form has no Region field, so a private endpoint whose host does not name its Region needs the YAML.

Details

Where to Find the Values

The URLs: the CloudWatch OTLP endpoint for metrics and the X-Ray OTLP endpoint for traces. Use the endpoints of one AWS Region, such as https://monitoring.us-east-1.amazonaws.com/v1/metrics and https://xray.us-east-1.amazonaws.com/v1/traces. Both must be https URLs. A URL with no path, or with only /, gets /v1/metrics or /v1/traces. A URL with any other path is used as it is set.
The Region: the target reads it from the host of each URL, such as monitoring.us-east-1.amazonaws.com. It also reads the host of a FIPS endpoint, such as monitoring-fips.us-east-1.amazonaws.com, and the hosts in China that end in amazonaws.com.cn.
The credentials: the access key ID, the secret access key and the session token of the AWS identity that the target signs in as. Leave the access key and the secret access key out to let the target find the credentials on the host.

How the Target Signs In

The target signs each request with AWS Signature Version 4. It signs a metrics request for the service monitoring, and a traces request for the service xray.

Without region, the target signs each request for the Region that the host of its URL names.
Set region when a URL does not name its Region, such as for a private endpoint. The setting then applies to both URLs, so both endpoints must be in that Region. The target refuses a region that differs from the Region that a host names.
To sign with an access key, set key and secret, and session for temporary credentials. Each can be plain text, an environment variable or a secret. The target reads them when it starts, and does not start when one of them is set but resolves to nothing.
If you do not set key and secret, the target finds credentials the same way the AWS SDK does. It checks the environment, then a shared profile, then the role of the host. The environment includes a web identity that environment variables name, as an EKS service account's role does, so such a role is used before a shared profile.

The identity needs permission to send metrics to CloudWatch and traces to X-Ray through their OTLP endpoints. AWS's setup guide for these endpoints gives it the CloudWatchAgentServerPolicy managed policy.

Traces Need Transaction Search

AWS asks you to turn on Transaction Search in CloudWatch before you send traces to the X-Ray OTLP endpoint. Turn it on before the target sends traces.

Session Tokens

The target reads a session token once, when it starts, and does not renew it when it expires. For temporary credentials, use the role of the host or a shared profile instead. A shared profile can assume a role, sign in with IAM Identity Center or run a credential process, and the AWS SDK renews these credentials. A profile that holds temporary keys copied into it is different: you must update it when the keys expire.

Metrics and Traces Only

The target sends metrics to metrics_url and traces to traces_url. It drops a log record, counts it as dropped, and logs a warning at most once a minute. Send logs with the Amazon CloudWatch Logs target.

Readings as Metrics

The target always sends the readings of metric devices as metrics. It ignores send_metrics.

Counters

The target sends counters as they are. Set cumulative_to_delta to true to send them as deltas. The conversion works as described in the OTLP page's Counters as deltas section:

The first point of a cumulative series is the baseline. The target keeps it and does not send it, so a restart of Director loses one interval of each series.
A counter that is already a delta goes out unchanged, with its first point.
While the setting is on, the target uses one worker.
One Director must see all the points of a series.

Size Limit

AWS refuses a metrics request over 5 MB before compression or with more than 1,000 data points. It refuses a traces request over 5 MB before compression. The target keeps each metrics request at or under 5,000,000 bytes and 1,000 data points, and each traces request at or under 5,000,000 bytes. It splits a batch that would be larger. A single record over a limit is dropped and counted as dropped, with a warning in the Director log.

AWS also refuses a traces request with more than 10,000 spans. The target does not count spans, so lower batch_size when your records each carry many spans.

Everything else works as on the OTLP page.

Examples

Endpoints in One Region, Signed with the Host's Credentials

Endpoints in one Region, signed with the credentials that the target finds on the host...

targets:
  - name: amazoncloudwatchotlp
    type: amazoncloudwatchotlp
    properties:
      metrics_url: "https://monitoring.us-east-1.amazonaws.com/v1/metrics"
      traces_url: "https://xray.us-east-1.amazonaws.com/v1/traces"

An Access Key

An access key, with both parts from a secret...

targets:
  - name: amazoncloudwatchotlp_key
    type: amazoncloudwatchotlp
    properties:
      metrics_url: "https://monitoring.us-east-1.amazonaws.com/v1/metrics"
      traces_url: "https://xray.us-east-1.amazonaws.com/v1/traces"
      key: "$secret{store=corp-vault,ref=aws/cloudwatch-access-key}"
      secret: "$secret{store=corp-vault,ref=aws/cloudwatch-secret-key}"

Private Endpoints

Private endpoints whose hosts do not name a Region, so region is set. It applies to both URLs...

targets:
  - name: amazoncloudwatchotlp_private
    type: amazoncloudwatchotlp
    properties:
      metrics_url: "https://<the private endpoint for metrics>/v1/metrics"
      traces_url: "https://<the private endpoint for traces>/v1/traces"
      region: "eu-west-1"

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

metrics_url is required for Amazon CloudWatch
traces_url is required for Amazon CloudWatch
metrics_url for Amazon CloudWatch must be an https URL with a host, such as https://monitoring.us-east-1.amazonaws.com/v1/metrics
traces_url for Amazon CloudWatch must be an https URL with a host, such as https://xray.us-east-1.amazonaws.com/v1/traces
region is required for Amazon CloudWatch: the metrics_url does not name its region, so set region, for example eu-west-1: the host of metrics_url names no Region, and region is not set. The same message names traces_url when its host is the one that names no Region.
region for Amazon CloudWatch differs from the region that the host of metrics_url names: leave region out, or set it to that region: region is set, and the host of metrics_url names another Region. The same message names traces_url when its host is the one that differs.
set both key and secret, or neither
the key and the secret of the Amazon CloudWatch target must both be set, or both be left out to use the credentials that the AWS SDK finds on the host
the key of the Amazon CloudWatch target resolved to nothing: check the environment variable or the secret it names: the setting is set, but its environment variable or secret is empty. The same message names secret or session.
failed to resolve the key of the Amazon CloudWatch target: ...: the environment variable or the secret could not be read. The same message names secret or session.
failed to load the AWS configuration: ...
no AWS credentials to sign the requests with: set key and secret, or give the host an AWS role

Warnings While It Runs

These appear in the Director log:

[OTLP] dropped N log records in the last minute: Amazon CloudWatch takes only metrics and traces: the target dropped log records, because it sends only metrics and traces. Director logs this warning at most once a minute.
[OTLP] dropped a record of N data points: Amazon CloudWatch refuses an OTLP metrics request of more than 1000 data points: a single metrics record held more data points than AWS takes in one request.
[OTLP] dropped a record of N bytes: Amazon CloudWatch refuses an OTLP metrics request over 5000000 bytes: a single metrics record was too large to send.
[OTLP] dropped a record of N bytes: Amazon CloudWatch refuses an OTLP traces request over 5000000 bytes: a single traces record was too large to send.

With cumulative_to_delta set to true, Director logs these warnings too:

[OTLP] dropped a record of up to N bytes: Amazon CloudWatch refuses an OTLP metrics request over 5000000 bytes: a single metrics record was too large to send. The target estimates the size in this case, so it logs this message in place of the metrics message above.
[OTLP] dropped N points of new series: cumulative_to_delta keeps at most 100000 series: the target already follows 100,000 series.

Errors While It Runs

Director logs one of these messages when it cannot sign a request:

could not get the AWS credentials to sign the request with: ...: the target could not get credentials. The rest of the message gives the reason.
could not sign the request: ...: signing the request failed. The rest of the message gives the reason.

The OTLP page's Troubleshooting entries apply too.

Synopsis​

Schema​

Configuration​

Connection​

Authentication​

Performance​

Scheduling​

Debug Options​

Settings That Do Not Apply​

The Console Form​

Details​

Where to Find the Values​

How the Target Signs In​

Traces Need Transaction Search​

Session Tokens​

Metrics and Traces Only​

Readings as Metrics​

Counters​

Size Limit​

Examples​

Endpoints in One Region, Signed with the Host's Credentials​

An Access Key​

Private Endpoints​

Troubleshooting​

Configuration Errors​

Warnings While It Runs​

Errors While It Runs​