Google Cloud Observability (Metrics and Traces)

Google Cloud Observability

Synopsis

Creates a Google Cloud Observability (metrics and traces) target that sends metrics to Cloud Monitoring and traces to Cloud Trace over OTLP.

The target is the OTLP target with Google's sign-in and limits built in. To send logs to Google Cloud, use the Google Cloud Logging target.

Schema

- name: <string>
  description: <string>
  type: gcpobservability
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    project_id: <string>
    authentication:
      method: <auto|manual|secret>
    credentials: <string>
    url: <string>
    cumulative_to_delta: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `gcpobservability`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Connection

Field	Required	Default	Description
`project_id`	Y	-	The ID of the Google Cloud project that receives the data. The target sends it as the quota project, in the `x-goog-user-project` header. It also adds it to every resource as the attribute `gcp.project_id`, unless the resource already has one. In the console: Project ID
`url`	N	`https://telemetry.googleapis.com`	The host of Google's Telemetry API. It must be an https URL with only a host. To keep the data in a region, set a regional endpoint, `https://telemetry.<region>.rep.googleapis.com`. The target sends metrics to `/v1/metrics` and traces to `/v1/traces` under it. YAML only
`cumulative_to_delta`	N	`false`	Sends counters as deltas in place of cumulative totals. While it is on, the target uses one worker. YAML only

Authentication

Field	Required	Default	Description
`authentication.method`	N	`auto`	How the target signs in to Google Cloud: `auto`, `manual` or `secret`. In the console: Authentication method, with the choices Application Default Credentials (`auto`), Service account (inline JSON) (`manual`) and Service account (environment variable) (`secret`)
`credentials`	N	-	With `manual`: the service account key, as JSON. It can be plain text, an environment variable or a secret. With `secret`: the name of the environment variable on the Director's host that holds the key, such as `GCP_SERVICE_ACCOUNT_KEY`. A reference to an environment variable or a secret works there too. Required when `authentication.method` is `manual` or `secret`, and ignored with `auto`. In the console: Service account credentials

Performance

Field	Required	Default	Description
`compression`	N	`true`	Compresses each request with gzip. YAML only
`timeout`	N	`30`	How long a request may take, in seconds
`batch_size`	N	`1000`	The most events sent in one request
`max_retries`	N	`3`	How many times a failed request is retried
`retry_delay`	N	`1`	How long to wait before a retry, in seconds
`disable_retries`	N	`false`	`true` sends each batch once, as on the OTLP page. In the console: Disable retries

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

Settings That Do Not Apply

The target always sends OTLP over HTTP with protobuf. So it ignores protocol, format, path and tls.

It always sends the readings of metric devices as metrics, so it ignores send_metrics.

The Console Form

The form has one step, Google Cloud Observability (metrics and traces) settings:

a notice, This target sends only metrics and traces, which says to use the Google Cloud Logging target to send logs to Google Cloud;
Project ID;
under Authentication: Authentication method, which is Application Default Credentials (the default), Service account (inline JSON) or Service account (environment variable). Application Default Credentials asks for nothing more. The inline JSON choice shows Service account credentials as a text area for the JSON key. The environment variable choice shows Service account credentials as a field for the name of the environment variable on Director that holds the key;
the Performance fields.

The form sends the credentials only under a service account choice. It has no URL field, so a regional endpoint needs the YAML.

Details

Where to Find the Values

The project ID: the ID of the Google Cloud project that receives the data.
The service account key: the JSON key of a service account. You need it only with manual or secret. With manual, give the key itself. With secret, put the key in an environment variable on the Director's host, and give the name of that variable.
The URL: leave it out to use https://telemetry.googleapis.com, the host of Google's Telemetry API. To keep the data in a region, set it to a regional endpoint, https://telemetry.<region>.rep.googleapis.com. The URL must be an https URL with only a host.

How the Target Signs In

The target asks Google for an access token for the scope https://www.googleapis.com/auth/cloud-platform. It sends the token with each request, together with the project ID in the x-goog-user-project header.

With auto, the default, the target uses the Application Default Credentials: the credentials that Google's client libraries find on the host.
With manual, the target uses the service account key in credentials. The key can be plain text, an environment variable or a secret.
With secret, credentials holds the name of the environment variable on the Director's host that holds the key, such as GCP_SERVICE_ACCOUNT_KEY. The target reads the key from that variable when it starts. A reference to an environment variable or a secret works there too.
The target does not start when the key resolves to nothing, such as when the environment variable is not set. So it never signs in as the identity of the host when you asked for a key.

The identity needs the Cloud Telemetry Writer role on the project, and the Service Usage Consumer role on the quota project. The quota project is the project in project_id.

The Project on Resources

The target adds project_id to every resource as the attribute gcp.project_id, unless the resource already has one.

Metrics and Traces Only

The target sends metrics to /v1/metrics and traces to /v1/traces under the URL. It drops a log record, counts it as dropped, and logs a warning at most once a minute. Send logs with the Google Cloud Logging target.

Readings as Metrics

The target always sends the readings of metric devices as metrics. It ignores send_metrics.

Counters

The target sends counters as they are. Set cumulative_to_delta to true to send them as deltas. The conversion works as described in the OTLP page's Counters as deltas section:

The first point of a cumulative series is the baseline. The target keeps it and does not send it, so a restart of Director loses one interval of each series.
A counter that is already a delta goes out unchanged, with its first point.
While the setting is on, the target uses one worker.
One Director must see all the points of a series.

Size Limit

Google takes at most 200 metric points in one request. The target splits a larger request into smaller ones. A single record with more than 200 points is dropped and counted as dropped, with a warning in the Director log.

Names Google Accepts

Cloud Monitoring stores the metrics as Prometheus series. Google refuses a metric whose name does not match [a-zA-Z][a-zA-Z0-9_:./-]*, or whose label keys do not match [a-zA-Z_][a-zA-Z0-9_.]*.

Everything else works as on the OTLP page.

Examples

Application Default Credentials

A project, signed in with the credentials that Google's client libraries find on the host...

targets:
  - name: gcpobservability
    type: gcpobservability
    properties:
      project_id: "my-project"

A Service Account Key and a Regional Endpoint

A service account key from a secret, and a regional endpoint...

targets:
  - name: gcpobservability_regional
    type: gcpobservability
    properties:
      project_id: "my-project"
      authentication:
        method: manual
      credentials: "$secret{store=corp-vault,ref=gcp/telemetry-writer-key}"
      url: "https://telemetry.<region>.rep.googleapis.com"

A Key in an Environment Variable

A service account key in an environment variable on the Director's host...

targets:
  - name: gcpobservability_env
    type: gcpobservability
    properties:
      project_id: "my-project"
      authentication:
        method: secret
      credentials: "GCP_SERVICE_ACCOUNT_KEY"

Troubleshooting

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

project_id is required for Google Cloud Observability
authentication.method for Google Cloud Observability must be auto, manual or secret
credentials are required for Google Cloud Observability when authentication.method is manual or secret
url for Google Cloud Observability must be an https URL with only a host, such as https://telemetry.googleapis.com
the credentials of the Google Cloud Observability target resolved to nothing: check the environment variable or the secret it names: the setting is set, but its environment variable or secret is empty. With secret, the environment variable that credentials names is not set or is empty.
failed to read the credentials of the Google Cloud Observability target: ...: the service account key could not be read. The rest of the message gives the reason.
failed to find Google credentials on the host for the Google Cloud Observability target: ...: authentication.method is auto, and Google's client libraries found no credentials on the host. The rest of the message gives the reason.
failed to resolve credentials: ...: the environment variable or the secret that credentials refers to could not be read. The rest of the message gives the reason.

Warnings While It Runs

These appear in the Director log:

[OTLP] dropped N log records in the last minute: Google Cloud Observability takes only metrics and traces: the target dropped log records, because it sends only metrics and traces. Director logs this warning at most once a minute.
[OTLP] dropped a record of N data points: Google Cloud Observability refuses an OTLP metrics request of more than 200 data points: a single metrics record held more than 200 points.

With cumulative_to_delta set to true, Director logs this warning too:

[OTLP] dropped N points of new series: cumulative_to_delta keeps at most 100000 series: the target already follows 100,000 series.

Errors While It Runs

Director logs this message when it cannot sign a request:

could not get a Google access token: ...: the target could not get an access token. The rest of the message gives the reason.

The OTLP page's Troubleshooting entries apply too.

Synopsis​

Schema​

Configuration​

Connection​

Authentication​

Performance​

Scheduling​

Debug Options​

Settings That Do Not Apply​

The Console Form​

Details​

Where to Find the Values​

How the Target Signs In​

The Project on Resources​

Metrics and Traces Only​

Readings as Metrics​

Counters​

Size Limit​

Names Google Accepts​

Examples​

Application Default Credentials​

A Service Account Key and a Regional Endpoint​

A Key in an Environment Variable​

Troubleshooting​

Configuration Errors​

Warnings While It Runs​

Errors While It Runs​