Cognite Data Fusion

Observability

Synopsis

Creates a Cognite Data Fusion target that sends metrics to a Cognite Data Fusion (CDF) project, as data points of time series. The console card says: "Send metrics to Cognite Data Fusion as time series."

note

The target follows the CDF API documentation.

Schema

- name: <string>
  description: <string>
  type: cognite
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    url: <string>
    project: <string>
    space: <string>
    token_url: <string>
    client_id: <string>
    client_secret: <string>
    scope: <string>
    time_series_id: <string>
    create_time_series: <boolean>
    compression: <boolean>
    timeout: <numeric>
    batch_size: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    headers:
      <key>: <value>
    tls:
      status: <boolean>
      verify: <boolean>
      cert_name: <string>
      key_name: <string>
      server_name: <string>
      min_tls_version: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be cognite
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
url	Y	-	The https base URL of the CDF cluster, such as https://westeurope-1.cognitedata.com. It takes no user, query or fragment. In the console: Base URL
project	Y	-	The name of the CDF project. It cannot contain /, ?, # or a space. The target sends data points to <url>/api/v1/projects/<project>/timeseries/data, and new time series to <url>/api/v1/projects/<project>/models/instances. In the console: Project name

Authentication

Field	Required	Default	Description
token_url	Y	-	The https address of the token service, such as https://login.microsoftonline.com/<tenant ID>/oauth2/v2.0/token for Microsoft Entra ID. It takes no user, query or fragment. In the console: Token URL
client_id	Y	-	The ID of the client that the target signs in as. In the console: Client ID
client_secret	Y	-	The secret of that client. In the console: Client secret
scope	N	the url followed by /.default	The scope that the target asks the token service for. The default is the form that Microsoft Entra ID uses, such as https://westeurope-1.cognitedata.com/.default. In the console: Token scope (Optional)

client_id and client_secret can each be plain text, an environment variable (${NAME}) or a secret ($secret{name}). The target does not start when one of them is set but resolves to nothing.

Time Series

Field	Required	Default	Description
space	Y	-	The data modeling space that holds the time series. The space must exist already, because the target does not create it. A space name has 1 to 43 characters. It starts with a letter, and the rest are letters, digits, - or _. A name of 43 characters ends with a letter or a digit. In the console: Data modeling space
time_series_id	N	{device}.{metric}	The template for the external ID of a time series. CDF calls the name of a time series its external ID, and the console calls this field External ID template, but the YAML field is time_series_id. See The ID Template. In the console: External ID template (Optional)
create_time_series	N	true	true lets the target create the time series that CDF does not have yet. false makes it drop the values of a time series that does not exist. See Creating Time Series. In the console: Automatic time series creation

Performance

Field	Required	Default	Description
compression	N	true	The target compresses each request with gzip and sends it with Content-Encoding: gzip. false turns that off. Not in the console form
timeout	N	30	As on the OTLP page. In the console: Timeout (Optional)
batch_size	N	1000	The most records in one batch. The target splits a batch into as many requests as its values need (see Requests and Answers). In the console: Batch size (Optional)
max_retries	N	3	As on the OTLP page. In the console: Maximum retries (Optional)
retry_delay	N	1	As on the OTLP page. In the console: Retry delay (Optional)
disable_retries	N	false	As on the OTLP page. In the console: Disable retries

Other Settings

• headers holds any other header that the target sends as it is. It cannot change Content-Type, Accept, Content-Encoding or Authorization. Otherwise it works as on the OTLP page.
• The tls settings work as on the OTLP page. The form has no TLS step: the tls settings are for a private link that uses a certificate of its own.
• cumulative_to_delta has no effect on this target.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

The debug options work as on the OTLP page.

The Console Form

The form has one step, Cognite Data Fusion settings, in four sections:

• Connection settings: Base URL and Project name, both required.
• Authentication: Client ID, Client secret (with the Manual / Environment variable / Vault picker, and masked on the review step) and Token URL, all required, and Token scope (Optional). With an empty scope, the target asks for the base URL followed by /.default.
• Time series: Data modeling space (required), External ID template (Optional, with {device}.{metric} as the hint text) and the switch Automatic time series creation, which is on to begin with. The form always saves the state of the switch.
• Performance: Timeout, Batch size, the switch Disable retries, Maximum retries and Retry delay, each Optional, with the defaults 30, 1000, off, 3 and 1. The two retry fields are hidden while Disable retries is on.

An Optional field that is left empty is not saved, so the default applies. The form has no field for compression, headers, tls or debug: set them in the YAML.

The form checks less than Director does: it refuses a slash in the project name, but Director also refuses ?, # and a space (see Configuration Errors). The form shows these messages next to a field:

• Base URL is required and Base URL must be an https URL
• Project name is required and Project name can't contain a slash (/)
• Client ID is required and Client secret is required
• Token URL is required and Token URL must be an https URL
• Data modeling space is required and Data modeling space must start with a letter and have up to 43 letters, digits, hyphens or underscores
• External ID template can use only {device}, {metric} and {attr.<key>} in braces
• Batch size must be a whole number of 1 or more, Maximum retries must be a whole number of 1 or more and Retry delay must be a whole number of at least 1 second

Details

What the Target Sends

Only metrics:

• the readings of metric devices;
• the metrics that the OpenTelemetry device receives.

The number points of gauges and sums become data points, each with a time (in milliseconds) and a number. Histograms, exponential histograms and summaries are not sent, because a time series holds numbers only.

A record that is not a metric, such as a log line, is counted as dropped, with a warning at most once a minute. So route only metrics to this target.

How the Data Reaches Cognite Data Fusion

The target sends data points to <url>/api/v1/projects/<project>/timeseries/data, and new time series to <url>/api/v1/projects/<project>/models/instances.

• Each series is the data points of one time series: a CogniteTimeSeries node of the core data model (space cdf_cdm, version v1), named by space and the external ID of the series.
• The space must exist already, because the target does not create it.
• Each request is compressed with gzip, unless compression is false.
• The target sends Content-Type: application/json and Authorization: Bearer <token> with each request.

Creating Time Series

When CDF answers a data request with a 400 that names time series it does not have, the target creates them and sends the values again.

• A new time series is numeric, and it is not a step series. Its name, description and source unit are the name, description and unit of the metric. When several metrics give one external ID, the first one names the time series.
• The create request only inserts. It sets existingVersion: 0 and skipOnVersionConflict, two fields of the CDF API, so it never changes a time series that exists already. One create request holds at most 1,000 time series.
• The target keeps no list of the time series that exist. It learns what is missing from the answers of CDF. So a time series that someone deletes in CDF is created again when the next value arrives.
• When CDF still reports a time series as missing after the target has created it, the target waits for retry_delay and sends the values again. It does this at most twice more. Then it drops those values and logs a warning. These two tries are separate from max_retries, and disable_retries does not turn them off.
• With create_time_series set to false, the target creates no time series. It drops the values of the time series that do not exist, logs a warning, and sends the rest.
• The client needs the right to write data points. When the target creates time series, the client also needs the right to write instances in the space.

Sign-in

The target signs in with OAuth 2.0 client credentials:

• It posts grant_type=client_credentials, client_id, client_secret and scope as a form to the token service, reads access_token and expires_in from the answer, and sends Authorization: Bearer <token> with each request.
• As on the AVEVA CONNECT page, it keeps the token until a minute before it expires. A token that lives less than two minutes is kept for half its lifetime. An answer without expires_in is taken as a lifetime of five minutes. The workers of the target share one token.
• A 401 from CDF drops the token. The target then asks for a new token and tries the request once more. A second 401 makes Director deliver the batch again later.
• A 400 or a 401 from the token service makes Director deliver the batch again later. The warning gives the status and, when the answer has an OAuth error code, that code. It never gives the client secret.

The ID Template

time_series_id is a template for the external ID of a time series:

• {device} is the name of the device, {metric} is the name of the metric, and {attr.<key>} is the value of the attribute with that key. As on the AVEVA CONNECT page, {attr.<key>} reads the attribute of the point, or else the attribute of its resource, such as service.instance.id or host.name.
• The default {device}.{metric} gives IDs such as router1.cpu.load.
• A template with an unknown placeholder, an empty {attr.} or a brace without its partner stops the target from starting. The error names the character where the problem is.
• An external ID can have any character. The target replaces none, except that a NUL character (the character with code 0) becomes _.
• An ID of more than 255 characters is shortened to 255 characters. The target keeps its first 238 characters, and adds - and the first 16 hex characters of the SHA-256 hash of the whole ID. It counts characters, not bytes. The same long ID always gives the same short ID, so a time series keeps its name from one run to the next, and two long IDs that differ only at the end get different short IDs.

One Series to One ID

As on the AVEVA CONNECT page:

• One time series holds the values of one series. A series is one metric of one instance of a device or service, from one instrumentation scope, with one set of point attributes. The target tells instances apart by service.name, service.namespace, service.instance.id and host.name.
• When two series give the same external ID, the target sends the first one and drops the values of the other, and logs a warning. Add {attr.service.instance.id} to the template to give each instance its own time series.
• The target keeps up to 100,000 external IDs, and a series that sends nothing for an hour loses its ID to the next series that asks for it. When 100,000 series hold an ID and none of them is stale, the target sends the values of a new ID without this check, so that no value is lost. Two series can then mix their values in that time series.

What Is Dropped Before Sending

The target drops a value that CDF would not take, before it sends anything:

• a value that is NaN or an infinity, or that is above 1e100 or below -1e100 (a point that records no value counts as NaN);
• a point without a timestamp, because the time of Director would differ when the batch is delivered again, and CDF would hold a second value;
• a value whose time is before 1900 or after 2099;
• a reading that lacks an attribute that the template names, in the point or in its resource;
• a value whose external ID is empty;
• a histogram, an exponential histogram or a summary, because a time series holds numbers only.

A warning with the number of dropped values and the reason is logged at most once a minute. These values are counted in the warnings and not as dropped records.

Requests and Answers

Each request has at most 100,000 data points and 10,000 time series, and about 1 MB of JSON before compression. The answer of CDF decides what happens next:

• A 2xx answer means that CDF took the request. CDF applies a request whole or not at all.
• A 400 that names missing time series makes the target create them (see Creating Time Series). With create_time_series set to false, the target drops the values of those time series instead.
• Any other 400, a 422, and any other 4xx answer that is not named here mean that CDF will never take the request. The target drops the values of the request and logs a warning with the status and the start of the message of CDF.
• A 401 makes the target try once more with a new token, as described under Sign-in. A second 401 makes Director deliver the batch again later.
• A 403, a 404 and a redirect that the target does not follow make Director deliver the batch again later, because waiting does not mend a permission or an address. The target follows only a 307 or a 308 redirect, which sends the same request to the new address.
• A 408, a 429 or a 5xx answer, and a failed connection, are tried again within the retry budget: the first try, and then at most max_retries more.
  • When the answer has a Retry-After header, the target waits for that time before the next try. The header holds a number of seconds or an HTTP date. The target waits for at most one minute, whatever the header asks for.
  • A header of 0, or a date that has already passed, means no wait: the target tries again at once.
  • A header that is neither a number of seconds nor a date, such as a negative number, is ignored.
  • Without a usable header, the target waits for retry_delay seconds.
  • When the budget is used up, Director delivers the batch again later.
• The records of a batch count as delivered once every request has been tried.
• A batch that fails part way is delivered again as a whole. CDF overwrites a value that has the same timestamp, so a value that was sent twice is stored once.

Workers

The workers of the target share one token, one list of external IDs and one count of dropped values. So the target can run several workers in parallel, and queue.parallelism applies as usual. The workers do not coordinate the creation of time series: each one learns from the answers of CDF, and that is safe because a create request only inserts.

Examples

Sign In with Microsoft Entra ID

Sending to a project with the default time series ID. The client ID and the client secret come from environment variables...

targets: - name: cognite_data_fusion type: cognite properties: url: "https://westeurope-1.cognitedata.com" project: "my-company-prod" space: "sensors" token_url: "https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/token" client_id: "${COGNITE_CLIENT_ID}" client_secret: "${COGNITE_CLIENT_SECRET}"

A Time Series per Site and Instance, Created in Advance

A time series for each site and instance, with the client secret from a secret store. The target creates no time series, so it drops the values of a time series that does not exist, with a warning...

targets: - name: cognite_plant type: cognite properties: url: "https://westeurope-1.cognitedata.com" project: "my-company-prod" space: "plant_a" token_url: "https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/token" client_id: "${COGNITE_CLIENT_ID}" client_secret: "$secret{cognite-client-secret}" time_series_id: "{attr.site}.{device}.{attr.service.instance.id}.{metric}" create_time_series: false

Troubleshooting

The hints below follow the CDF documentation and the notes of Director.

Configuration Errors

Director refuses a configuration it cannot use, with one of these messages:

• url is required for Cognite Data Fusion, for example https://westeurope-1.cognitedata.com
• url for Cognite Data Fusion must be an https URL with a host and no user, query or fragment, such as https://westeurope-1.cognitedata.com
• project is required for Cognite Data Fusion
• project for Cognite Data Fusion must not hold a slash, a question mark, a hash or a space
• space is required for Cognite Data Fusion
• space for Cognite Data Fusion must start with a letter, hold only letters, digits, underscores and hyphens, and have at most 43 characters: a space of 43 characters must also end with a letter or a digit.
• token_url is required for Cognite Data Fusion, for example https://login.microsoftonline.com/my-tenant/oauth2/v2.0/token
• token_url for Cognite Data Fusion must be an https URL with a host and no user, query or fragment, such as https://login.microsoftonline.com/my-tenant/oauth2/v2.0/token
• client_id is required for Cognite Data Fusion
• client_secret is required for Cognite Data Fusion
• create_time_series for Cognite Data Fusion must be true or false
• time_series_id template has an unknown placeholder {name} at character 10: use {device}, {metric} or {attr.<key>}
• time_series_id template has an empty attribute name in {attr.} at character N: write the name after attr., as in {attr.site}
• time_series_id template has a { at character N with no closing }: close every placeholder with }, and the messages for a } with no opening { and for a { inside a placeholder
• batch_size must be greater than 0, got N
• max_retries must be zero or greater, got N
• retry_delay must be zero or greater, got N
• both tls.cert_name and tls.key_name must be provided together or omitted together: the message appears when tls.status is true.
• failed to resolve the client_id of the Cognite Data Fusion target: ...: the environment variable or the secret could not be read. The same message names client_secret.
• the client_id of the Cognite Data Fusion target resolved to nothing: check the environment variable or the secret it names: the setting is set, but its environment variable or secret is empty. The same message names client_secret.

Warnings While It Runs

These appear in the Director log. Each line ends with target=<target name>, which the lines below leave out.

These come at most once a minute:

• [OTLP] dropped N records that are not metrics in the last minute: Cognite Data Fusion takes only metrics: the route sends log records or spans to this target.
• [OTLP] dropped N values in the last minute: Cognite Data Fusion <reason>, where the reason is one of:
  • takes no value that is missing, NaN, infinite or larger than 1e100
  • takes no value without a timestamp
  • takes no value with a timestamp before 1900 or after 2099
  • has no value for an attribute the time_series_id template names
  • cannot take an empty external ID
  • stores no histograms or summaries
  • already writes this external ID for another series, <external ID>: two series give the same external ID. Add an attribute to the template that tells them apart.
  • names a time series that does not exist, and create_time_series is off: set create_time_series to true, or create the time series first.
  • is still missing after it was created: CDF kept answering that the time series is missing after the target created it and tried twice more.
  • refused the values: status N, <message>: CDF refused the request for good, and the target dropped its values. The message is the start of the message of CDF, and it is left out when CDF gave none.
  • could not create the time series: status N, <message>: CDF refused to create the time series, and the target dropped their values. The message of CDF says why. Check that the space exists and that the client may write instances in it.
  • refused other values, which this warning does not list: more than 20 different reasons came up within the minute, and the rest are counted here.

These come on every try that fails, before the target tries again:

• [OTLP] sign-in-refused target=<target name> err=<message>: the token service refused the client. The target does not try again.
• [OTLP] sign-in-error target=<target name> attempt=N err=<message>: the token service could not be reached, or it answered with something that is not a token. The target tries again within the retry budget.
• [OTLP] send-error url=<url> file=<file> target=<target name> bytes=N attempt=N err=<message>: a request did not get an answer from CDF, such as a refused connection or a timeout. The target tries again within the retry budget.

Errors While It Runs

Director delivers the batch again once the cause is fixed.

• Cognite Data Fusion refused the insert request with status 401: CDF refused a new token as well as the old one. Check token_url, client_id, client_secret and scope.
• Cognite Data Fusion refused the insert request with status 403: the client has no right to do this. It needs the right to write data points, and, when the target creates time series, the right to write instances in the space.
• Cognite Data Fusion refused the insert request with status 404: the address is wrong. Check url and project.
• Cognite Data Fusion refused the insert request with status 301 (or a 302 or 303): the address redirects, and the target does not follow such a redirect. Use the final address of the cluster in url.
  • The same messages exist with create request in place of insert request, when the target creates time series.
  • The log line [OTLP] send-failed url=<url> file=<file> target=<target name> bytes=N attempts=N budget=N lastStatus=N lastErr=<message> has the same message, followed by the message of CDF and request id <id>. Cognite support asks for that request ID.
• Cognite Data Fusion: the identity service refused the sign-in with status 401, invalid_client, so check client_id and client_secret: sign-in refused: the token service refused the client. The part , invalid_client is the OAuth error code, and it is left out when the answer has none. The status can also be 400. The target does not try again.
• failed to send after N of M attempts: <last error>: the retry budget is used up, and M is max_retries plus one. The last error says why:
  • status 429, status 503 or another 408 or 5xx status: CDF is busy or down.
  • a connection error or a timeout: check the network path to CDF, and timeout.
  • the identity service at <token URL> answered status N: the token service answered with a status that is not a token or a refusal.
  • the identity service at <token URL> answered something other than a token: check token_url.
  • ask <token URL> for a token: ...: the token service could not be reached.