Snowflake

Synopsis

Generates a time-sortable Snowflake ID and stores it in a target field. The ID is made from the clock of the process that runs the pipeline and needs no input field. The IDs of one process never repeat. IDs from different Agents or Directors can, rarely, be the same: see Uniqueness.

Schema

- snowflake:
    target_field: <ident>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

The following fields are used to define the processor:

Field	Required	Default	Description
target_field	Y	-	Field to store the generated Snowflake ID. There is no default: an empty value fails with target_field cannot be empty
description	N	-	Explanatory note
if	N	-	Condition to run
ignore_failure	N	false	Continue processing if ID generation fails
ignore_missing	N	false	Skip if field doesn't exist
on_failure	N	-	Error handling processors
on_success	N	-	Success handling processors
tag	N	-	Identifier
disabled	N	false	When true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details

Generates a Snowflake ID and stores it in the specified target field. Each ID holds three numbers: the time in milliseconds, a count of the IDs made in that millisecond, and a number that stands for the process that made the ID. The time comes first, so the IDs are roughly time-sortable.

note

Snowflake IDs are positive 63-bit integers, so they fit a signed 64-bit integer. They are typically represented as strings. They are time-sortable (IDs created later will generally be larger than IDs created earlier), which makes them useful for chronological sorting in databases.

IDs made by the current version are larger than IDs made by earlier versions: a new ID is about a quarter larger than one made just before the upgrade. It is still larger than every earlier ID, so sorting by ID keeps working across the upgrade.

note

One process makes at most 512 IDs a millisecond, which is about half a million a second. A pipeline that stamps more records than that with this processor, in one Director or Agent process, waits for the next millisecond, so it slows down. Earlier versions made at most 25,600 IDs a second.

The processor is useful for assigning identifiers to log entries, tracking events, and creating correlation IDs. Before relying on the IDs as unique keys for records from more than one Agent or Director, see Uniqueness.

warning

Snowflake IDs carry the time they were made and the number of the process that made them, so they should not be used for cryptographic purposes. If you need IDs for security-sensitive operations, consider using a dedicated cryptographic random generator.

Uniqueness

The processor runs wherever a pipeline runs: in Directors, in Agents, and in the platform when it runs a pipeline test. Two processes that use the same process number can make the same ID. Whether two IDs can be the same depends on where they come from:

Where the two IDs come from	Can they be the same?
One process	No. Its IDs never repeat, and later IDs are larger
Two platform processes	No
A platform process and a Director or Agent	No
Two replicas or cluster nodes of one Director, once each holds its lease	No
A Director process that holds its lease, and an Agent or any other process without a lease	No
Two processes without a lease: two Agents, an Agent and a standalone Director, a Director replica in its first minute, and so on	Yes, rarely. They are unique only with high probability
Processes of two different Directors that both hold leases	Yes, rarely, as above

Leases. A Director whose replicas or cluster nodes share a store — a Director cluster, replicas on a shared bus, or serverless replicas coordinated by the platform — leases each of them a number of its own. A replica gets its number about a minute after it starts. A standalone Director, and every Agent, picks its number at random when it starts.

How rarely. Processes that pick their number at random pick one of 2,048. With 30 of them, two share a number about one time in five. Directors with leases pick from 1,024 numbers each, so with 30 such Directors, two share a number about one time in three. Two processes that share a number make the same ID whenever both stamp a record in the same millisecond with the same count, which happens often on a busy pipeline.

What a duplicate costs. A target that de-duplicates by document ID, such as Elasticsearch with this ID as _id, keeps one record and overwrites or drops the other. To de-duplicate records from more than one Agent or Director, combine the ID with a field that names the source, such as the host name or the device.

Examples

Basic

Generating a basic Snowflake ID...	{ "event": { "type": "login" } } - snowflake: target_field: event.id
adds a unique identifier:	{ "event": { "type": "login", "id": "1637812502315610112" } }

Nested Fields

Generating ID in a nested structure...	{ "transaction": { "amount": 29.99, "currency": "USD" } } - snowflake: target_field: transaction.metadata.id
creates nested fields automatically:	{ "transaction": { "amount": 29.99, "currency": "USD", "metadata": { "id": "1637812635481661440" } } }

Correlation

Generating correlation ID for distributed tracing...	{ "service": { "name": "payment-gateway" }, "request": { "method": "POST", "path": "/api/v1/payments" } } - snowflake: target_field: trace.id if: "trace.id == nil"
adds tracking ID only if not already present:	{ "service": { "name": "payment-gateway" }, "request": { "method": "POST", "path": "/api/v1/payments" }, "trace": { "id": "1637812728377626624" } }

Multi-ID

Generating multiple IDs for different purposes...	{ "user": { "email": "user@example.com" }, "session": {} } - snowflake: target_field: session.id - snowflake: target_field: transaction.id - set: field: event.metadata.generated_at value: "{{@timestamp}}"
creates multiple unique identifiers:	{ "user": { "email": "user@example.com" }, "session": { "id": "1637812912861540352" }, "transaction": { "id": "1637812912907677696" }, "event": { "metadata": { "generated_at": "2023-07-30T15:42:19.123Z" } } }