Skip to main content

Snowflake

Synopsis​

Generates a unique Snowflake ID and stores it in a target field. The ID is generated from the node clock and needs no input field.

Schema​

- snowflake:
    target_field: <ident>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration​

The following fields are used to define the processor:

FieldRequiredDefaultDescription
target_fieldY-Field to store the generated Snowflake ID. There is no default: an empty value fails with target_field cannot be empty
descriptionN-Explanatory note
ifN-Condition to run
ignore_failureNfalseContinue processing if ID generation fails
ignore_missingNfalseSkip if field doesn't exist
on_failureN-Error handling processors
on_successN-Success handling processors
tagN-Identifier
disabledNfalseWhen true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details​

Generates a unique Snowflake ID and stores it in the specified target field. Snowflake IDs are 64-bit unique identifiers designed for distributed systems, combining a timestamp, worker ID, and sequence number to create globally unique values that are also roughly time-sortable.

note

Snowflake IDs are 64-bit integers typically represented as strings. They are time-sortable (IDs created later will generally be larger than IDs created earlier), which makes them useful for chronological sorting in databases.

The processor is useful for assigning unique identifiers to log entries, tracking events across distributed systems, and creating correlation IDs without coordination between nodes.

warning

While Snowflake IDs are designed to be unique across distributed systems, they should not be used for cryptographic purposes. If you need IDs for security-sensitive operations, consider using a dedicated cryptographic random generator.

Examples​

Basic​

Generating a basic Snowflake ID...

{
  "event": {
    "type": "login"
  }
}
- snowflake:
    target_field: event.id

adds a unique identifier:

{
  "event": {
    "type": "login",
    "id": "1637812502315610112"
  }
}

Nested Fields​

Generating ID in a nested structure...

{
  "transaction": {
    "amount": 29.99,
    "currency": "USD"
  }
}
- snowflake:
    target_field: transaction.metadata.id

creates nested fields automatically:

{
  "transaction": {
    "amount": 29.99,
    "currency": "USD",
    "metadata": {
      "id": "1637812635481661440"
    }
  }
}

Correlation​

Generating correlation ID for distributed tracing...

{
  "service": {
    "name": "payment-gateway"
  },
  "request": {
    "method": "POST",
    "path": "/api/v1/payments"
  }
}
- snowflake:
    target_field: trace.id
    if: "trace.id == nil"

adds tracking ID only if not already present:

{
  "service": {
    "name": "payment-gateway"
  },
  "request": {
    "method": "POST",
    "path": "/api/v1/payments"
  },
  "trace": {
    "id": "1637812728377626624"
  }
}

Multi-ID​

Generating multiple IDs for different purposes...

{
  "user": {
    "email": "user@example.com"
  },
  "session": {}
}
- snowflake:
    target_field: session.id
- snowflake:
    target_field: transaction.id
- set:
    field: event.metadata.generated_at
    value: "{{@timestamp}}"

creates multiple unique identifiers:

{
  "user": {
    "email": "user@example.com"
  },
  "session": {
    "id": "1637812912861540352"
  },
  "transaction": {
    "id": "1637812912907677696"
  },
  "event": {
    "metadata": {
      "generated_at": "2023-07-30T15:42:19.123Z"
    }
  }
}