Deduplicate

Analytics Flow Control

Synopsis

Drops duplicate events cluster-wide within a configurable time window by hashing a set of field values into a key and writing it atomically to a shared NATS JetStream KV bucket. The first event for a given key passes through; every subsequent event within the window is either dropped or tagged with _vmetric.dedup_seen: true.

Schema

- deduplicate:
    fields:
      - <ident>
    bucket: <string>
    ttl: <integer>
    method: <string>
    drop: <boolean>
    tag: <string>
    description: <text>
    if: <script>
    disabled: <boolean>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>

Configuration

Field	Required	Default	Description
fields	Y		Ordered list of log entry field paths used to compute the dedup key. Field order is significant — [src, dst] and [dst, src] produce different keys for the same values.
bucket	N	"default"	Key-namespace prefix within the shared persistent KV bucket. Keys are stored as dedup.<bucket>.<hash>. Use a distinct name per dedup scope to prevent unrelated pipelines from sharing key space.
ttl	N	3600	Dedup window in seconds, applied as a per-key TTL on each write. Keys expire individually after this duration; a new event with the same key then passes through and starts a new window. Changing ttl takes effect on keys written after the change — existing keys keep the window they were created with.
method	N	xxhash	Hash algorithm for key derivation. Accepted values: xxhash (default, fastest, 16-char hex), sha-1 (40-char hex), sha-256 (64-char hex). Use sha-1 or sha-256 for compliance scenarios requiring cryptographic collision resistance.
drop	N	true	When true, duplicate events are dropped. When false, duplicates pass through with _vmetric.dedup_seen: true set, allowing downstream processors to branch on the duplicate flag.
tag	N		Identifier for this processor instance.
description	N		Explanatory note.
if	N		Condition that must be true for the processor to run.
disabled	N	false	When true, the processor is skipped.
ignore_failure	N	false	Continue pipeline processing if the processor encounters an error (including NATS connectivity failures). Use for fail-open behavior during network partitions. This does not mean "keep filtering and ignore errors" — see the warning below.
ignore_missing	N	false	When true, missing fields are incorporated into the hash using a marker prefix rather than causing a failure. Two events both missing the same field share a key; events missing different fields do not collide.
on_failure	N		Processors to run when this processor fails.
on_success	N		Processors to run when this processor passes an event through.

warning

ignore_failure: true silently turns this processor off. The drop is signalled to the pipeline as an error value, and the ignore_failure check runs before that value is inspected — so a matched event is kept instead of dropped, no error is logged, and the pipeline reports success. The processor appears to run normally while filtering nothing.

Use it only if you genuinely want a pass-through. To tolerate real errors without losing the filtering, leave ignore_failure unset and handle the failure with on_failure instead.

Details

The deduplicate processor uses NATS JetStream KV Create semantics for atomic first-occurrence detection. Each event's key is computed from the values of the configured fields in declaration order. kv.Create succeeds only when the key does not already exist in the bucket — giving "first occurrence wins" semantics with a single round-trip per event and no client-side coordination. Concurrent events for the same key on different nodes race to create the entry; the loser gets ErrKeyExists and is treated as a duplicate.

Output fields on passing events: _vmetric.dedup_key is set to the computed hash string on the first occurrence, making the key visible to downstream processors.

Output fields on duplicates with drop: false: Both _vmetric.dedup_key and _vmetric.dedup_seen: true are set on the log entry before it continues downstream.

Key namespace and TTL: bucket is not a NATS bucket name — it is a key-namespace prefix inside a single shared persistent KV bucket, and keys are written as dedup.<bucket>.<hash>. That shared bucket has no bucket-level TTL and also holds state that must never expire, such as WEC bookmarks, which is why the dedup window is applied as a per-key TTL at write time instead.

Changing ttl therefore needs no administrative action: keys written after the change carry the new window, and existing keys expire on the window they were created with. All pipelines using the same bucket value share a key space.

warning

Do not delete the underlying KV bucket to change a dedup window. It is shared infrastructure — removing it destroys unrelated subsystem state, and it achieves nothing, because the TTL is per key rather than per bucket.

Field order in fields is significant: [src, dst] and [dst, src] produce different keys for the same field values. Field boundaries are demarcated internally so fields: [ab, c] and fields: [a, bc] cannot collide.

Network latency: Each event requires one NATS round-trip (~0.1–1 ms on localhost). A network partition surfaces as a processor error. Set ignore_failure: true for fail-open behavior where events pass through when the KV store is unreachable.

Relationship to suppress: The suppress processor provides equivalent in-process per-pipeline deduplication with no NATS round-trip. Use suppress for single-node or single-pipeline dedup; use deduplicate when dedup state must be shared across multiple pipelines or multiple Director nodes.

Examples

Basic Deduplication

Dropping repeated network events that share the same source, destination, and action...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert" } - deduplicate: fields: - src - dst - action ttl: 300
First occurrence passes with _vmetric.dedup_key set; identical events within 300 seconds are dropped...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert", "_vmetric": { "dedup_key": "a3f8c1d2e4b5f6a7" } }

Isolated Bucket per Pipeline Scope

Using a named bucket to keep dedup state isolated from other pipelines...	- deduplicate: fields: - host - event.id bucket: firewall-dedup ttl: 600
Only events processed by pipelines referencing firewall-dedup share this key space; other pipelines using the default bucket are unaffected...

Tag Duplicates Instead of Dropping

Passing duplicate events downstream tagged for conditional branching instead of dropping them...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert" } - deduplicate: fields: - src - dst - action ttl: 60 drop: false
Duplicate events pass through with _vmetric.dedup_seen: true set; a downstream if processor can branch on this flag...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert", "_vmetric": { "dedup_key": "a3f8c1d2e4b5f6a7", "dedup_seen": true } }

SHA-256 for Compliance Scenarios

Using SHA-256 for cryptographic collision resistance in high-cardinality key spaces...	- deduplicate: fields: - transaction.id - user.id - event.code bucket: compliance-dedup ttl: 3600 method: sha-256
Keys are 64-character hex strings derived via SHA-256; collision probability is negligible even across billions of distinct events...

Fail-Open on NATS Unavailability

Continuing pipeline processing when the NATS JetStream store is unreachable...	- deduplicate: fields: - src - dst - action ttl: 300 ignore_failure: true
When the KV bucket cannot be reached, the processor error is suppressed and the event passes through — dedup is best-effort rather than blocking...