Pattern

Parse

Synopsis

Extracts structured patterns from log messages, identifying key components and normalizing variable content.

Schema

- pattern:
    field: <ident>
    target_field: <ident>
    description: <text>
    if: <script>
    custom_patterns: <map[string]string>
    tokenize_all: <boolean>
    tokenizer: <string>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

The following fields are used to define the processor:

Field	Required	Default	Description
field	Y	-	Field containing the message to analyze
target_field	N	field	Field to store pattern information
description	N	-	Documentation note
custom_patterns	N	-	Map of custom regex patterns to use
tokenize_all	N	false	Use all built-in patterns for tokenization. Applies to the default tokenizer only
tokenizer	N	default	How the processor finds a message's pattern: default or search. See Tokenizers
if	N	-	Condition to run
ignore_failure	N	false	See Handling Failures
ignore_missing	N	false	If true, skip if field doesn't exist
on_failure	N	-	See Handling Failures
on_success	N	-	See Handling Success
tag	N	-	Identifier
disabled	N	false	When true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details

The processor extracts a normalized representation of log messages by first removing numeric or other identifiable values, and then replacing the standard patterns with tokens in order to generate a consistent pattern representation. It uses the following patterns:

EMAIL	Email address format
IP	IP address detection
NUMBER	Numeric values
PATH	File system paths
TIMESTAMP	Date and time formats
URL	Web URLs

For each processed message, the processor generates a pattern string, a unique hash, and an identifier. It limits the pattern to 100 words, and requires words to be at least 2 characters long.

warning

The processor may modify the input message to create a generalized pattern.

Long messages are truncated, and specific identifiers are replaced with generic tokens.

Tokenizers

tokenizer chooses how the processor finds a message's pattern:

default

The processor's own tokenizer, described above. Leaving the option out gives the same result

search

The processor writes the same template that the patterns() function of Search finds for the message on its own. Search masks numbers, IP addresses, hex values, ids, times, URLs and emails. A path stays one word, with its variable parts masked. Use this when you want the patterns stored at ingestion to match the ones a Search query finds over the same messages. A query over many messages also groups similar templates, replacing a word that varies among them with <*>. The processor reads one message at a time and doesn't do this, so a stored pattern can still have a word where the query shows <*>

• The default doesn't change. Switching an existing pipeline to search changes the patterns it writes, and so their hash and id values. Patterns stored before the switch won't match the new ones.
• custom_patterns work with both tokenizers. The processor applies them first.
• tokenize_all only affects the default tokenizer. The search tokenizer always masks its token types.
• The output is unchanged. The target field still gets pattern, hash and id.
• Any other value is refused with the error tokenizer "x" is not supported: use default or search.

Examples

Basic

Extracting a pattern from a log message...	{ "message": "2019-07-24 12:06:21,688 package.name [DEBUG] got 10 things in 3.1s" } - pattern: field: message target_field: log_pattern
creates a normalized one:	{ "message": "2019-07-24 12:06:21,688 package.name [DEBUG] got 10 things in 3.1s", "log_pattern": { "pattern": "package.name got things in", "hash": "...", "id": "..." } }

Custom

Adding a custom pattern for error codes...	{ "message": "Application error ERR1234: connection timeout" } - pattern: field: message target_field: log_pattern custom_patterns: ERROR_CODE: "ERR\\d{4}"
applies it:	{ "message": "Application error ERR1234: connection timeout", "log_pattern": { "pattern": "Application error <ERROR_CODE> connection timeout", "hash": "...", "id": "..." } }

Tokenizing

Using all built-in patterns...	{ "message": "User admin@example.com logged in from 192.168.1.100" } - pattern: field: message target_field: log_pattern tokenize_all: true
replaces all the known patterns:	{ "message": "User admin@example.com logged in from 192.168.1.100", "log_pattern": { "pattern": "User <EMAIL> logged in from <IP>", "hash": "...", "id": "..." } }

Search Tokenizer

Finding the pattern the way Search does...	{ "message": "2026-10-24T17:03:13.412Z ERROR checkout-service failed to charge card for order 77812: payment declined by issuer (code 51)" } - pattern: field: message tokenizer: search
writes the template Search's patterns() function finds for the message:	{ "message": { "pattern": "<TIME> ERROR checkout-service failed to charge card for order <NUM>: payment declined by issuer (code <NUM>)", "hash": "...", "id": "..." } }

For the same message, tokenize_all: true with the default tokenizer writes <TIMESTAMP> ERROR checkout-service failed to charge card for order <NUMBER> payment declined by issuer.

Error Handling

Handling non-string inputs...	{ "message": 12345 } - pattern: field: message target_field: log_pattern on_failure: - append: field: tags value: pattern_parse_error
adds an error tag and continues execution:	{ "message": 12345, "tags": ["pattern_parse_error"] }