Regex Extract

Parse Cribl Compatible

Synopsis

Extracts named fields from text using regular expressions with named capture groups.

Schema

- regex_extract:
    field: <ident>
    regex: <string>
    lang: <enum>
    target_field: <ident>
    additional_regex: <string[]>
    description: <text>
    field_name_format: <string>
    filter: <script>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    max_exec: <integer>
    on_failure: <processor[]>
    on_success: <processor[]>
    overwrite_existing: <boolean>
    tag: <string>

Configuration

The following fields are used to define the processor:

Field	Required	Default	Description
field	N	message	Field containing text to extract from. Under lang: js the default is _raw
regex	Y	-	Regular expression with named capture groups. Under lang: js, parsed as a /pattern/flags literal, where the g flag scans for all matches rather than only the first
lang	N	-	Regex dialect. Empty selects the native engine; js (javascript, ecmascript) selects the ECMAScript-compatible engine used for Cribl pipelines
filter	N	-	Cribl-style JavaScript truthiness expression evaluated after if. A falsy result silently skips the processor. Distinct from if, which uses the native expression language
target_field	N	-	Field to store extracted values when no named groups are used
additional_regex	N	-	Additional patterns to match after primary regex
description	N	-	Explanatory note
field_name_format	N	-	Template for formatting extracted field names (${name})
if	N	-	Condition to run
ignore_failure	N	false	Continue on regex match failures
ignore_missing	N	false	Continue if source field doesn't exist
max_exec	N	100	Maximum number of matches to process
on_failure	N	-	See Handling Failures
on_success	N	-	See Handling Success
overwrite_existing	N	false	Replace existing fields instead of converting to array
tag	N	-	Identifier

Details

The processor supports dynamic field naming using _NAME_ and _VALUE_ pattern pairs, field name formatting, and handling of multiple matches.

Golang regular expressions provied named capture groups to extract fields.

warning

Complex regular expressions on large texts may impact performance

Each named group becomes a field in the output. Special _NAME_n and _VALUE_n pairs allow dynamic field naming based on extracted content.

note

The _NAME_n and _VALUE_n pairs must use matching indices, e.g. _NAME_0 with _VALUE_0

Multiple regex patterns, array conversion for duplicate fields, field name templating, and match count limiting are also supported.

Field names are automatically sanitized to remove invalid characters. However, the field_name_format should produce valid field names. Also, when overwrite_existing is set to false, duplicate matches are converted to arrays.

warning

Be careful with the max_exec setting when dealing with high-frequency matches.

Consider using ignore_failure when regex patterns might not match all inputs.

Examples

Basic

Extracting a numeric value with a static field name...	{ "message": "metric1=23 metric2=42" } - regex_extract: field: message regex: "metric1=(?<metric1>\\d+)"
creates a new field:	{ "message": "metric1=23 metric2=42", "metric1": "23" }

Complex Logs

Extracting multiple fields from structured log...

{ "message": "462559d4a487[471]: 172.23.0.6 - - [26/Feb/2024:20:22:38 +0000] \"GET /path HTTP/1.1\" 200 87533" } - regex_extract: field: message regex: "^(?<container_id>[^\\s]+)\\[(?<process_id>\\d+)\\]:\\s+(?<remote_host>[^\\s]+)\\s+(?<remote_user>-)\\s+(?<auth_user>-)\\s+\\[(?<timestamp>[^\\]]+)\\]\\s+\"(?<request_method>\\w+)\\s+(?<requested_url>[^\\s]+)\\s+(?<http_version>[^\"]+)\"\\s+(?<status>\\d+)\\s+(?<bytes>.+)$"

yields HTTP log components:

{ "message": "462559d4a487[471]: 172.23.0.6...", "container_id": "462559d4a487", "process_id": "471", "remote_host": "172.23.0.6", "request_method": "GET", "requested_url": "/path", "http_version": "HTTP/1.1", "status": "200", "bytes": "87533" }

Dynamic Fields

Extracting key-value pairs as dynamic fields...	{ "message": "name=\"John Doe\" age=30 email=\"john@example.com\"" } - regex_extract: field: message regex: "(?<_NAME_0>[^\\s=]+)=\"?(?<_VALUE_0>(?<=\")[^\"]*|[^\\s\"]+)"
creates new fields based on the extracted names:	{ "message": "name=\"John Doe\" age=30 email=\"john@example.com\"", "name": "John Doe", "age": "30", "email": "john@example.com" }

Formatting

Formatting extracted field names...	{ "message": "key=value" } - regex_extract: field: message regex: "(?<_NAME_0>[^=]+)=(?<_VALUE_0>.+)" field_name_format: "${name}_field"
adds suffixes:	{ "message": "key=value", "key_field": "value" }

Multi-Match

Extracting multiple matches with array conversion...	{ "message": "value=1 value=2 value=3 value=4 value=5" } - regex_extract: field: message regex: "value=(?<value>\\d+)" max_exec: 3
creates an array of up to max_exec matches:	{ "message": "value=1 value=2 value=3 value=4 value=5", "value": ["1", "2", "3"] }

Structured Data

Using multiple regexes with structured data...	{ "message": "<134>1 2020-12-22T17:06:08Z CORP_INT_NLB CheckPoint 18160 - [action:\"Accept\"; conn_direction:\"Internal\"]" } - regex_extract: field: message regex: "\\[(?<__fields>.*?)\\]" additional_regex: - "(?<_NAME_0>[^:]+):\"(?<_VALUE_0>[^\"]+)\""
extracts nested key-value pairs:	{ "message": "<134>1 2020-12-22T17:06:08Z...", "action": "Accept", "conn_direction": "Internal" }