Regex Extract

Parse Cribl Compatible

Synopsis

Extracts named fields from text using regular expressions with named capture groups.

Schema

- regex_extract:
    field: <ident>
    regex: <string>
    lang: <enum>
    target_field: <ident>
    additional_regex: <string[]>
    description: <text>
    field_name_format: <string>
    filter: <script>
    if: <script>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    max_exec: <integer>
    on_failure: <processor[]>
    on_success: <processor[]>
    overwrite_existing: <boolean>
    tag: <string>

Configuration

The following fields are used to define the processor:

Field	Required	Default	Description
`field`	N	`message`	Field containing text to extract from. Under `lang: js` the default is `_raw`
`regex`	Y	-	Regular expression with named capture groups. Under `lang: js`, parsed as a `/pattern/flags` literal, where the `g` flag scans for all matches rather than only the first
`lang`	N	-	Regex dialect. Empty selects the native engine; `js` (`javascript`, `ecmascript`) selects the ECMAScript-compatible engine used for Cribl pipelines
`filter`	N	-	Cribl-style JavaScript truthiness expression evaluated after `if`. A falsy result silently skips the processor. Distinct from `if`, which uses the native expression language
`target_field`	N	-	Field to store extracted values when no named groups are used. A bare `$Table` value instead switches on column routing — see below
`additional_regex`	N	-	Additional patterns to match after primary regex
`description`	N	-	Explanatory note
`field_name_format`	N	-	Template for formatting extracted field names (`${name}`)
`if`	N	-	Condition to run
`ignore_failure`	N	`false`	Continue on regex match failures
`ignore_missing`	N	`false`	Continue if source field doesn't exist
`max_exec`	N	`100`	Maximum number of matches to process
`on_failure`	N	-	See Handling Failures
`on_success`	N	-	See Handling Success
`overwrite_existing`	N	`false`	Replace existing fields instead of converting to array
`tag`	N	-	Identifier
`disabled`	N	`false`	When `true`, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details

The processor supports dynamic field naming using _NAME_ and _VALUE_ pattern pairs, field name formatting, and handling of multiple matches.

Golang regular expressions provied named capture groups to extract fields.

warning

Complex regular expressions on large texts may impact performance

Each named group becomes a field in the output. Special _NAME_n and _VALUE_n pairs allow dynamic field naming based on extracted content.

Routing Groups to a Virtual Table

Setting target_field to a bare $Table name — no dot, no field path — switches the processor into column-routing mode: every named group is written to the same-named column of that virtual table instead of to the event map.

- regex_extract:
    field: message
    regex: "(?<Object>[^ ]+) (?<Operation>[^ ]+)"
    target_field: $ASimAuditEventLogs

The prefix lives on the processor because a Go capture-group name cannot contain $. A group whose name is not a column of that table fails loudly rather than being dropped — group names are written by the pipeline author, so a mismatch is an authoring error, not wire noise.

In this mode target_field is not a container: the unnamed-group behaviour above does not apply, and nothing is written to a field called $Table.

note

The _NAME_n and _VALUE_n pairs must use matching indices, e.g. _NAME_0 with _VALUE_0

Multiple regex patterns, array conversion for duplicate fields, field name templating, and match count limiting are also supported.

Field names are automatically sanitized to remove invalid characters. However, the field_name_format should produce valid field names. Also, when overwrite_existing is set to false, duplicate matches are converted to arrays.

warning

Be careful with the max_exec setting when dealing with high-frequency matches.

Consider using ignore_failure when regex patterns might not match all inputs.

Regular Expression Semantics

Patterns compile with Go's RE2 engine. A pattern RE2 rejects — one using a lookaround or a backreference — is retried on a .NET-compatible backtracking engine, so both syntaxes are accepted. Every match made by that fallback engine is bounded by a 100ms timeout: a pattern that exceeds it fails the record with an error, and the rest of the pipeline continues.

Matching is unanchored and case-sensitive. A plain string matches anywhere in the field — anchor with ^ and $, and prefix the expression with (?i) for case-insensitive matching.

A pattern is rejected before it compiles when it:

exceeds 1000 characters,
uses more than 100 quantifiers (*, +, {n,m}),
nests groups more than 10 levels deep, or
takes longer than 100ms to compile.

These semantics describe the native engine. Under lang: js the pattern is parsed as a /pattern/flags literal and matched by the ECMAScript-compatible engine instead.

Examples

Basic

Extracting a numeric value with a static field name...	`{ "message": "metric1=23 metric2=42" }` `- regex_extract: field: message regex: "metric1=(?<metric1>\\d+)"`
creates a new field:	`{ "message": "metric1=23 metric2=42", "metric1": "23" }`

Complex Logs

Extracting multiple fields from structured log...

{
  "message": "462559d4a487[471]: 172.23.0.6 - - [26/Feb/2024:20:22:38 +0000] \"GET /path HTTP/1.1\" 200 87533"
}

- regex_extract:
    field: message
    regex: "^(?<container_id>[^\\s]+)\\[(?<process_id>\\d+)\\]:\\s+(?<remote_host>[^\\s]+)\\s+(?<remote_user>-)\\s+(?<auth_user>-)\\s+\\[(?<timestamp>[^\\]]+)\\]\\s+\"(?<request_method>\\w+)\\s+(?<requested_url>[^\\s]+)\\s+(?<http_version>[^\"]+)\"\\s+(?<status>\\d+)\\s+(?<bytes>.+)$"

yields HTTP log components:

{
  "message": "462559d4a487[471]: 172.23.0.6...",
  "container_id": "462559d4a487",
  "process_id": "471",
  "remote_host": "172.23.0.6",
  "request_method": "GET",
  "requested_url": "/path",
  "http_version": "HTTP/1.1",
  "status": "200",
  "bytes": "87533"
}

Dynamic Fields

Extracting key-value pairs as dynamic fields...

{
  "message": "name=\"John Doe\" age=30 email=\"john@example.com\""
}

- regex_extract:
    field: message
    regex: "(?<_NAME_0>[^\\s=]+)=\"?(?<_VALUE_0>(?<=\")[^\"]*|[^\\s\"]+)"

creates new fields based on the extracted names:

{
  "message": "name=\"John Doe\" age=30 email=\"john@example.com\"",
  "name": "John Doe",
  "age": "30",
  "email": "john@example.com"
}

Formatting

Formatting extracted field names...	`{ "message": "key=value" }` `- regex_extract: field: message regex: "(?<_NAME_0>[^=]+)=(?<_VALUE_0>.+)" field_name_format: "${name}_field"`
adds suffixes:	`{ "message": "key=value", "key_field": "value" }`

Multi-Match

Extracting multiple matches with array conversion...

{
  "message": "value=1 value=2 value=3 value=4 value=5"
}

- regex_extract:
    field: message
    regex: "value=(?<value>\\d+)"
    max_exec: 3

creates an array of up to max_exec matches:

{
  "message": "value=1 value=2 value=3 value=4 value=5",
  "value": ["1", "2", "3"]
}

Structured Data

Using multiple regexes with structured data...

{
  "message": "<134>1 2020-12-22T17:06:08Z CORP_INT_NLB CheckPoint 18160 - [action:\"Accept\"; conn_direction:\"Internal\"]"
}

- regex_extract:
    field: message
    regex: "\\[(?<__fields>.*?)\\]"
    additional_regex:
      - "(?<_NAME_0>[^:]+):\"(?<_VALUE_0>[^\"]+)\""

extracts nested key-value pairs. __fields is an ordinary named group, so it is written to the event too — remove it with Remove if it is not wanted:

{
  "message": "<134>1 2020-12-22T17:06:08Z...",
  "__fields": "action:\"Accept\"; conn_direction:\"Internal\"",
  "action": "Accept",
  "conn_direction": "Internal"
}

Synopsis​

Schema​

Configuration​

Details​

Routing Groups to a Virtual Table​

Regular Expression Semantics​

Examples​

Basic​

Complex Logs​

Dynamic Fields​

Formatting​

Multi-Match​

Structured Data​