Event Split Rules

Synopsis

Event split rules split one incoming message into separate logs. For example, an Azure Event Hubs message {"records":[...]} that holds 35 records becomes 35 logs. From that point on, every count shows 35 logs, including the device's own In count.

Azure sends Diagnostic Settings, Activity Log, Entra ID and Defender XDR data to Event Hubs in batches: each message holds many logs. Without rules, the device treats each message as one log, unless its split_mode splits it. In one Defender XDR capture, 13 messages held 259 logs: the console counted 13, and the Azure packs, which expect one log per record, could not map them. Event split rules make each record a log of its own before anything else reads it, so pipelines do not have to split them.

A device names the rule sets it uses in its event_split_rules property. These device types take it. The message devices split each message they receive, and the file devices split each file they read (see What Happens to a File):

Device	Type	Splits each
Event Hubs	eventhubs, in Kafka endpoint mode too	message
Azure Service Bus	servicebus	message
Kafka	kafka	message
Google Cloud Pub/Sub	pubsub	message
HTTP	http	message
Azure Blob Storage	azblob	file
Amazon S3	awss3	file
Google Cloud Storage	gcs	file

The replay devices (azblobreplay, awss3replay and gcsreplay) do not take it.

Configuration

event_split_rules is an ordered list of rule set keys:

properties:
  event_split_rules:
    - azure_records
    - json_lines

Each rule set is a list of rules. The Director tries the rules in order, across the rule sets in the order the device lists them, and the first rule that matches splits the message. When no rule matches, the device handles the message as it would without rules: it follows its split_mode, or keeps the message whole.

The Director splits each message as the device receives it, before anything else reads it. Event Hubs, Service Bus, Kafka and Pub/Sub still acknowledge a message only after all of its logs are delivered.

A Director reads the list from version 2.5.1. An older one ignores it, and the device works as it does without rules: a message device follows its split_mode, or keeps each message as one log, and a file device reads each file by its file extension. The console warns when one of the device's Directors runs an older version.

Where Rule Sets Come From

The platform's Library holds rule sets, and sends each Director the rule sets that its devices list in event_split_rules. The Director has no built-in rule sets of its own. Built-in Library rule sets, such as azure_records, reach it in the same way as any other rule set.

Three built-in rule sets make each row of a table a JSON log: csv_header (CSV with a header), w3c_extended (IIS / W3C extended logs) and zeek_logs (Zeek logs). csv_header has no condition, so it matches every message and every file the rules are checked against: list it after the rule sets that look for a format. See Library: Event Split Rules for every built-in rule set.

A standalone Director reads each rule set from config/library/event_split_rule/<key>.json, in the format the Library stores:

{
  "name": "Azure records",
  "inspectBytes": 256,
  "rules": [
    {
      "name": "Records",
      "enabled": true,
      "condition": "message =~ /^\\s*\\{\\s*\"records\"\\s*:\\s*\\[/",
      "type": "json_array",
      "settings": { "arrayField": "records" },
      "maxLogSize": 10485760
    }
  ]
}

Rule Sets

Field	Required	Default	Description
name	N	-	Rule set name
inspectBytes	N	256	How much of each message, from its start, a condition can read: 16 to 4096 bytes
rules	Y	-	1 to 50 rules. Each rule needs a name of its own

Rules

Field	Required	Default	Description
name	Y	-	Rule name, unique within the rule set
enabled	N	false	Only a rule with enabled: true is tried. A rule that is turned off is skipped
condition	N	-	Which messages the rule applies to (see Conditions). An empty condition matches every message
type	Y	-	How the rule splits a message (see Rule Types)
settings	N	-	The type's settings
maxLogSize	N	10485760 (10 MB)	The largest log the rule makes, from 1024 bytes to 10 MB. The Director cuts a longer log into pieces, so no data is lost, but a JSON log that was cut is no longer valid JSON

Conditions

A condition is written in the pipeline expression language, as a processor's if: is. It can read:

• message: the first inspectBytes bytes of the message, or of the file after it is decompressed
• source: the place the message came from, or the file's path (below)
• device.type and device.name

Device type	source
eventhubs	The event hub, in Kafka endpoint mode too
kafka	The topic
servicebus	The queue, or <topic>/<subscription>
pubsub	The subscription ID
http	The path the device listens on (url)
azblob	The file: <container>/<blob>
awss3, gcs	The file: <bucket>/<key>

Rule Types

Type	What it does	Settings
none	Keeps the whole message as one log	-
lines	Makes each line a log. A line that starts with a space or a tab stays with the line before it, as a stack trace's lines do. A blank line ends a log	-
separator	Splits at each occurrence of a fixed string. Blank pieces are dropped	separator
regex	Splits where a pattern matches. Each match marks the start or the end of a log. The pattern can use lookahead and lookbehind, and must not match an empty string	pattern, boundary (start or end)
jsonl	Makes each JSON value a log: JSON lines, or JSON objects written over several lines	-
json_array	Makes each element of a JSON array a log. The array is the whole message, or the field arrayField names, such as records or data.items	arrayField, copyFields, removeFields
csv	Makes each row of a CSV table a JSON log. The first row names the columns. In the console: CSV	delimiter, quoteChar, escapeChar
header	Makes each line of a table that has header lines a JSON log, as IIS, W3C and Zeek write their logs. The header line that lists the fields names the columns. In the console: File header	headerLineRegex, fieldsLineRegex, delimiterRegex, nullValue, cleanFields

A json_array rule can change the logs it makes:

• copyFields lists fields of the message to add to each log, under the field's last name. For example, meta.region is added as region. A log that already has a field of that name keeps its own. Across all of a message's logs, the copied fields can add at most the size of the message, or 1 MB for a message smaller than 1 MB. If the copied fields would add more, the Director does not split the message and keeps it as one log.
• removeFields lists fields to remove from each log, such as properties.raw.

A log keeps the bytes it arrived with, unless its rule copies or removes fields, or is a csv or header rule.

CSV

A csv rule reads the table's first row as the column names. That row makes no log. Every setting is optional, and each is one character:

Setting	Console label	Default	Description
delimiter	Delimiter	,	Goes between values. A tab works too: in the console, write it as \t
quoteChar	Quote character	"	Goes around a value that holds the delimiter or a line break. Must not be the delimiter
escapeChar	Escape character	"	Inside quotes, makes the next quote character part of the value. With the default, "" inside quotes is one ". Must not be the delimiter

File Header

A header rule is for tables whose header lines name the columns, such as IIS and W3C extended logs and Zeek logs. Lines that headerLineRegex matches are header lines and make no logs. The header line that fieldsLineRegex matches lists the column names, and a later one names the columns again from there on. Every other line is split into values where delimiterRegex matches. Every setting is optional:

Setting	Console label	Default	Description
headerLineRegex	Header line pattern	^#	Matches the header lines
fieldsLineRegex	Fields line pattern	^#[Ff]ields[:]?\s+(.*)	Matches the header line that lists the column names. Needs one group, in parentheses, around the list of names
delimiterRegex	Delimiter pattern	\s	Matches what separates two values: by default, any white space character. Must not match an empty string. Zeek logs need \t, since their values can hold spaces
nullValue	Null value	empty	A value equal to it is left out of the log, such as -. Empty keeps every value. At most 256 bytes
cleanFields	Clean field names	false	Each character of a column name that is not a letter or a digit becomes _, so id.orig_h becomes id_orig_h. A pipeline reads a dot in a field name as a path, so this keeps such names usable

Tables

A csv rule and a header rule make each row of a table a JSON object. Every value is a string, under its column's name, in the columns' order:

• A row with more values than the table has columns names the rest _extra_1, _extra_2 and so on.
• A row with fewer values leaves the last columns out.
• An empty column name becomes _column_N, where N is the column's number, counting from 1. A name that an earlier column already has becomes name_N, with N counting up from the column's number until the name is free.
• A row can have at most 10,000 values, the header included. The Director does not split a message with a wider row, and keeps the message whole, as one log.
• An empty line makes no log. A header rule also skips a line that holds only white space.
• The logs of a message can hold at most 8 times the message's size, or 1 MB for a smaller message. Each log repeats the column names, so a table of short values makes much larger logs. When the logs would hold more, the Director keeps the message whole, as one log.

Details

What Happens to a Message

Situation	Result
A rule matches and splits the message	Each piece is a log of its own, from the start. Every later count counts the logs
No rule matches	The device behaves as it does without rules. It follows its split_mode, or keeps the message whole
A rule matches but cannot split the message, for example a json_array rule given text that is not JSON	The message is kept whole, as one log. The Director logs a warning at most once a minute. Later rules are not tried
A log is longer than the rule's maxLogSize	The log is cut into pieces. No bytes are lost, but a cut JSON log is no longer valid JSON
A message would make too many logs	The message is kept whole, as one log (see Limits)
A message's logs are too large to travel together	The message is kept whole, as one log. All of a message's logs travel in one payload, so that its source acknowledges it only once they are all delivered. When the device's buffer, or one payload, cannot hold them all, the message is not split
The device names a rule set the Director does not have, or one that is not valid	The Director skips that rule set and logs a warning at most once a minute

What Happens to a File

The Azure Blob Storage, Amazon S3 and Google Cloud Storage devices read files, not messages. On them, the unit is the file:

Situation	Result
A file arrives	The Director checks the rules once for the file, against the first inspectBytes bytes of the file after it is decompressed. A condition reads the file's path as source
A rule matches	The rule splits the whole file as the Director reads it. A large file is never held in memory whole
No rule matches	The file is read as before, by the decoder for its extension, and the device's split_mode then splits each record. The rules are not tried again on each record
An archive	Each file inside the archive is matched on its own
A Parquet file	Never matched: it is not text
A file of an extension the device does not read, such as .csv	Refused before the rules are checked, as on a device without rules. A rule that would match it does not make it readable
The file's logs are delivered	The device still acknowledges the file as a whole. Its queue message, or its place in the listing, waits until all of the file's logs are delivered

A file is split as its text would be split as one message, with these differences:

• A file has no limit on how many logs it makes, and a json_array rule has no limit on the elements it splits.
• A rule makes logs of the file's text as it is. A lines rule makes each line of a .log file a log, without the {"message": ...} wrapper the extension's decoder adds.
• When a rule cannot split part of a file, the logs it made so far stay. The rest of the file is sent as it is, in pieces of at most maxLogSize, and the Director writes a warning that says from which byte. When the rule fails at the start of the file, the whole file is sent as it is.
• A csv or header rule sends the rest of a file as it is, from the row where it stops, when a row is longer than twice maxLogSize plus 1 MB, when a row has more than 10,000 values, or when the logs so far would hold more than 8 times what the Director has read of the file (1 MB near the start of the file).

Limits

A json_array rule splits an array of at most 10,000 elements. Every rule makes at most 10,000 logs of a message, or one log for every 16 bytes of a larger message. This count includes the pieces of logs that were cut because they were too long. When a rule would make more, the Director keeps the message whole, as one log.

The same limit applies when the device's split_mode splits a message that no rule matched.

Warnings

The Director writes a warning to its log at most once a minute for each of these:

• It does not have a rule set the device names. It skips that rule set.
• A rule set is not valid. It skips that rule set.
• A rule matched a message but could not split it, for example a json_array rule given text that is not JSON, or a message that would make too many logs. It keeps the message whole, as one log.
• A message's logs are too large to be handed on together. It keeps the message whole, as one log.

Direct Path

A device with event split rules never takes the direct path, because the direct path sends each message whole.

Edits and Restarts

A rule set that you edit applies from the next message, without restarting the device. Changing the device's event_split_rules list, including its order, restarts the device (see Changes That Restart a Device).

Crash Recovery

With crash_resistance, a device's queue files on disk are recovered after a crash as they were written:

• A file written while the device had rules holds logs that were split already, so they are not split again.
• A file written before the device had rules holds whole messages. They are split by the device's split_mode, as they would have been before the crash, and not by the rules.