Normalize

Transform

Synopsis

Converts log field names between different normalization formats (ECS, CIM, ASIM, CEF, LEEF, CSL, OCSF, UDM, GELF).

Schema

- normalize:
    source_format: <string>
    target_format: <string>
    field: <ident>
    ignore_unmapped_fields: <boolean>
    ignore_rules: <boolean>
    use_json_parser: <boolean>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

Field	Required	Default	Description
source_format	N	auto-detect	Source format (ecs, cim, asim, cef, leef, csl, ocsf, udm, gelf)
target_format	Y	-	Target format (ecs, cim, asim, cef, leef, csl, ocsf, udm, gelf)
field	N	-	Nested field containing the data to normalize
ignore_unmapped_fields	N	false, or true for csl	Drop fields with no mapping in the target format. Unset, it is false everywhere except a csl target, where it defaults to true
ignore_rules	N	false	Skip schema enforcement for the target format. See below for which targets enforce
use_table	N	false	Write the normalized result into a virtual table rather than the map
source_table	N	false	Read the source from the entry's selected virtual table instead of the map, and drop that table afterwards. With no table selected the source is the map, so map-mode records are unaffected
select_table	N	true	use_table only. Whether this normalize also claims the marshal source. Set false for an intermediate normalize whose table is not the terminal one
additional_extensions	N	true	CSL target only. Whether fields with no CSL mapping are folded into the AdditionalExtensions string. Set false when the destination is ASIM only, where that string is never emitted
use_json_parser	N	false	CSL target only. Write unmapped fields as structured keys under AdditionalExtensions.<name> instead of leaving them at the root. It does not parse the source field as JSON
description	N	-	Documentation note
if	N	-	Conditional expression
ignore_failure	N	false	Skip processing errors
on_failure	N	-	Error handling processors
on_success	N	-	Success handling processors
tag	N	-	Identifier for logging
disabled	N	false	When true, the processor is skipped and the event continues to the next one. Lets you take a processor out of the path without removing its configuration

Details

The processor automatically detects source formats when not specified and handles field name transformations while preserving values. Format detection is case-insensitive.

Supported Formats

Format	Description
ecs	Elastic Common Schema
cim	Splunk Common Information Model
asim	Microsoft Sentinel ASIM
cef	Common Event Format
leef	Log Event Extended Format
csl	Microsoft Sentinel CommonSecurityLog
ocsf	Open Cybersecurity Schema Framework
udm	Google SecOps Unified Data Model
gelf	Graylog Extended Log Format

Schema Enforcement

Schema enforcement runs by default for five target formats: asim, csl, ocsf, udm and gelf. The rules normalize timestamps, validate event types, normalize security actions and bring field values into line with the target schema. Set ignore_rules: true to skip it.

The other targets — ecs, cef and leef — map fields and apply no enforcement pass, so ignore_rules has no effect on them.

Field Mappings

The processor automatically detects the source format based on characteristic fields. Common mappings between formats:

	ECS	CIM	ASIM	UDM
Network
	source.ip	src	SrcIp	principal.ip
	destination.ip	dest	DstIp	target.ip
	network.direction	direction	NetworkDirection	network.direction
Event
	@timestamp	_time	TimeGenerated	metadata.event_timestamp
	event.action	action	EventType	metadata.product_event_type
	event.severity	severity	EventSeverity	security_result.severity
User
	source.user.name	user	ActorUsername	principal.user.userid
	source.user.id	user_id	ActorUserId	principal.user.windows_sid
	source.user.email	user_email	ActorUserEmail	principal.user.email_addresses

See the Appendix sections ECS, CIM, ASIM, CEF, LEEF, CSL, OCSF, UDM and GELF for the full field mappings.

warning

Field mapping is non-reversible if the targeted format doesn't have equivalent fields. Test the conversions beforehand with sample data.

Examples

ECS to CIM

ECS fields...	{ "source": { "ip": "128.232.110.120" }, "destination": { "ip": "192.168.1.1" }, "network": { "direction": "inbound" } } - normalize: source_format: ecs target_format: cim
are mapped to CIM fields:	{ "src": "128.232.110.120", "dest": "192.168.1.1", "direction": "inbound" }

CIM to ECS

CIM fields...	{ "src": "128.232.110.120", "dest": "192.168.1.1", "direction": "outbound" } - normalize: source_format: cim target_format: ecs
are mapped to ECS fields:	{ "source": { "ip": "128.232.110.120" }, "destination": { "ip": "192.168.1.1" }, "network": { "direction": "outbound" } }

Auto-detection

Auto-detection discovers CIM...	{ "_time": "2023-01-01T00:00:00Z", "src": "128.232.110.120", "dest": "192.168.1.1", "direction": "outbound" } - normalize: target_format: asim
and maps the fields to ASIM:	{ "TimeGenerated": "2023-01-01T00:00:00Z", "SrcIp": "128.232.110.120", "DstIp": "192.168.1.1", "NetworkDirection": "outbound" }

Error Handling

Handling conversion errors...	{ "source": { "invalid": true } } - normalize: target_format: cim on_failure: - set: field: error value: "Conversion failed"
captures the error information:	{ "source": { "invalid": true }, "error": "Conversion failed" }

ECS to UDM

ECS network event fields...	{ "@timestamp": "2024-01-15T10:30:00.000Z", "source": { "ip": "192.168.1.100", "port": 54321, "user": { "name": "jdoe" } }, "destination": { "ip": "10.0.0.50", "port": 443 }, "event": { "action": "connection", "severity": "low" } } - normalize: source_format: ecs target_format: udm
are mapped to Google SecOps UDM fields:	{ "metadata": { "event_timestamp": 1705315800000000, "event_type": "NETWORK_CONNECTION" }, "principal": { "ip": "192.168.1.100", "port": 54321, "user": { "userid": "jdoe" } }, "target": { "ip": "10.0.0.50", "port": 443 }, "security_result": { "severity": "LOW", "action": "UNKNOWN_ACTION" } }