Normalize

Transform

Synopsis

Converts log field names between different normalization formats (ECS, CIM, ASIM, CEF, LEEF, CSL, OCSF, UDM).

Schema

- normalize:
    source_format: <string>
    target_format: <string>
    field: <ident>
    ignore_unmapped_fields: <boolean>
    ignore_rules: <boolean>
    use_json_parser: <boolean>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

Field	Required	Default	Description
source_format	N	auto-detect	Source format (ecs, cim, asim, cef, leef, csl, ocsf, udm)
target_format	Y	-	Target format to convert to
field	N	-	Nested field containing the data to normalize
ignore_unmapped_fields	N	false	Skip fields without mappings in the target format
ignore_rules	N	false	Skip schema enforcement rules for the target format
use_json_parser	N	false	Parse the source field as JSON before normalization
description	N	-	Documentation note
if	N	-	Conditional expression
ignore_failure	N	false	Skip processing errors
on_failure	N	-	Error handling processors
on_success	N	-	Success handling processors
tag	N	-	Identifier for logging

Details

The processor automatically detects source formats when not specified and handles field name transformations while preserving values. Format detection is case-insensitive.

Supported Formats

Format	Description
ecs	Elastic Common Schema
cim	Splunk Common Information Model
asim	Microsoft Sentinel ASIM
cef	Common Event Format
leef	Log Event Extended Format
csl	Common Schema Log
ocsf	Open Cybersecurity Schema Framework
udm	Google SecOps Unified Data Model

Schema Enforcement

When converting to UDM or OCSF, the processor applies schema enforcement rules by default. These rules normalize timestamps to microseconds, validate event types, normalize security actions, and ensure field values conform to the target schema specification. Set ignore_rules: true to skip enforcement.

Field Mappings

The processor automatically detects the source format based on characteristic fields. Common mappings between formats:

	ECS	CIM	ASIM	UDM
Network
	source.ip	src	SrcIp	principal.ip
	destination.ip	dest	DstIp	target.ip
	network.direction	direction	NetworkDirection	network.direction
Event
	@timestamp	_time	TimeGenerated	metadata.event_timestamp
	event.action	action	EventType	metadata.product_event_type
	event.severity	severity	EventSeverity	security_result.severity
User
	source.user.name	user	ActorUsername	principal.user.userid
	source.user.id	user_id	ActorUserId	principal.user.windows_sid
	source.user.email	user_email	ActorUserEmail	principal.user.email_addresses

See Appendix sections CIM and ECS for details.

warning

Field mapping is non-reversible if the targeted format doesn't have equivalent fields. Test the conversions beforehand with sample data.

Examples

ECS to CIM

ECS fields...	{ "source": { "ip": "128.232.110.120" }, "destination": { "ip": "192.168.1.1" }, "network": { "direction": "inbound" } } - normalize: source_format: ecs target_format: cim
are mapped to CIM fields:	{ "src": "128.232.110.120", "dest": "192.168.1.1", "direction": "inbound" }

CIM to ECS

CIM fields...	{ "src": "128.232.110.120", "dest": "192.168.1.1", "direction": "outbound" } - normalize: source_format: cim target_format: ecs
are mapped to ECS fields:	{ "source": { "ip": "128.232.110.120" }, "destination": { "ip": "192.168.1.1" }, "network": { "direction": "outbound" } }

Auto-detection

Auto-detection discovers CIM...	{ "_time": "2023-01-01T00:00:00Z", "src": "128.232.110.120", "dest": "192.168.1.1", "direction": "outbound" } - normalize: target_format: asim
and maps the fields to ASIM:	{ "TimeGenerated": "2023-01-01T00:00:00Z", "SrcIp": "128.232.110.120", "DstIp": "192.168.1.1", "NetworkDirection": "outbound" }

Error Handling

Handling conversion errors...	{ "source": { "invalid": true } } - normalize: target_format: cim ignore_failure: true on_failure: - set: field: error value: "Conversion failed"
captures the error information:	{ "source": { "invalid": true }, "error": "Conversion failed" }

ECS to UDM

ECS network event fields...	{ "@timestamp": "2024-01-15T10:30:00.000Z", "source": { "ip": "192.168.1.100", "port": 54321, "user": { "name": "jdoe" } }, "destination": { "ip": "10.0.0.50", "port": 443 }, "event": { "action": "connection", "severity": "low" } } - normalize: source_format: ecs target_format: udm
are mapped to Google SecOps UDM fields:	{ "metadata": { "event_timestamp": 1705315800000000, "event_type": "NETWORK_CONNECTION" }, "principal": { "ip": "192.168.1.100", "port": 54321, "user": { "userid": "jdoe" } }, "target": { "ip": "10.0.0.50", "port": 443 }, "security_result": { "severity": "LOW", "action": "UNKNOWN_ACTION" } }