Smart Engine

Analytics Flow Control Security

Synopsis

Registers your KQL detection rules and forwards only what those rules could ever need. An event that no detection could match is dropped; an event that survives keeps only the fields the applicable detections read or return.

The detections are your existing SIEM rules, used unchanged. The engine does not convert them into a pipeline — for that, see From KQL.

Schema

- smart_engine:
    detections:
      - name: <string>
        query: <string>
    library: <string[]>
    select: <object>
    table_field: <ident>
    unmatched_tables: <enum>
    mark_only: <boolean>
    mark_field: <ident>
    value_reduction: <boolean>
    field_reduction: <boolean>
    critical_fields: <string[]>
    description: <text>
    if: <script>
    tag: <string>
    disabled: <boolean>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>

Configuration

Either detections or library must be set. With neither, the processor fails — it does not drop.

Field	Required	Default	Description
`detections`	Y*		Detection rules given inline. Each entry is a `name` and a `query`, where `query` is raw KQL taken verbatim from the SIEM rule.
`library`	Y*		Names of Microsoft Sentinel analytics-rule packs assigned through the platform Library. Globs are accepted, so `"*"` selects every assigned pack. Only each rule's KQL is used. Unioned with `detections`.
`select`	N	all rules	Narrows which rules from `library` are registered. See below.
`table_field`	N	`_vmetric.table`	The event field holding the table name a detection is scoped against.
`unmatched_tables`	N	`keep`	What to do with an event that no detection is scoped to: `keep` or `drop`.
`mark_only`	N	`false`	Dry run. Marks each event with the decision the engine would have taken, but never drops or reduces anything.
`mark_field`	N	`_smart`	Root field the markers are written under.
`value_reduction`	N	`true`	Strip individual values from comma-separated fields.
`field_reduction`	N	`true`	Strip fields no applicable detection needs.
`critical_fields`	N		Fields kept on every kept event, whatever the detections read. Names only: they never rescue an event from being dropped, and an absent field is not created.
`description`	N		Explanatory note.
`if`	N		Condition that must be true for the processor to run.
`tag`	N		Identifier for this processor instance.
`disabled`	N	`false`	When `true`, the processor is skipped.
`ignore_failure`	N	`false`	Continue pipeline processing if the processor fails.
`on_failure`	N		See Handling Failures.
`on_success`	N		See Handling Success.

* = One of detections or library is required.

Loading include...

Select

All criteria are combined with AND; the values within one criterion are combined with OR. Matching is case-insensitive.

Field	Description
`tables`	Rules whose declared data types include any of these, such as `SecurityEvent` or `CommonSecurityLog`.
`connectors`	Rules by required data connector id.
`severities`	`informational`, `low`, `medium`, or `high`.
`kinds`	`Scheduled` or `NRT`.
`tactics`	MITRE tactic. Separator-insensitive, so `CommandAndControl` and `command-and-control` are equivalent.
`techniques`	MITRE technique id. A parent id also selects its sub-techniques, so `T1059` selects `T1059.001`.
`ids`	Rule id, matched exactly.
`name_prefix`, `name_contains`, `name_regex`	Rule title matching.
`desc_contains`	Rule description matching.
`tags`	Rule tags.

Details

How an Event Is Decided

For each event, in order:

The event's table is read from table_field.
The applicable detections are selected: those scoped to that table, plus any scoped to no table at all. A detection's table is the source table named in its KQL.
Each applicable detection's condition is evaluated against the event. The condition is the detection's pre-aggregation where chain.
The event is dropped only when at least one detection was applicable and none of them matched.
If no detection was applicable, unmatched_tables decides. The default is to keep.

An event kept because no detection applied to it is forwarded whole — there is no mask to apply. Reduction only happens to events that some detection actually matched.

What Survives on a Kept Event

The surviving fields are the union of:

Every field read by the conditions of all applicable detections — matched or not. This is deliberate. A detection that did not match may still test a field for absence, and stripping that field could make the rule fire at the SIEM against an event that should never have triggered it.
Every field returned by the matched detections, meaning their project, project-keep, distinct, and summarize columns.
The critical_fields.
The marker, table, and system fields.

Everything else is removed.

Masking operates on top-level fields. A detection reading Properties.CommandLine preserves the whole Properties object, not just that one subfield. The engine over-keeps rather than under-keeps.

Fail Open

If the engine cannot understand a detection, that detection matches every event it is scoped to, and reduction is switched off for those events. It never discards an event because it failed to understand a rule.

A detection fails open when its query uses a construct the condition compiler does not handle — join, union, mv-expand, parse, evaluate, externaldata and the like — when the query cannot be fully parsed, or when its where reads a field computed by an earlier extend.

The consequence is worth stating directly: a rule the engine cannot parse produces no savings. And if such a rule also has no resolvable source table, it applies to every event in the pipeline, which switches reduction off entirely — one such rule in a large pack is enough to make the processor a no-op.

The engine warns when this happens, naming the detection and saying that it disables reduction globally until it is scoped to a table or narrowed with a project or summarize. If you register a large pack with library: ["*"] and see no volume reduction, that warning identifies the rule responsible.

Time Windows and Thresholds

Time predicates such as ago(), between, and now() are treated as true. The event has just arrived, so it is inside any recency window by definition; the SIEM re-checks the window at query time. The time field itself is still preserved, so that check can be made.

A threshold applied after a summarize — | summarize count() by User | where count_ > 5 — is ignored per event. Every contributing event is kept, even one that could never on its own reach the threshold, because the SIEM cannot compute the aggregate without them. Reduction from such a rule comes from its pre-aggregation where, not from the threshold.

Whole-Row Detections

A detection with no project, summarize, distinct, or project-keep presents the entire row at the SIEM, so nothing may be stripped from an event it matched. A matched whole-row detection therefore disables field reduction for that event.

Value reduction is disabled more readily: any applicable whole-row or fail-open detection disables it, whether or not it matched.

Value Reduction

For a string field that contains commas, and that some applicable detection tests with a positive literal membership test — ==, =~, has, has_any, in, and their relatives — the value is split on commas and a part is kept when a tested term appears in it as a whole word, matched case-insensitively. Kept parts are rejoined, preserving the original spacing.

Whole-word matching is what the SIEM itself applies when it re-runs the detection downstream, so a part carrying extra tokens around the term survives: a test for sshd keeps sshd (pid 1), not only a part that is exactly sshd.

Value reduction is blocked for any field that an applicable detection returns as a result column, or reads in any way other than a positive literal membership test: a regular expression, contains, startswith, a relational comparison, a negation, an absence test, or as an argument to a function.

Set value_reduction: false to keep field values untouched and reduce fields only.

Markers

Field	Written	Description
`_smart.rules`	On every kept event	The names of the matched detections, in configuration order. An empty list when the event was kept by the `unmatched_tables` policy.
`_smart.keep`	Only under `mark_only`	The decision the engine would have taken.

Both are written under mark_field. Nothing is written to a dropped event.

Rollout

Deploy with mark_only: true first. Every event is marked with the decision the engine would have made and nothing is dropped or stripped, so the volume of events carrying _smart.keep: false tells you the reduction to expect, and the distribution of _smart.rules tells you which detections are carrying it. Set mark_only: false to enforce.

Examples

Reducing a Matched Event

Registering a detection that reads two fields and returns two others...

{
  "_vmetric": { "table": "SigninLogs" },
  "ResultType": "50126",
  "UserPrincipalName": "alice@example.com",
  "IPAddress": "10.0.0.7",
  "UserAgent": "Mozilla/5.0",
  "CorrelationId": "abc-123"
}

- smart_engine:
    detections:
      - name: failed-signin
        query: "SigninLogs | where ResultType == '50126' | project UserPrincipalName, IPAddress"

The detection matches, so the event is kept and reduced to the fields the rule reads and returns. UserAgent and CorrelationId are needed by nothing and are removed...

{
  "_vmetric": { "table": "SigninLogs" },
  "_smart": { "rules": ["failed-signin"] },
  "ResultType": "50126",
  "UserPrincipalName": "alice@example.com",
  "IPAddress": "10.0.0.7"
}

Dropping an Event No Detection Wants

The same detection, against a successful sign-in...

{
  "_vmetric": { "table": "SigninLogs" },
  "ResultType": "0",
  "UserPrincipalName": "bob@example.com"
}

- smart_engine:
    detections:
      - name: failed-signin
        query: "SigninLogs | where ResultType == '50126' | project UserPrincipalName, IPAddress"

A detection was applicable to this table and it did not match, so the event is dropped. Nothing is forwarded...

// dropped

Keeping Fields the Detections Do Not Read

Registering an assigned Sentinel pack, narrowed to one table, and guaranteeing two fields the rules may not mention...	`- smart_engine: library: - "*" select: tables: - SecurityEvent critical_fields: - TimeGenerated - Computer`
Every assigned pack's `SecurityEvent` rules are registered. `TimeGenerated` and `Computer` survive on every kept event even when no rule reads them — but they never rescue an event from being dropped...	`{ "_smart": { "rules": ["Suspicious process executed"] }, "TimeGenerated": "2026-07-14T09:12:00Z", "Computer": "WS-014", "Process": "powershell.exe" }`

Measuring Before Enforcing

Running the same configuration as a dry run...

{
  "_vmetric": { "table": "SigninLogs" },
  "ResultType": "0",
  "UserPrincipalName": "bob@example.com"
}

- smart_engine:
    mark_only: true
    detections:
      - name: failed-signin
        query: "SigninLogs | where ResultType == '50126' | project UserPrincipalName, IPAddress"

The event would have been dropped, so it is marked and forwarded intact. Counting events with _smart.keep set to false gives the reduction to expect before anything is discarded...

{
  "_vmetric": { "table": "SigninLogs" },
  "_smart": { "rules": [], "keep": false },
  "ResultType": "0",
  "UserPrincipalName": "bob@example.com"
}

Synopsis​

Schema​

Configuration​

Select​

Details​

How an Event Is Decided​

What Survives on a Kept Event​

Fail Open​

Time Windows and Thresholds​

Whole-Row Detections​

Value Reduction​

Markers​

Rollout​

Examples​

Reducing a Matched Event​

Dropping an Event No Detection Wants​

Keeping Fields the Detections Do Not Read​

Measuring Before Enforcing​