Linux Datasets

Linux datasets define collection rules for Linux devices (dataset type linux). Most collector types, including file log collection, are configured through the dataset creation wizard — see Management; the connection event and script execution collectors below are configured via device YAML. For the dataset and profile model, see Overview.

Datasets do not run over agentless SSH

A Linux device collecting agentless over SSH runs inventory collection only — every dataset assigned to it is skipped, including the collectors on this page. Assign datasets to Linux devices running the VirtualMetric Agent. See Linux for the two deployment modes.

Collector Types

• Linux System Events (linux_host_log_collector): File path input for the system log source.
• Linux Audit Events (linux_audit_report_log_collector): File path input for the audit log source.
• Linux Firewall Events (linux_firewall_log_collector): File path input for the firewall log source.
• Linux TCP Connections (linux_tcp_connection_collector): Established TCP connections with process attribution and destination geolocation.
• Linux Remote Connections (linux_remote_connection_collector): Remote login sessions (SSH and console) from host session records.
• Linux Script Execution (linux_execute_collector): Runs a script or executable and collects its standard output as log events.
• Kubernetes Pod Logs (linux_kubernetes_pod_log_collector): Container logs written by the kubelet under /var/log/pods, decoded as CRI and filtered by namespace, pod label, or owner kind.
• Kubernetes Metrics (linux_kubernetes_metrics_collector): Node, pod, and container usage and status read from the kubelet, with optional cluster-object and cAdvisor series.

Connection Event Collectors

The TCP connection (linux_tcp_connection_collector) and remote connection (linux_remote_connection_collector) collectors capture host network activity as ECS events. The TCP collector reports established TCP connections with process attribution and destination geolocation. The remote connection collector reports remote login sessions (SSH, derived from pts* devices) and local console sessions (tty*) from the host session records. Both accept only a pipeline_name property for routing; collection cadence, connection caching, and country-level geolocation are handled automatically.

note

These collectors are configured via device YAML files under config/devices/ — they are not available in the dataset creation wizard.

Enabling TCP and remote connection collection on a Linux device, routing both to a pipeline...	devices: - id: 123458 name: linux-host type: linux status: true definitions: - name: linux_tcp_connection_collector status: true properties: pipeline_name: my-pipeline - name: linux_remote_connection_collector status: true properties: pipeline_name: my-pipeline
Each established TCP connection is emitted as an ECS event with process and destination geo...	{ "@timestamp": "2026-07-24T10:15:03.472Z", "event": { "dataset": "linux.tcp_connection", "category": ["network"], "type": ["connection", "start"], "action": "established" }, "host": { "name": "linux-host", "ip": ["10.0.0.12"] }, "network": { "transport": "tcp", "type": "ipv4" }, "source": { "ip": "10.0.0.12", "port": 52344 }, "destination": { "ip": "140.82.112.3", "port": 443, "geo": { "country_iso_code": "US", "country_name": "United States" } }, "process": { "name": "curl", "pid": 4821 } }
A remote login session is emitted with the authenticated user and session type...	{ "@timestamp": "2026-07-24T10:12:55.000Z", "event": { "dataset": "linux.remote_connection", "category": ["authentication", "session"], "type": ["start"], "action": "SSH" }, "message": "admin@10.0.0.5", "host": { "name": "linux-host", "ip": ["10.0.0.12"] }, "network": { "transport": "tcp" }, "user": { "name": "admin" }, "source": { "ip": "10.0.0.5", "geo": { "country_iso_code": "US", "country_name": "United States" } } }

Script Execution

The linux_execute_collector runs a script or executable on the Linux host and collects its standard output as log events. Each execution is defined as an input with its own interpreter, arguments, schedule, and pipeline routing, so a single dataset can run multiple scripts.

note

This collector is configured via device YAML files under config/devices/. Scripts must be placed in the agent's user/scripts (operator-owned, survives upgrades) or package/scripts (product-shipped) directory; the path property is a file name only.

Input properties

Each entry under inputs: defines one script execution. Inputs share the standard dataset input frame (id, name, status) plus the following properties:

Field	Required	Type	Default	Description
path	Y	string	—	Script or executable file name (not a path). Resolved under the agent's scripts directories. Absolute paths and .. traversal are rejected.
interpreter	N	string	—	Interpreter that runs the script. Linux: sh, bash, or exe. Windows: ps1, cmd, or exe. exe runs path directly. There is no implicit default; if unset (or unrecognized), shell is used instead.
shell	N	string	—	Custom interpreter for the fall-through case (e.g. python3 or an executable path). On Windows with interpreter: ps1, selects the PowerShell edition: pwsh (7+), powershell (5.1), or unset (auto).
args	N	list	—	Arguments passed to the script or executable.
environment	N	map	—	KEY: VALUE pairs appended to the inherited process environment.
working_directory	N	string	inherit	Working directory for the child process.
timeout	N	int	120	Maximum run time in seconds. A non-positive value falls back to 120. On timeout, the entire child process group is terminated.
max_bytes	N	int	0	Cap on captured output in bytes. 0 means unbounded.
emit_on_error	N	boolean	false	When true, a non-zero exit code still emits the captured stdout. A timeout or a failure to start never emits, regardless of this setting.
execution_policy_bypass	N	boolean	false	Windows ps1 only. Adds -ExecutionPolicy Bypass to the PowerShell invocation.
cron	N	string	—	Per-input schedule (cron expression). An empty value or "0" runs the script on every collection cycle.
max_retries	N	int	1	Number of times a failed run is retried on subsequent collection cycles.
skippable	N	boolean	true	When true, missed schedule slots collapse into a single run. When false, each missed slot is replayed.
pipeline_name	N	string	—	Route the output to a specific preprocessing pipeline by name.
ignore_time	N	boolean	true	Skip time-based filtering on the emitted events.

Output

Captured stdout is converted to log events one line at a time:

• Trailing line terminators are trimmed once; if nothing remains, no events are emitted (an empty or whitespace-only run produces nothing).
• The remaining output is split on newlines, producing one event per line. A trailing carriage return on each line is stripped, so CRLF output is handled.
• Interior blank lines are preserved; only trailing blank lines from a final newline are dropped.
• stderr is never emitted as events. The exit code is logged and gates emit_on_error.

Example

Running a health-check shell script every five minutes and routing its output to a pipeline...	devices: - id: 123459 name: linux-host type: linux status: true definitions: - name: linux_execute_collector status: true inputs: - id: 3001 name: Disk Health status: true properties: path: disk-health.sh interpreter: bash args: - "--all" cron: "*/5 * * * *" timeout: 30 pipeline_name: my-pipeline
Each line of standard output becomes a separate log event...	{ "message": "/dev/sda1 usage 62%" } { "message": "/dev/sdb1 usage 41%" }

File Log Collection

The linux_file_log_collector collects lines from arbitrary log files on Linux hosts. It supports glob path expansion, lookback-based backfill, multiline parsing, include/exclude filtering, character-set decoding, and per-pipeline routing. The path syntax uses Linux-style paths (e.g. /var/log/myapp/*.log).

note

File log datasets can be created through the dataset creation wizard (Linux File Logs) or configured via device YAML files under config/devices/ using the schema below.

Device-level property

Field	Required	Type	Default	Description
file_log_concurrency	N	int	8	Maximum number of inputs processed in parallel per device. A zero or negative value resolves to 8.

Input properties

Each entry under inputs: defines one file log source. Inputs share the standard dataset input frame (id, name, status) plus the following properties:

Field	Required	Type	Default	Description
path	Y	string	—	File path or glob pattern. Supports wildcards (e.g. /var/log/myapp/*.log, C:\Logs\*\app-*.log).
start_date	N	int	300	Lookback window in seconds: only files modified within this window are read. A negative value (e.g. -1) disables time-based filtering. 0 is not a way to narrow the window — it resolves to the 300 default.
ignore_cache	N	boolean	false	Skip the persisted file-position cache (re-read from the start).
ignore_old_date	N	boolean	false	Skip old-date filtering.
ignore_retention	N	boolean	false	Skip retention filtering.
ignore_time	N	boolean	false	Skip time-based filtering.
date_format	N	string	—	Custom timestamp layout using Java-style tokens (e.g. yyyy-MM-dd HH:mm:ss). Used to extract an event timestamp from each line.
line_parser	N	object | string	—	Multiline detection rules. See Line parser.
filter_mode	N	string	exclude	include or exclude. Synonyms: inclusive/allow (include), exclusive/deny (exclude).
filter_rules	N	array	—	Include/exclude rules. See Filter rules. Alias: filters.
encoding	N	string | int	—	Character encoding for the file. Accepts an alias (e.g. utf-8, windows-1252) or a numeric decoder ID. See Encoding aliases.
pipeline_name	N	string	—	Route matched lines to a specific preprocessing pipeline by name.

Line parser

line_parser controls how multi-line log entries are reassembled. It accepts either an object or a bare string shorthand.

Accepted type values:

Value	Aliases	Behavior
regex	1	A regex pattern detects where a new entry begins.
newline	new_line, 2	Each raw line is a separate entry.
string	prefix, 3	Lines beginning with a literal string mark a new entry.

Object fields:

Field	Required	Type	Description
type	N	string | int	One of the values above. If omitted and regex is set, regex is assumed.
regex	N	string	Pattern used by regex, or the literal prefix used by string/prefix. Compiled with Go's RE2 engine, which does not support lookarounds or backreferences.
value	N	string	Alias for regex.
date_based	N	boolean	Merge continuation lines using date boundaries detected via date_format.
has_space	N	boolean	Treat leading whitespace on a line as a continuation of the previous entry.

A bare string at line_parser: is treated as type: regex with that pattern:

line_parser: '^\d{4}-\d{2}-\d{2}'

Filter rules

filter_rules is an array of include/exclude rules applied to the content of each record, after line parsing. The effect of each match (keep or drop) is determined by filter_mode. These rules select content; the files themselves are selected by path.

Each rule accepts:

Field	Required	Type	Description
type	N	string | int	regex (1) or string (2). If omitted, inferred from which field is set.
regex	N	string	Regex pattern matched against the record. Compiled with a .NET-compatible engine, so lookarounds and backreferences are available — unlike line_parser.regex, which uses RE2.
source	N	string	Literal text matched against the record, with optional leading and trailing wildcards. Not a file path — use path to select files.
value	N	string	Alias for source.

A bare array of regex strings is also accepted and is equivalent to type: regex for each entry.

A string rule matches its text literally: regex metacharacters carry no special meaning, so *[error]* looks for the six characters [error] rather than a character class. Wildcards are positional and only recognized at the ends:

source	Matches a record that
*text*	contains text
text*	starts with text
*text	ends with text
text	contains text

An interior * is matched as a literal asterisk and reported in the Console Logs — use type: regex for pattern matching. A source that is empty once its wildcards are removed matches nothing.

A rule the reader cannot use — an invalid regex, or a string rule with an empty literal — is reported and then discounted. When no rule in the set is usable, filtering is inactive and no record is dropped, including under filter_mode: include.

Encoding aliases

encoding accepts a numeric decoder ID or one of the following aliases. Dashes, underscores, dots, and spaces are ignored during alias lookup, so utf-8, utf_8, and UTF 8 all resolve identically.

Alias(es)	ID
utf8	1
utf8bom	2
utf16be	3
utf16le	4
utf16bebom	5
utf16lebom	6
gbk	11
latin1, iso88591	15
windows1250, cp1250	50
windows1251, cp1251	51
windows1252, cp1252	52
windows1256, cp1256	56

Example

Collecting a rotating application log on Linux with regex-based multiline detection and an include filter for errors and warnings...

devices: - id: 123457 name: app-linux type: linux status: true properties: file_log_concurrency: 2 definitions: - name: linux_file_log_collector status: true inputs: - id: 1001 name: Application Logs status: true properties: path: /var/log/myapp/*.log start_date: 300 date_format: yyyy-MM-dd HH:mm:ss line_parser: type: regex regex: '^\d{4}-\d{2}-\d{2}' date_based: true filter_mode: include filter_rules: - type: regex regex: '^(ERROR|WARN)' encoding: utf-8 pipeline_name: my-pipeline

Kubernetes Pod Logs

The linux_kubernetes_pod_log_collector reads the container log files the kubelet writes under /var/log/pods on the node. Each matched file is decoded as CRI, and its path is parsed into the namespace, pod name, pod UID, and container name that identify it — no Kubernetes API call is involved in that step. Namespace filtering works from the parsed path alone; label and owner-kind filtering additionally consult a pod watcher.

note

Kubernetes pod log datasets can be created through the dataset creation wizard (Kubernetes Pod Logs, under the Linux category) or configured via device YAML files under config/devices/. They can be assigned only to Linux devices.

The agent must be able to read /var/log/pods on the node, so a containerized agent needs that path mounted. The decoder property is fixed to cri and cannot be changed, which is why the wizard does not offer a decoder selector for this type.

Input properties

Each entry under inputs: defines one pod log source. Inputs share the standard dataset input frame (id, name, status) plus the following properties:

Field	Required	Type	Default	Description
path	N	string	/var/log/pods/*/*/*.log	Glob or comma-separated globs matching pod log files. The kubelet layout is <root>/<namespace>_<pod>_<uid>/<container>/<N>.log. An empty or unset value falls back to the default.
exclude_paths	N	string	—	Glob or comma-separated globs pruned from the path match set. A different dialect than path: each pattern is matched against the whole path, ** is not supported, * never crosses /, and the pattern must mirror the full segment depth of path (e.g. /var/log/pods/*/sidecar/*.log). A malformed pattern excludes nothing rather than everything.
namespaces	N	string	—	Comma-separated namespace allowlist, evaluated from the parsed path. Empty means every namespace is eligible.
exclude_namespaces	N	string	kube-system,kube-node-lease,kube-public	Comma-separated namespace denylist. Evaluated after namespaces and always wins, so a namespace in both lists is excluded. Set it to an empty value to collect the system namespaces.
label_selector	N	string	—	Pod-label filter in the Kubernetes label-selector grammar: equality (key=value), inequality (key!=value), existence (key, !key), and set-based (key in (v1,v2), key notin (v1,v2)), combined with AND. Requires API access. An unparseable value disables label filtering for that input and is reported once.
exclude_owner_kinds	N	string	—	Comma-separated list of controlling owner-reference kinds to exclude, e.g. DaemonSet, Job, StatefulSet. Case-insensitive. Requires API access.
unparsed_paths	N	string	collect	What to do with a matched file whose path does not parse into a pod identity. collect reads the file anyway, skipping the four filters above for that file; drop skips the file. Any other value resolves to collect.
api_server	N	string	in-cluster	API server URL, used only when label_selector or exclude_owner_kinds is set. Must be an absolute https:// URL. When empty, it is derived from KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT.
token_path	N	string	/var/run/secrets/kubernetes.io/serviceaccount/token	Service account token file. Same gate as api_server.
ca_path	N	string	/var/run/secrets/kubernetes.io/serviceaccount/ca.crt	CA certificate file used to verify the API server. Same gate as api_server.
start_date	N	int	300	Lookback window in seconds: only files modified within this window are read. A negative value (e.g. -1) disables time-based filtering. 0 resolves to the 300 default.
pipeline_name	N	string	—	Route decoded lines to a specific preprocessing pipeline by name.

The reader properties are the same as file log collection — ignore_cache, ignore_old_date, ignore_time, ignore_retention, date_format, line_parser, filter_mode, filter_rules, and encoding — and the device-level file_log_concurrency applies here too. See File Log Collection for their semantics.

Filters degrade to collection, not exclusion

label_selector and exclude_owner_kinds need a synchronized pod watcher. When the watcher cannot be built, has not synchronized yet, or has no record of the pod, the file is collected rather than excluded, and the condition is reported once. The API server URL is derived from environment variables the kubelet injects into pods, so an agent running as a host process outside a pod has no in-cluster API access and these two filters are inert. namespaces and exclude_namespaces are unaffected — they never consult the watcher.

Reaching the API server requires a service account with list and watch on pods.

Output

The emitted record's source field carries the parsed identity as <namespace>/<pod>/<pod-uid>/<container>; a file whose path did not parse keeps its raw file path instead. Structured k8s.* fields are not added by this collector — to attach them, route the records through the Kubernetes processor, which looks the pod up in the cluster and writes k8s.namespace, k8s.pod, k8s.deployment, and k8s.node.

Example

Collecting application pod logs while excluding the system namespaces and DaemonSet-owned pods...

devices: - id: 123460 name: k8s-node-1 type: linux status: true definitions: - name: linux_kubernetes_pod_log_collector status: true inputs: - id: 4001 name: Application Pods status: true properties: path: /var/log/pods/*/*/*.log exclude_namespaces: kube-system,kube-node-lease,kube-public label_selector: app=web,tier!=debug exclude_owner_kinds: DaemonSet unparsed_paths: drop start_date: 300 pipeline_name: my-pipeline

Each decoded container log line becomes a record carrying the pod identity parsed from its path...

{ "message": "listening on :8080", "source": "prod/web-1/8f3c2b1a-5d6e-4f70-9a2b-1c3d4e5f6a7b/web" }

Kubernetes Metrics

The linux_kubernetes_metrics_collector reads node, pod, and container usage and status from the kubelet on the node, and can optionally add cluster-object counts from the API server and a curated set of cAdvisor series. Each metric sample is emitted as its own record.

note

Kubernetes metrics datasets can be created through the dataset creation wizard (Kubernetes Metrics, under the Linux category) or configured via device YAML files under config/devices/. They can be assigned only to Linux devices.

Input properties

Each entry under inputs: defines one metrics source. Inputs share the standard dataset input frame (id, name, status) plus the following properties:

Field	Required	Type	Default	Description
kubelet_url	N	string	https://127.0.0.1:10250	Kubelet base URL. Must use https; an http URL is rejected before the request is made.
token_path	N	string	—	Service account token file presented to the kubelet as a bearer token. See the caution below — there is no in-cluster fallback on this path.
ca_path	N	string	—	CA certificate file used to verify the kubelet. Ignored while insecure_skip_verify is true.
insecure_skip_verify	N	boolean	true	Skip TLS verification of the kubelet certificate. Set it to false to verify against ca_path. Applies to the kubelet endpoints only, never to the API server.
node_name	N	string	the device name	Node name stamped onto every record and used to scope the pod watcher.
collect_cluster_objects	N	boolean	false	Also read deployment, daemonset, statefulset, job, cronjob, service, and namespace counts from the API server.
scrape_cadvisor	N	boolean	false	Also scrape the kubelet cAdvisor endpoint for a curated set of container CPU, filesystem, and network series.
api_server	N	string	in-cluster	API server URL used by the pod watcher and by collect_cluster_objects. Derived from KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT when empty.
pipeline_name	N	string	—	Route emitted metrics to a specific preprocessing pipeline by name.

retention is read at the definition level and falls back to the device retention.

token_path is effectively required

The kubelet endpoints apply no in-cluster default for token_path: when it is empty, no authorization header is sent at all. On a cluster with the usual kubelet authentication that means every request is rejected and the dataset emits nothing but a single reported warning. Set token_path explicitly. The in-cluster defaults do apply to api_server, which is a separate path used only by the watcher and by collect_cluster_objects.

node_name must match the Kubernetes node

When node_name is unset it falls back to the device name. If that does not equal the node name in Kubernetes, the pod watcher matches nothing and no pod or container status metrics are produced, while usage metrics continue to arrive — a partial result rather than a visible failure.

The token needs nodes/stats for the kubelet endpoints, and list plus watch on pods for the status metrics. collect_cluster_objects additionally needs list on deployments, daemonsets, and statefulsets (apps/v1), jobs and cronjobs (batch/v1), and services and namespaces (core/v1).

Cluster objects are collected per node, with no leader election

collect_cluster_objects reads cluster-wide counts. Nothing elects a single reporter, so enabling it on the datasets of several nodes multiplies every cluster-object series by the number of nodes that have it on. Enable it on exactly one node dataset per cluster.

Each source degrades independently: a kubelet, watcher, cluster-object, or cAdvisor failure is reported once and yields no records from that source, while the others continue. A cAdvisor scrape is capped at 50,000 records; beyond that the excess is discarded and reported.

Output

Each sample is emitted as a metric record:

Field	Description
metric.name	Metric name, e.g. pod_cpu_usage_nanocores.
metric.value	Numeric value.
metric.type	gauge or counter.
metric.source	Origin of the sample: kubelet_summary, cadvisor, kube_api, or watcher.
k8s.namespace, k8s.pod, k8s.container, k8s.node, k8s.kind, k8s.name	Identity of the object the sample describes. Empty fields are omitted.
@timestamp	Sample time.

Example

Collecting kubelet metrics from one node with verification enabled, adding cluster objects on this node only...

devices: - id: 123461 name: k8s-node-1 type: linux status: true definitions: - name: linux_kubernetes_metrics_collector status: true inputs: - id: 5001 name: Node Metrics status: true properties: kubelet_url: https://127.0.0.1:10250 token_path: /var/run/secrets/kubernetes.io/serviceaccount/token ca_path: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: false node_name: k8s-node-1 collect_cluster_objects: true scrape_cadvisor: true pipeline_name: my-pipeline

Every sample is emitted as its own record, carrying the object it describes...

{ "metric": { "name": "pod_cpu_usage_nanocores", "value": 42, "type": "gauge", "source": "kubelet_summary" }, "k8s": { "namespace": "default", "pod": "web-1", "node": "k8s-node-1" }, "@timestamp": "2026-08-25T10:15:03.472Z" }