DataStream Stats
Synopsis
The device subscribes to the Director's internal NATS JetStream stats stream and republishes route, target, device, pipeline, device-resource, queue, and log-severity metrics through standard pipelines and routes. Each device instance attaches its own durable consumer with an independent read position. Output field naming is selectable between ECS, Cribl, and Prometheus conventions.
Schema
- id: <numeric>
name: <string>
description: <string>
type: stats
tags: <string[]>
pipelines: <pipeline[]>
status: <boolean>
properties:
field_format: <string>
Configuration
The following fields are used to define the device:
Device
| Field | Required | Default | Description |
|---|---|---|---|
id | Y | - | Unique numeric identifier |
name | Y | - | Device name |
description | N | - | Optional description |
type | Y | - | Must be stats |
tags | N | - | Array of labels for categorization |
pipelines | N | - | Array of preprocessing pipeline references |
status | N | true | Enable/disable the device |
Output
| Field | Required | Default | Description |
|---|---|---|---|
field_format | N | cribl | Output field naming convention. One of ecs, cribl, prometheus. |
An unrecognized field_format is not rejected. The device logs a warning at startup and falls through to the internal ECS field names, so a typo produces a working collector whose field names do not match what was asked for.
Details
What Gets Emitted
Records fall into one of seven input_type categories:
- route — per-route counters including events and bytes in/out, latency, errors, and drops
- target — per-target counters with the same shape as route records, plus a target identifier
- device — per-device counters tracking events and bytes processed by each device
- pipeline — per-pipeline
execution_countandexecution_timeaggregates - device_resource — host resource usage with
resource_typeofvolume,cpu,memory,folder, ornats, plus an identifier, item count, and total/utilized sizes - queue — queue depth and discard counters including pending and discarded files and bytes
- log_severity — per-interval deltas of Warning and Error/Critical debug-log entries, attributed to the device or target that recorded them
Every record carries the full base field set. Fields that do not apply to a given row are present with an empty string or a zero value.
An input type the device has no mapping for still converts, but reaches the target as input_type: "unknown" stripped of its metric columns. The collector logs a warning once per batch when this happens.
Output Fields
The tables below give the emitted field name in each of the three conventions. The ECS column is also the internal representation, used verbatim when field_format is ecs or unrecognized.
Base Fields
Present on every record.
| ECS | Cribl | Prometheus | Description |
|---|---|---|---|
timestamp | _time | timestamp_ms | Record timestamp, RFC3339 with nanosecond precision |
event_time | _event_time | event_time | Time the record was produced, RFC3339 with nanosecond precision |
route_name | cribl_route | route | Route that processed the events |
target_name | cribl_output | target | Destination name |
pipeline_name | cribl_pipe | pipeline | Pipeline name |
source_name | cribl_input | source | Source name |
output_name | _output_name | output | Output identifier |
vendor_name | _vendor | vendor | Vendor name |
processor_name | _processor_name | processor | Processor name |
target_type | _target_type | target_type | Target type identifier |
director_name | cribl_host | instance | Director name |
cluster_name | _cluster_name | cluster | Cluster name |
collector_hostname | _collector_hostname | collector_hostname | OS hostname of the process that produced the record. In containerized deployments this is the replica or pod name, and the only per-replica discriminator |
device_name | _device_name | device | Device name |
device_type | _device_type | device_type | Device type |
device_id | _device_id | device_id | Device identifier |
director_id | _director_id | director_id | Director identifier |
cluster_id | _cluster_id | cluster_id | Cluster identifier |
target_id | _target_id | target_id | Target identifier |
definition_id | _definition_id | definition_id | Definition identifier |
input_type | _input_type | input_type | Record category. One of the seven listed above, or unknown |
type | _type | direction | in or out. Absent on log_severity records |
object_kind | _object_kind | object_kind | device or target. Present only on log_severity records |
Both timestamps are RFC3339 nanosecond strings in all three conventions, including Prometheus, where the field is named timestamp_ms.
type and object_kind are mutually exclusive. Log-severity records omit type rather than carrying a third value in it, so a consumer keyed on type sees an absent key instead of an unexpected enum member.
Route, Target, and Device Fields
| ECS | Cribl | Prometheus | Description |
|---|---|---|---|
events_in | _events_in | events_in_total | Inbound event count |
events_out | _events_out | events_out_total | Outbound event count |
bytes_in | bytes_in | bytes_in_total | Inbound bytes |
bytes_out | bytes_out | bytes_out_total | Outbound bytes |
latency_ns | _latency_ns | latency_nanoseconds | Batch processing wall time, in nanoseconds |
error_count | _error_count | errors_total | Errors recorded in the interval |
dropped_count | _dropped_count | dropped_total | Events dropped in the interval |
target_count | _target_count | target_count | Number of targets involved |
compressed_bytes | _compressed_bytes | compressed_bytes_total | Compressed byte count on out records. On in records the same column carries the average per-record delivery latency in nanoseconds, or 0 when the producer took no samples |
target_identifier | _target_identifier | target_identifier | Target endpoint URL or address. Set on out records |
source_hostname | _source_hostname | source_hostname | Hostname parsed from the received events. Set on in records |
Pipeline Fields
| ECS | Cribl | Prometheus | Description |
|---|---|---|---|
execution_count | _execution_count | execution_count_total | Pipeline executions in the interval |
execution_time | _execution_time | execution_time_nanoseconds | Aggregate execution time, in nanoseconds |
Device Resource Fields
| ECS | Cribl | Prometheus | Description |
|---|---|---|---|
resource_type | _resource_type | resource_type | volume, cpu, memory, folder, or nats |
resource_identifier | _resource_identifier | resource_identifier | Mount point, CPU core, DIMM locator, folder name, or message-store item |
resource_count | _resource_count | resource_count | Number of items of this resource type on the host |
total_size | _total_size | total_size_bytes | Total capacity. See the unit note below |
utilized_size | _utilized_size | utilized_size_bytes | Utilized capacity. See the unit note below |
crash_detected | _crash_detected | crash_detected | 1 when the process restarted after an unclean exit, otherwise 0 |
startup_detected | _startup_detected | startup_detected | 1 on the first resource record after a process start, otherwise 0 |
total_size and utilized_size are reported in kilobytes for volume, memory, and folder, and in megahertz for cpu. The nats resource type overloads both columns per identifier, reporting connection and message-store counters rather than a capacity. The _bytes suffix the Prometheus convention gives these two fields is a naming convention, not a unit.
On device-resource records source_name and target_name are always empty: the underlying columns carry the resource type and identifier instead.
Queue Fields
| ECS | Cribl | Prometheus | Description |
|---|---|---|---|
queue_pending_files | _queue_pending_files | queue_pending_files | Files pending in the queue |
queue_pending_bytes | _queue_pending_bytes | queue_pending_bytes | Bytes pending in the queue |
queue_discarded_files | _queue_discarded_files | queue_discarded_files_total | Files discarded from the queue |
queue_discarded_bytes | _queue_discarded_bytes | queue_discarded_bytes_total | Bytes discarded from the queue |
Log Severity Fields
| ECS | Cribl | Prometheus | Description |
|---|---|---|---|
log_warning_count | _log_warning_count | log_warnings_total | Warning entries recorded since the previous emit |
log_error_count | _log_error_count | log_errors_total | Error and Critical entries recorded since the previous emit |
The attributed object rides in device_id, device_name, and device_type for both kinds: on a target record those three columns carry the target's identifier, name, and type. object_kind says which kind they describe. A record is emitted only when at least one of the two counts is non-zero.
Output Formats
Three field naming conventions are supported:
- ecs — flat snake_case internal representation using ECS-aligned field names such as
route_name,target_name,latency_ns, anderror_count - cribl — Cribl Stream internal-metrics naming, with
cribl_*system fields (cribl_route,cribl_output,cribl_pipe,cribl_input,cribl_host) and underscore-prefixed custom fields (_time,_type,_events_in,_error_count).bytes_inandbytes_outare first-class Cribl fields and carry no prefix - prometheus — Prometheus base-unit naming with
_total,_nanoseconds, and_bytessuffixes (events_in_total,latency_nanoseconds,bytes_in_total), andinstanceanddirectionas labels
Deprecated Properties
poll_interval and workers are accepted but control nothing. The device stopped polling the stats database on a timer when it moved to a durable JetStream consumer, so neither property has anything left to configure. Unknown device properties are never rejected, so configurations that still set them remain valid, and neither is validated any more.
Delivery
Each device attaches one durable consumer to the stats stream under the role device-<id>, so every stats device reads the stream independently with its own position and catches up from the earliest retained batch on start. The stream retains roughly five minutes of batches: a device stopped for longer resumes from the live edge, losing the interval in between.
Delivery is best-effort. A batch that fails to convert is acknowledged and dropped rather than redelivered, so a single bad batch cannot stall the consumer or arrive twice.
Lifecycle
The device has no external network connectivity. On start it reports ConnectionStateConnected and subscribes to the JetStream stats stream with its per-device durable consumer, retrying with backoff until JetStream is ready. If 120 seconds pass without an internal heartbeat, the device reports ConnectionStateErrorHeartbeat and stops the collector.
Restart Triggers
Changing field_format triggers a clean collector restart. It is read once and handed to the poller at startup. Changes to other configuration fields do not restart the collector.
Examples
Basic Configuration
Creating a basic statistics publisher with default Cribl output naming... | |
Route records arrive with Cribl field naming... | |
ECS Format
Configuring flat ECS field naming for the emitted records... | |
Route records use ECS-aligned snake_case field names... | |
Cribl Format
Configuring Cribl Stream internal-metrics field naming explicitly... | |
Target records use Cribl system fields with underscore-prefixed custom fields... | |
Prometheus Format
Configuring Prometheus base-unit field naming for the emitted records... | |
Route records use Prometheus naming with | |
Log Severity Records
Collecting per-device Warning and Error counts alongside the throughput metrics... | |
The record identifies the attributed object through | |
Device Resource Records
Host resource usage arrives as one record per volume, core, or memory module... | |
Sizes are in kilobytes for volumes, so this reports a 100 GB mount at 50 percent utilization... |