Service Configuration

This reference covers the configuration files and fields available for self-managed Director deployments. Managed Directors receive their configuration from the DataStream platform automatically; the fields documented here are relevant when you manage Director infrastructure directly.

For Director concepts and GUI-based setup, see Overview. For installation procedures, see Deployment.

The reference is split across three pages:

Page	Covers
Service Configuration (this page)	vmetric.yml — identity, network, queue, processing defaults, persistent storage
Environment Files	config/environments/*.yml — topology, clustering, ACL, TLS, VIP
Health and Limits	Health check thresholds and the hardcoded internal timing constants

Configuration Files

A Director reads two categories of configuration at startup:

File	Location	Purpose
Service configuration	vmetric.yml in the Director working directory	Core service settings (identity, network, queue, processing defaults)
Environment configuration	config/environments/*.yml	Topology, clustering, ACL, and TLS

Supported formats: YAML (.yml / .yaml), TOML (.toml), and VMF (.vmf). Maximum file size: 1 MB.

Config file lookup order (first match wins):

1. vmetric.toml
2. vmetric.yml
3. vmetric.yaml

Debug Mode

When the virtualmetric=debug environment variable is set, the service loads vmetric-local.yml (or .toml / .yaml) instead of the standard file. This is intended for local development and troubleshooting only -- do not use in production.

Minimal Standalone Director

The smallest working service configuration: one Director, one listener port, console debug output.

vmetric.yml:

director:
  id: 1
listen:
  address: "0.0.0.0"
  port: 8080
debug:
  level: 1
  console:
    status: true

This file is not sufficient on its own -- a Director also needs an environment file declaring its topology, with a node name matching director.id. For the matching config/environments/env.yml, see Standalone Director in Environment Files.

---

Director Identity

Field	YAML Path	Type	Default	Description
ID	director.id	int64	--	Unique Director instance identifier
Status	director.status	bool	true	Enable or disable the Director service
Stream	director.stream	string	--	NATS stream name for Director replies

Network

Field	YAML Path	Type	Default	Description
Address	listen.address	string	0.0.0.0	Bind address for all listeners
External Address	listen.external_address	string	--	Advertised address for remote connections (comma-separated for multiple)
Port	listen.port	int	8890	Main collector listening port
WebSocket Port	listen.ws_port	int	9080	VMMQ WebSocket port
HTTP Config Port	listen.httpconfigport	int	80 / 443	HTTP configuration server port

Queue

Field	YAML Path	Type	Default	Description
Limit	queue.limit	int	172800 (48 h)	Queue size limit in seconds
Parallelism	queue.parallelism	int	CPU count	Queue processing parallelism
Non-Existing Target Limit	queue.non_existing_target_limit	int	86400 (24 h)	TTL for queued data to non-existing targets (seconds)
Cleanup Interval	queue.cleanup_interval	int	900 (15 min)	Cleanup interval for non-existing target queues (seconds)
DeQueue Interval	queue.dequeue_interval	int	1	DeQueue processing interval in seconds

Production Standalone with ACL

A single-node production Director: an advertised external_address for remote collectors, a raised queue limit, and file logging in place of console output.

vmetric.yml:

director:
  id: 1
listen:
  address: "0.0.0.0"
  external_address: "10.0.1.100"
  port: 8890
queue:
  limit: 900000000
debug:
  level: 1
  console:
    status: false
  log:
    status: true

The access control rules that accompany it are set in the environment file -- see Production Standalone with ACL in Environment Files.

Debug and Logging

Field	YAML Path	Type	Default	Description
Level	debug.level	int	0	Debug verbosity: 0 = off, 1--3 = increasing detail
Console Status	debug.console.status	bool	false	Enable console debug output
Log Status	debug.log.status	bool	false	Enable file-based debug logging
Remote	debug.remote	bool	false	Enable remote debug mode (disables device tasks when true)

Fleet

Fleet settings control how a Director connects to the DataStream cloud platform. Self-managed Directors that operate independently (no fleet connection) set self_managed: true and omit the fleet block.

Field	YAML Path	Type	Default	Description
Self-Managed	self_managed	bool	--	If true, Director runs independently without fleet connection
Fleet Type	fleet.type	string	--	Connection type (e.g., httpclient)
Fleet Instance	fleet.instance	string	--	Fleet environment instance name
Fleet Address	fleet.address	string	--	Fleet server WebSocket URL
Fleet Token	fleet.token	string	--	Authentication token for fleet

Fleet-Managed Director

A Director that draws its configuration from the DataStream platform: self_managed: false with a populated fleet block. No environment file is required -- topology comes from the fleet.

vmetric.yml:

self_managed: false
fleet:
  type: httpclient
  instance: production
  address: wss://fleet.virtualmetric.io/ws
  token: "fleet-auth-token-abc123"
director:
  id: 2028574672450228224
listen:
  address: "0.0.0.0"
  port: 8890
debug:
  level: 1
  console:
    status: true

Data Processing Defaults

These defaults apply to all data processing unless overridden at the device or target level.

Field	YAML Path	Type	Default	Description
Compression	compression	bool	true	Enable data compression
Compression Level	compression_level	string	--	Compression level (algorithm-specific)
Reuse	reuse	bool	true	Enable connection reuse
Normalization	normalization	bool	true	Enable data normalization
Smart Data Engine	smart_data_engine	bool	true	Enable SDxL processing
No Buffer	no_buffer	bool	false	Disable buffering
Batch Size	batch_size	int64	10000	Read batch size
Write Batch Size	write_batch_size	int64	1000	Write batch size
Workers	workers	int	CPU count	Worker count
Chunk Size	chunk_size	int	4194304 (4 MB)	Data chunk size in bytes
Min Chunk Count	min_chunk_count	int	10	Minimum chunk count
Stats Enabled	stats_enabled	bool	false	Enable per-device, per-route, and per-target metrics collection. Fleet-managed Directors override this to true automatically.
Stats Drop On Full	stats_drop_on_full	bool	true	When the stats channel buffer is full, drop new metrics rather than blocking the record producer.
Disable Direct	disable_direct	bool	false	Disable the in-process listener-to-target fast path. When false, the direct path engages automatically when eligible.
Max Buffer Size	max_buffer_size	int | string	128MB	Default ingest-buffer admission cap for memory-mode listener devices. Overridable per device. Sets a per-device heap reservation -- see Listener Memory Sizing before raising or lowering it.

Collector Resource Management

Controls resource allocation and scheduling for device collectors.

Field	YAML Path	Type	Default	Description
Retry Interval	retry_interval	int	30	Device retry delay in seconds
Start Delay	start_delay	int	1	Collector startup delay in seconds
Windows Start	windows_start	int	524288000 (500 MB)	Memory threshold for Windows collectors (bytes)
Windows Start Buffer	windows_start_buffer	bool	--	Buffer mode for Windows collectors
Monitor Interval	monitor_interval	int	30	Monitor interval in seconds
Flush Interval	flush_interval	int	1	Log flush interval in seconds
Modules	modules	bool	true	Enable or disable module loading

Listener Memory Sizing

A memory-mode listener reserves its ingest buffers eagerly at start, so its memory cost is a fixed floor that does not depend on traffic. max_buffer_size sets that reservation and the burst admission cap that goes with it.

Set it per device under properties:, or service-wide as the top-level max_buffer_size shown under Data Processing Defaults. The device value wins.

# vmetric.yml -- applies to every memory-mode listener
max_buffer_size: 32MB

devices:
  - id: 1
    name: syslog_edge
    type: syslog
    properties:
      max_buffer_size: 16MB   # this device only; overrides the service value

Accepted values

Value	Result
A bare integer	Byte count
A size string	Binary units. 32MB and 32MiB are exact synonyms -- both mean 32 x 1024². Prefixes KB through PB (and KiB through PiB) are accepted
A positive value below 16MB	Raised to the 16 MiB minimum, and logged. Below it a rotation cycle cannot hold a useful batch
0, a negative value, or an unparseable string	Ignored and logged; the service value, then the default, applies instead

warning

0 is not "unlimited". It would disable the admission cap and let the listener grow without bound, so it is refused. So is a negative or unparseable value -- and because those are discarded rather than rejected, the listener starts on the default with only a log line to record it. Remove the key to take the default deliberately.

Each listener costs three times this value

The reservation splits the cap across shards, gives each shard 1.5x its slice for encode headroom, and holds two VMFL slots per shard. The shard count cancels, so the multiple is 3x on any core count:

max_buffer_size / shards    # shards = NumCPU/4, capped by GOMAXPROCS, clamped to [1,16]
  x 1.5                     # per-shard encode headroom
  x 2                       # two VMFL slots per shard
= 3 x max_buffer_size

At the 128MB default that is 384 MiB of live heap committed before the first record arrives, and roughly twice that as an RSS ceiling once garbage-collector headroom is counted.

warning

The reservation is per listener worker, not per device. With reuse: true -- the default -- a device runs one listener per worker, each with its own buffers, so the real cost is workers x 3 x max_buffer_size. Multiply by the worker count before sizing a host.

Device	Default workers	Worst case at 128MB
syslog, tcp	CPU cores / 8, uncapped	8 workers on a 64-core host = 3 GiB per device
udp, http	1 to 4 on a core-count ladder, capped at 4	4 workers = 1.5 GiB per device
Any device with reuse: false	Forced to 1	384 MiB per device

A syslog device with enable_udp: true runs a second listener for the UDP side and reserves for it separately, doubling the figure again.

The eager commit is deliberate -- it avoids reallocation churn on the ingest hot path -- so lowering the value reduces the reservation rather than deferring it.

max_buffer_size	Heap floor per listener	Suits
16MB (minimum)	48 MiB	Many low-volume devices on one Director
32MB	96 MiB	Moderate volume
128MB (default)	384 MiB	High-volume, wire-speed listeners

Under a cgroup memory or CPU limit the default already scales down on its own, to the smallest of 128MB, one quarter of the memory limit, and 32 MiB per CPU core of quota, floored at the 16 MiB minimum. It is computed once at startup. Bare-metal and virtual-machine hosts get no such scaling and always pay full size, which is what this key is for.

note

Under a cgroup memory limit, a second cap bounds the bytes admitted across all listeners together at one quarter of that limit. Raising max_buffer_size on one device does not necessarily buy it more admission.

What happens when the cap is exhausted

The cap is also the burst admission gate, so lowering it shortens how long a burst may outrun the router. What that costs depends on the protocol:

Device	Behavior
syslog over TCP, tcp	Graceful. The read loop blocks, the TCP window closes, and senders slow down
syslog with UDP, udp, the flow listeners	No backpressure. The read loop stalls and the kernel discards datagrams
http	The request waits up to 30 seconds, then returns 503 with Retry-After: 30
splunk-hec	Same, plus HEC status code 9 (server busy). A batch that times out part-way leaves the already-encoded prefix stored and returns 503 for the whole request, so a client retry duplicates that prefix
otlp	Waits up to 30 seconds, then answers per transport: 503 over OTLP/HTTP, gRPC status UNAVAILABLE over OTLP/gRPC
wec	No backpressure and no rejection. The SOAP endpoint acknowledges the batch before the write outcome is known, so an event that cannot be admitted is dropped with only a log entry

warning

Size UDP and wec listeners conservatively -- they are the two that lose data rather than slowing the sender when the cap is exhausted. UDP has no flow control to fall back on, and wec has already acknowledged the batch by the time the write is attempted. A syslog device with enable_udp: true carries the UDP exposure even when network is tcp.

Interaction with crash resistance

The reservation applies only in memory mode. With persistent_storage.crash_resistance enabled the listener takes the disk-mode path and never pre-commits these buffers, so the key narrows admission but frees nothing. See Persistent Storage.

note

Memory mode is a property of the Director, not of a device: crash_resistance is a service-level flag, so a Director is either entirely memory-mode or entirely disk-mode. There is no per-device escape.

Listeners this applies to

	Device types
Reserve 3x and honor the cap	syslog, tcp, udp, http, otlp, splunk-hec, wec, snmp-trap, smtp, tftp, netflow, sflow, ipfix, elastic, file, every honeypot device, the message-queue devices (kafka, nats, rabbitmq, redis), and the cloud pull devices (amazon-s3, azure-blob-storage, event-hubs, google-cloud-pub-sub)
Honor the cap, reserve nothing	estreamer, ccf, amazon-security-lake, google-cloud-storage, and the database devices (mssql, mysql, oracle, postgres)
Ignore the key entirely	datastream-stats, proofpoint-on-demand, microsoft-graph-api, datagen, azure-monitor, microsoft-sentinel, and the linux and windows agent devices

A syslog or tcp device also skips the reservation when it has no routes, or when every route it feeds targets only discard.

Profiler

Field	YAML Path	Type	Default	Description
Profile Port	profiler.director_profile_port	int	6061	pprof profiler port
Timeout	profiler.timeout	int	60	Profiler and metrics collection timeout in seconds

Load Balancer

Field	YAML Path	Type	Default	Description
Mode	loadbalancer.mode	string	--	Set to dynamic for dynamic load balancing across cluster nodes

Three-Node HA Cluster

loadbalancer.mode: dynamic spreads work across the nodes of a cluster. The same vmetric.yml goes on all three nodes -- only director.id differs.

vmetric.yml:

director:
  id: 1  # Change to 2, 3 on other nodes
listen:
  address: "0.0.0.0"
  external_address: "10.0.1.101"
  port: 8890
queue:
  limit: 900000000
debug:
  level: 2
  console:
    status: true
  log:
    status: true
loadbalancer:
  mode: dynamic

The cluster topology itself -- node addresses, inter-node TLS, ACL, and VIP failover -- is declared in the environment file. See Three-Node HA Cluster with VIP in Environment Files.

Persistent Storage

Controls whether pipeline data survives a restart, and how far upstream that durability extends. The two flags combine into four modes:

persistent_storage.status	crash_resistance	Effect
false (default, standalone)	forced false	In-memory VMFL/VMF and JetStream MemoryStorage for the pipeline streams. Maximum throughput; records buffered in process memory are lost on a crash.
true	false	JetStream FileStorage plus the NATS Object Store for payloads. VMFL/VMF writers stay in-memory; durability is provided by NATS.
true	true	Enterprise: local VMFL/VMF queue writers also go to disk, flushing each record to the OS page cache -- process-crash durable end-to-end.
false	true	Ignored -- crash_resistance is forced false and the mode collapses to the first row. The override is silent: no warning is logged.

note

The first row applies to the three pipeline streams (ingester, router, sender). The two fleet control streams and the Director data stream are created with FileStorage regardless of persistent_storage.status, so some disk is reserved in every mode.

Field	YAML Path	Type	Default	Description
Status	persistent_storage.status	bool	false	Enable the durable NATS JetStream pipeline layer (JetStream FileStorage + NATS Object Store). Defaults to true on fleet-managed Directors.
Crash Resistance	persistent_storage.crash_resistance	bool	false	Also write the local VMFL/VMF queue files to disk, flushing each record to the OS page cache so the upstream pipeline hops survive a process crash (SIGKILL, OOM, panic). Enterprise deployments; only effective when persistent_storage.status: true (forced false otherwise).
Max Age	persistent_storage.max_age	numeric	172800	Retention horizon in seconds for the durable streams (48 hours). A non-positive value falls back to the default.

note

crash_resistance also decides how listeners allocate their ingest buffers: memory mode reserves them eagerly, disk mode does not. See Listener Memory Sizing.

note

crash_resistance targets process-crash durability, not power loss. With persistent_storage.status: true, power-loss durability is provided by NATS -- best-effort on a single node (default 2-minute sync interval) or per-write via RAFT quorum on a cluster. For power-loss-sensitive workloads, run a cluster.

In clustered or environment-file deployments, place the persistent_storage block -- including its nested payload and pipeline_bus -- under a node's or the cluster's properties: block rather than at the top level of vmetric.yml. See Embedded NATS below, and External Kafka or Redpanda and Azure Event Hubs under Pipeline Bus.

Cluster properties are applied first and node properties are applied on top, so a value set on a node overrides the cluster value for that node. Nothing validates consistency across a cluster: divergent values are accepted silently and each node runs with its own resolved flags. Keep them uniform -- nodes of a cluster share the same NATS and pipeline layer, and mixed durability settings produce behavior that is difficult to reason about.

For Orchestrated Directors, the processing mode and the external storage connection are also configurable through the web interface -- see Persistent Storage.

Embedded NATS

Durable pipeline on the embedded NATS backend -- the default when no payload or pipeline_bus is set. JetStream switches to FileStorage and payloads move to the NATS Object Store; VMFL/VMF writers stay in-memory. In a cluster, put these settings on the cluster's properties (they must be uniform across all nodes), not per-node.

config/environments/env.yml:

environments:
  - name: "1"
    status: true
    nodes:
      - name: "1"
        status: true
        properties:
          persistent_storage:
            status: true

For the Enterprise crash-resistant mode, add crash_resistance: true -- the local VMFL/VMF queue writers also go to disk, flushed per record:

persistent_storage:
  status: true
  crash_resistance: true

Payload

Configures the payload backend that carries log record bytes between pipeline stages. The payload and pipeline_bus blocks nest inside persistent_storage; the full key paths are persistent_storage.payload.* and persistent_storage.pipeline_bus.*.

Field	YAML Path	Type	Default	Description
Type	persistent_storage.payload.type	string	""	Payload backend: inline (bytes carried in the pipeline message), object_store (NATS Object Store), memory (in-process store), or "" (automatic — memory when persistent storage is off, object_store when on).
Max Size	persistent_storage.payload.max_size	string	""	Maximum payload size per record. Accepts a plain integer (bytes) or a size suffix such as 1mb or 512kb. When the payload type is inline defaults to 1 MB to fit within Kafka and Event Hubs message limits; unlimited for other backends.

Pipeline Bus

Configures the transport that moves stage-to-stage pipeline notifications between the ingester, router, and sender workers.

Field	YAML Path	Type	Default	Description
Type	persistent_storage.pipeline_bus.type	string	nats	Stage-to-stage transport: nats (embedded VMMQ, default), kafka (external Kafka broker), or eventhubs (Azure Event Hubs, treated as kafka against its Kafka-compatible endpoint). An unrecognized value logs a warning and falls back to nats.
Address	persistent_storage.pipeline_bus.address	string	--	Comma-separated Kafka broker addresses. Required when the pipeline bus type is kafka.
Topic Prefix	persistent_storage.pipeline_bus.topic_prefix	string	vm-pipeline	Prefix for stage topics. Topics are named <prefix>.<stage> (e.g., vm-pipeline.sender).
Algorithm	persistent_storage.pipeline_bus.algorithm	string	--	Kafka SASL algorithm: plain, scram-sha-256, scram-sha-512, or gssapi.
Username	persistent_storage.pipeline_bus.username	string	--	Kafka SASL username.
Password	persistent_storage.pipeline_bus.password	string	--	Kafka SASL password.
TLS Status	persistent_storage.pipeline_bus.tls.status	bool	false	Enable TLS for the Kafka broker connection.
TLS Insecure Skip Verify	persistent_storage.pipeline_bus.tls.insecure_skip_verify	bool	false	Skip broker certificate verification.
TLS Min Version	persistent_storage.pipeline_bus.tls.min_tls_version	string	tls1.2	Minimum TLS version for the broker connection.
TLS Max Version	persistent_storage.pipeline_bus.tls.max_tls_version	string	tls1.3	Maximum TLS version for the broker connection.

note

pipeline_bus.type: kafka requires payload.type: inline. If payload.type is not inline, the Director logs a warning and reverts to the nats transport.

External Kafka or Redpanda

Routes the router-to-sender hop over an external Kafka-compatible broker. Requires payload.type: inline -- with any other payload type the bus silently falls back to nats. The topic (vm-pipeline.sender) and consumer group (<director-name>-sender) auto-create where the broker allows it.

config/environments/env.yml:

environments:
  - name: "1"
    status: true
    nodes:
      - name: "1"
        status: true
        properties:
          persistent_storage:
            status: true
            payload:
              type: inline
              max_size: 512kb
            pipeline_bus:
              type: kafka
              address: "{broker-1}:9092,{broker-2}:9092"
              topic_prefix: vm-pipeline
              # Optional SASL auth -- omit the block below for a plaintext broker.
              # algorithm: plain            # plain | scram-sha-256 | scram-sha-512 | gssapi
              # username: "{username}"
              # password: "{password}"
              # tls:
              #   status: true

The inline payload rides base64-encoded inside the bus message, so max_size caps the raw payload before encoding (~4/3 on the wire). 512kb (~683 KB encoded) stays under a default broker's ~1 MB message.max.bytes; raise it only in step with the broker limit. For crash resistance, add crash_resistance: true under persistent_storage (the shipped example also raises max_size to 1mb).

Azure Event Hubs

Uses the Event Hubs Kafka endpoint as the pipeline bus -- the durable-broker option for ephemeral deployments (e.g. Azure Container Apps) with no durable local disk, where in-flight data survives a container restart inside the managed broker. SASL PLAIN over TLS on port 9093.

config/environments/env.yml:

environments:
  - name: "{director-id}"
    status: true
    nodes:
      - name: "{director-id}"
        status: true
        properties:
          persistent_storage:
            status: true
            payload:
              type: inline
              max_size: 512kb
            pipeline_bus:
              type: kafka
              address: "{namespace}.servicebus.windows.net:9093"
              topic_prefix: vm-pipeline
              algorithm: plain
              username: "$ConnectionString"
              password: "Endpoint=sb://{namespace}.servicebus.windows.net/;SharedAccessKeyName={policy};SharedAccessKey={key}"
              tls:
                status: true

Event Hubs does not auto-create topics: pre-create the event hub vm-pipeline.sender and the consumer group <director-name>-sender before starting the Director -- a missing topic drops 100% of pipeline traffic. Keep payload.max_size at or below the namespace tier's message cap (Basic 256 KB, Standard/Premium 1 MB, Dedicated 20 MB), measured on the raw payload before base64. Add crash_resistance: true under persistent_storage for the Enterprise crash-resistant mode.

---

Environment Variables

Variable	Value	Effect
virtualmetric	debug	Loads vmetric-local.yml instead of vmetric.yml

Debug config file lookup order:

1. {serviceName}-local.toml
2. {serviceName}-local.yml
3. {serviceName}-local.yaml