Persistent Storage
Synopsis
Persistent storage determines whether pipeline data survives a restart, and where the Director buffers that data. It is administered through two surfaces:
- The Director creation wizard and detail view in the web interface — covered in the sections below.
- The YAML configuration reference — for self-managed and infrastructure-as-code deployments. The
persistent_storageblock, its durability mode matrix, and the payload and pipeline bus settings are documented under Service Configuration.
The web interface exposes a subset of the YAML surface: it sets the processing mode, the external storage connection, and five buffer settings -- retention, the total size limit, the pipeline lag limit, the queue size limit and the inline payload size (see Buffer Settings). The payload backend type, the per-queue size keys under persistent_storage.streams and the stats stream settings are YAML-only.
Availability
The storage type and connection settings appear only when both of the following hold:
- The Director mode is Orchestrated or Serverless.
- The processing mode is Persistent or Crash-resistant.
Managed and Self-managed Directors keep their persistent queue on the host's own disk, so they have no external storage to connect to. Selecting In-memory hides the settings, since no external storage is involved.
An Orchestrated Director always uses the configured broker as its pipeline bus. A Serverless Director does so only when its connection names a broker address; with the address left empty it keeps its queue inside the Director. With the Built-in queue storage type, either kind keeps its queue in its shared state storage instead.
The Buffer settings block is broader than the connection form: it appears in every Director mode once the processing mode is Persistent or Crash-resistant, because every mode then keeps a persistent queue. See Buffer Settings.
A Director whose pipeline bus is a broker, Apache Kafka or Azure Event Hubs, saves its position with the broker once a second, not at every acknowledgement, and only up to the first record that has not finished. After a crash it reads again what it delivered since its last save, normally less than a second's worth, and more while saves fail or one unfinished record holds the position back: duplicates, never a loss. See Saved Position.
Accessing
Navigate to Home > Fleet Management > Directors and click
To change the connection or buffer settings on an existing Director, open it from the Directors table, stay on the
Processing Modes
Under
| Mode | Description |
|---|---|
| In-memory | Highest throughput with the lowest latency, using RAM as the buffer. Capacity is limited to available memory. |
| Persistent | Extends buffer capacity to your disk size and ensures data durability across process restarts. Built for handling longer outages without risking data loss. |
| Crash-resistant | Guarantees data recovery even after OS crashes or power loss. Built for pipelines where losing a single event is not an option. |
The processing mode maps onto the persistent_storage.status and persistent_storage.crash_resistance flags described under Service Configuration. The web interface does not expose those flags directly.
Persistent and Crash-resistant are tier-gated. Without the corresponding entitlement, the tile is disabled and carries an
The default selection is Persistent when that tier is available, and In-memory otherwise. It is never Crash-resistant by default, even when that tier is available.
The processing mode is fixed for the life of the Director process. A Director that receives its configuration from the platform applies the mode from the first configuration delivered to it, which is why a brand-new one waits for that delivery before it builds its pipeline rather than starting on a guess — see Director Waiting for Its First Configuration.
Shared State Storage
The replicas of a Serverless Director, and the machines of an Orchestrated Director, share some state: which one leads, which one runs which device, how far each device has read, and the bookmarks of the Windows Event Collector.
| Choice | What it means | Fields |
|---|---|---|
| In memory | The replicas share nothing. When a replica stops, its state is lost, and devices that need shared state can't run on the Director. | None |
| Azure Blob Storage | The state lives in a container of an Azure storage account. | Storage account, Container, Prefix (Optional), Authentication method, Client ID (Optional, with a managed identity) |
| Amazon S3 | The state lives in an S3 bucket, or in a store that works like S3, such as MinIO. | Bucket, Prefix (Optional), Region, Endpoint (Optional), Path-style addressing, Authentication |
| Redis | The state lives in a Redis server that keeps its data. | Address, Port, TLS, Username, Password (Optional), Database (Optional) |
| Azure Managed Redis | As Redis, but the Director signs in with a Microsoft Entra identity, so no secret is stored. | Address, Port (10000 by default), Authentication method, Client ID (Optional) |
| Network share | The state lives in a folder that every virtual machine of the Director mounts. Orchestrated Directors only. | Path |
A new Director starts with In memory. A Director created before this setting existed shows
The Stores
Azure Blob Storage signs in with one of five methods: an Azure managed identity, the Default Azure credential, a connection string, an account key, or a shared access signature (SAS). A connection string names its own account, so
- Azure Container Apps and Azure Container Instances: the install template gives the identity to the container. With a
Client ID , the template asks for that identity's resource ID, in the parametersharedStateIdentityId. Without one, the container signs in as its own system-assigned identity, which exists only after the first deployment, so grant its role after that. - Kubernetes: the Director signs in as the service account's Azure workload identity. Turn on
Run under a dedicated service account in the advanced settings and enter the identity's client ID there. - Docker: a managed identity can't be used, because a compose file can't give the container an Azure identity.
- Orchestrated Directors: the managed identity of the Azure virtual machines or scale set the Director runs on.
Amazon S3 signs in with the Default AWS credentials, the ones the Director finds where it runs, such as the IAM role of its virtual machine or pod, or with an Access key. Either way they need to read, write, list and delete the objects in the bucket. Turn on us-east-1.
Redis and Azure Managed Redis must keep their data: turn on persistence, and set the eviction policy to noeviction. Otherwise the replicas lose their shared state when the store restarts or runs out of memory. These stores hold shared state only, so the built-in queue can't run in them. An identity for Azure Managed Redis needs read and write access to the instance through Microsoft Entra authentication.
Network share takes the absolute path at which every virtual machine of the Director mounts the folder. The Director must be able to read and write it.
Create the bucket and the network share's folder before you choose them. The Director creates a blob container only when its identity is allowed to create one.
On Azure Container Apps and Azure Container Instances, the identity the shared state signs in with is part of the Director's deployment. After changing it, re-install the Director with the regenerated template; the console says so before the save.
Write Interval
Every store but In memory has a
A longer interval means fewer writes, and so fewer billed calls on Azure Blob Storage and Amazon S3. The cost is what a crash loses: after a crash, the Director may collect up to one interval of data again, and with the built-in queue it can lose up to one interval of queued data.
Crash-resistant mode writes every change to the store before it confirms it, so it ignores the interval, and the console hides the field.
Devices That Need Shared State
Some devices keep their progress between restarts: a cursor, a checkpoint, a bookmark, or the last run of a schedule. When such a device moves to another replica, that replica must be able to read the progress, so these devices can't run on a Director whose shared state is In memory:
- The Windows Event Collector and eStreamer
- The cloud bucket and stream readers: Amazon S3, Azure Blob Storage, Google Cloud Storage, Amazon Kinesis and Amazon CloudWatch
- The database readers
- The API pollers: Microsoft Graph API, CCF and the vendor devices built on it, Microsoft Sentinel incidents, Proofpoint, and the virtualization and Kubernetes pollers
- The File device, and the Website and service check devices
- Windows, Linux, Oracle Solaris, AIX and macOS hosts read without an agent
Choosing In memory shows the warning
When a Store Is Chosen
- A replica starts the devices that need shared state only once it reaches the store. Devices that keep no shared state, such as a syslog listener, start at once. A wrong secret or a store the Director can't reach keeps the other devices stopped, and the Director's health reports why.
- A change of store starts the shared state afresh: devices start again from their first position. A new secret for the same store keeps the state.
What It Costs
Azure Blob Storage and Amazon S3 bill every call. One write call a second costs about $17 a month on Azure's hot tier and about $13 a month on S3 Standard, and both price a list call like a write. The calls below were counted with three replicas on an in-memory store, so they are not Azure or S3 figures. On Blob Storage and S3 each call takes tens of milliseconds, so a crash-resistant Director combines more writes into each call there.
| Set-up | Writes what changed | Write calls per second | Read and list calls per second |
|---|---|---|---|
| Shared state, no devices | every 10 seconds (the default) | 0.6 | 1.8 |
| Shared state, 20 devices, a checkpoint every second | every 10 seconds (the default) | 0.7 | 1.8 |
| Shared state, 20 devices, a checkpoint every second | every second | 1.7 | 1.8 |
| The same, crash-resistant | at once | 15.0 | 1.8 |
| Built-in queue, 100 or 1,000 messages of 1 KiB a second | every second (the default) | 1.6 | 1.8 |
| Built-in queue, 1,000 messages a second, crash-resistant | at once | 1,000 | 1.8 |
Most of the calls of a quiet Director are its lease, which does not depend on the write interval. For 20 devices, the 10-second default costs about $8 to $16 a month less on Azure than an interval of 1 second, and about $6 to $12 less on S3.
The shared_state keys the console sets are listed under Service Configuration.
Storage Type
When the processing mode is Persistent or Crash-resistant,
| Type | Description |
|---|---|
| Apache Kafka | Cloud-agnostic distributed streaming. High availability, durability, and strong delivery guarantees. |
| Azure Event Hubs | Azure-native event streaming. Managed identity support and tight Azure integration. |
| Built-in queue | The Director's own queue runs in its shared state storage. Every write to Azure Blob Storage or Amazon S3 is a billed operation, so the storage cost can be high under load. |
Built-in queue is offered only when the shared state storage is Azure Blob Storage, Amazon S3 or a network share, which can hold the queue; Redis and Azure Managed Redis hold shared state only. A Built-in queue chosen before the store changed to one that can't hold it reads as no choice. The tile has no connection form of its own: the queue uses the shared state storage's settings. See What It Costs for the calls it makes.
Selecting Apache Kafka or Azure Event Hubs reveals its connection form.
Apache Kafka
The following fields are used to configure the Kafka connection:
| Field | Required | Default | Description |
|---|---|---|---|
Broker Address (Optional) | N | 0.0.0.0 | Kafka broker address |
Port | Y | 9092 | Kafka broker port. Must be between 1 and 65535 |
Topic prefix | Y | - | Prefix used for the topics this Director writes to |
Algorithm (Optional) | N | None | SASL mechanism. Leave as None to disable authentication. Accepts Plain, SCRAM-SHA-256, SCRAM-SHA-512, and GSSAPI (Kerberos) |
Username (Optional) | N* | - | SASL username |
Password (Optional) | N* | - | SASL password |
TLS encryption | N | Disabled | Enable TLS for the broker connection |
* = Shown only when an Algorithm other than None is selected
Username and Password accept a value entered directly, an environment variable, or a Vault secret, selected from the dropdown beside each field.
Azure Event Hubs
The following fields are used to configure the Event Hubs connection:
| Field | Required | Default | Description |
|---|---|---|---|
Address | N | - | Event Hubs Kafka endpoint address |
Topic prefix | Y | - | Prefix used for the event hubs this Director writes to |
Connection string | Y | - | Event Hubs connection string |
Connection string accepts a value entered directly, an environment variable, or a Vault secret.
The Event Hubs form asks for three values where Kafka asks for seven. The Director supplies the rest when it saves the configuration: the SASL mechanism is set to plain, the username to $ConnectionString, and TLS is enabled. These are fixed for the Event Hubs Kafka endpoint and cannot be changed from the web interface.
Buffer Settings
Undelivered events wait in the Director's persistent queues until their destination acknowledges them. Five settings control this buffer. Retention, Total size limit and Queue size limit bound how much can accumulate while a destination is down. Pipeline lag limit keeps the Director's own pipeline from falling too far behind. Inline payload size limits the size of a single payload. They sit in a collapsed block headed
The block appears for every Director mode when the processing mode is Persistent or Crash-resistant, and not at all for In-memory. Each field is optional: left empty, the Director's own default applies.
| Field | Units | Left empty | Description |
|---|---|---|---|
Retention | Seconds, Hours, Days | 48 hours | The ceiling on how long an undelivered event survives a downstream outage. Delivered data is removed as soon as it is acknowledged, so this bounds only what is still waiting. Sets persistent_storage.max_age, in seconds |
Total size limit | Bytes, MB, GB, TB, % of disk | 75% of the disk space the Director can use (see below) | The most undelivered data the Director keeps on its disk. When the Director reaches it, it deletes the oldest undelivered data first. Retention still applies, so the Director also deletes data that is older than the retention period. Sets persistent_storage.max_size, as a size such as 200GB or as a share of the disk such as 50%. Not shown on Orchestrated Directors, or on Serverless Directors whose connection names a broker address |
Pipeline lag limit | Seconds, Minutes, Hours | 60 seconds | How far the Director's pipeline may fall behind. The Director measures this as the time its pipeline needs to process the data that is waiting. Beyond the limit, the Director takes data in only as fast as the pipeline processes it. To do so, it pauses its sources for a moment at a time. 0 turns the limit off. Sets persistent_storage.max_lag, in seconds. Not shown on Orchestrated Directors, or on Serverless Directors whose connection names a broker address |
Queue size limit | Bytes, MB, GB, TB | Each queue's own default | A disk ceiling for each of the Director's three pipeline queues (ingester, router and sender), not a total shared between them. When a queue fills, its oldest waiting events are dropped so delivery keeps running. Each queue's cap is reserved on disk up front, so the caps together have to fit in the free disk. A size set for one queue under persistent_storage.streams in YAML replaces this figure for that queue. Sets persistent_storage.max_bytes, in bytes. Not shown on Orchestrated Directors |
Inline payload size | KB, MB | 512 KB with Persistent, 1 MB with Crash-resistant | The largest payload carried inside a pipeline bus message. Applies only to a Director that uses a broker as its pipeline bus (see Availability). Sets persistent_storage.payload.max_size, as a suffixed string such as 512kb |
Each value must be a whole number. MB, GB and TB are binary multiples: 1 GB is 1,073,741,824 bytes.
Retention and Queue size limit reach the Director in every mode. Left empty, the queue size limit falls back to each queue's shipped default: 1 GB for the sender queue and 512 MB each for the ingester and router queues, scaled down to fit when the Director detects a disk budget. A size set for one queue under persistent_storage.streams in YAML takes precedence over this field for that queue -- see Service Configuration.
Total size limit and Retention are two separate limits, and both apply. The Director deletes undelivered data when it is older than the retention period, or when the stored data reaches the total size limit, whichever comes first. A percentage is a share of the disk space the Director can use: the free space on the disk that holds its queues, plus the space its stored data already takes there. The limit is not exact. The Director checks the total every 10 seconds, so the stored data can pass the limit by what arrives between two checks. Set it below the disk space you can give the stored data.
Left empty, the total size limit shows no limit, but the Director still applies a default: 75% of the disk space it can use. On a platform with a storage budget, such as Azure Container Apps, the default is the part of that budget the Director's own queues leave free. Every deletion is counted: the Director logs a warning, and the Data loss card on its
Pipeline lag limit keeps the Director from taking in more data than its pipeline can process. It does not hold data back for a target that is down or slow. That data waits on disk, within the retention period and the total size limit. While the Director pauses its sources, a TCP sender waits, and an HTTP sender gets the answer 503 with a Retry-After header. The HTTP sender must then send the request again. Sources whose senders cannot be asked to wait, such as syslog over UDP, are never paused. A change to the limit takes effect without a restart.
Inline payload size has an effect only where the payload rides inside the broker's messages: on an Orchestrated Director, and on a Serverless Director whose connection names a broker address, in both cases with Apache Kafka or Azure Event Hubs as the storage type. Keep it at or below the broker's message limit. Managed and Self-managed Directors, and a Serverless Director without a broker address, keep their payloads in the NATS Object Store instead, which has no inline cap: the field is shown and saved for them, but the value is not applied. With the Built-in queue storage type, the Director receives no inline payload size at all, so the value is not applied there either.
The helper text under Inline payload size reads 512 KB only for an Orchestrated Director in Persistent mode. A Serverless Director that uses a broker as its pipeline bus also applies 512 KB in Persistent mode when the field is left empty, although its helper text reads 1 MB.
The default: 48h, no limit for the total size limit, default: 60 seconds for the pipeline lag limit, automatic, and default: 512 KB or default: 1 MB. A pipeline lag limit of 0 shows no limit.
Editing After Creation
Open the Director from the Directors table and click
The connection fields for the selected backend remain editable. The following are fixed at creation and display as read-only rows:
- Director mode
- Processing mode
- Persistent storage type
Changing any of these requires creating a new Director.
The buffer settings stay editable, with one exception. On a Self-managed Director the fields are shown disabled with their current values, under the note persistent_storage keys listed in Service Configuration.