Azure Blob Storage
Synopsis
Azure Blob Storage device reads files from an Azure Blob Storage container by dequeuing messages from an Azure Storage Queue that notifies the device of new blobs.
Schema
- id: <numeric>
name: <string>
description: <string>
type: azblob
tags: <string[]>
pipelines: <pipeline[]>
status: <boolean>
properties:
container_name: <string>
connection_string: <string>
tenant_id: <string>
client_id: <string>
client_secret: <string>
account: <string>
blob_size: <numeric>
mode: <enum>
prefix: <string>
workers: <numeric>
timeout: <numeric>
batch_size: <numeric>
Configuration
The following fields are used to define the device:
Device
| Field | Required | Default | Description |
|---|---|---|---|
id | Y | Unique numeric identifier | |
name | Y | Device name | |
description | N | - | Optional description |
type | Y | Device type identifier (must be azblob) | |
tags | N | - | Array of labels for categorization |
pipelines | N | - | Array of preprocessing pipeline references |
status | N | true | Enable/disable the device |
Connection
The device requires either a connection string or a service principal with account name. container_name is required in all cases.
| Field | Required | Default | Description |
|---|---|---|---|
container_name | Y | Azure Storage Queue name used for blob notifications | |
connection_string | Y* | Azure Storage connection string for authentication | |
tenant_id | Y* | Azure tenant ID for service principal authentication | |
client_id | Y* | Azure client ID for service principal authentication | |
client_secret | Y* | Azure client secret for service principal authentication | |
account | Y* | Azure storage account name for service principal authentication | |
blob_size | N | 100000000 | Maximum blob size in bytes to process (100 MB) |
mode | N | queue | Ingestion mode: queue or poll. See Ingestion Modes |
prefix | N | - | Restrict ingestion to keys under this prefix. In poll mode it may carry date tokens |
workers | N | max(NumCPU/4, 1) | Concurrent download and decode workers, 1-64 |
timeout | N | 5 | Idle backoff in seconds when the notification queue is empty (1-3600) |
batch_size | N | 32 | Maximum queue messages retrieved per dequeue (1-32; Azure caps at 32) |
* = Conditionally required: either connection_string or the combination of tenant_id, client_id, client_secret, and account must be provided.
Ingest Buffer
| Field | Required | Default | Description |
|---|---|---|---|
max_buffer_size | N | 128MB | Ingest buffer admission cap. Also sets this listener's eager heap reservation |
max_buffer_size is the ingest-buffer admission cap, and is distinct from any buffer_size field above, which sizes the network read buffer. Left unset it is 128MB, which commits 384 MiB of heap per listener worker before any data arrives — and with reuse: true a device runs one worker per listener. Size strings are binary, and 32MB and 32MiB are exact synonyms.
See Listener Memory Sizing for the arithmetic, the sizing table, and what happens when the cap is exhausted.
Details
Ingestion Modes
This device offers two ingestion modes, selected with mode:
mode | Behaviour |
|---|---|
queue | Default. The device consumes notifications from the bucket's event queue and fetches each object as it is announced. |
poll | The device lists the bucket on a schedule and ingests what it has not seen before. No queue is needed. |
Any other value fails with mode field not valid.
Poll mode suits a bucket with no notification wiring, and a historical backfill, since a listing can be bounded to a date range. It costs a list operation per cycle, so it is the wrong choice for a high-rate bucket that already has a queue.
Poll Mode Properties
These are read only when mode: poll is set.
| Field | Required | Default | Description |
|---|---|---|---|
interval | N | 60 | Seconds between list cycles. Ignored when cron is set |
cron | N | - | Cron expression for the list cycle, taking precedence over interval |
workers | N | max(NumCPU/4, 1) | Concurrent download and decode workers, 1–64 |
max_retries | N | 0 | Consecutive cycles an object may fail transiently before the poller gives up on it and moves the cursor past it. 0 means never give up |
start_date | N | - | Lower bound of the listing window. Accepts Unix seconds, RFC 3339, a timestamp without an offset (read in timezone), or a relative expression. Unset or negative means the whole bucket |
end_date | N | - | Upper bound, exclusive. Unset means unbounded; a negative value is an error, and a value at or before start_date is rejected |
timezone | N | UTC | Zone used to interpret an offset-less start_date or end_date |
The Cursor
Poll mode keeps a since-cursor so a restart or a cluster failover does not re-ingest what it already has. The cursor is scoped to the listing target — the bucket and prefix together — so changing either starts a fresh one.
A stored cursor takes precedence over start_date, but only while the window is unchanged: editing start_date, end_date or prefix discards the cursor and re-evaluates the new window. A bounded window that has been fully drained stops listing and idles until the window changes.
Date Tokens in prefix
In poll mode prefix may carry date tokens, so a listing walks only the partitions that fall inside its window: %Y, %y, %m, %d, %j, %H, and %% for a literal percent. The finest token present sets the partition step.
A token-bearing prefix needs a bounded start_date — there is no way to walk partitions from the beginning of a bucket — and a window spanning too many partitions is rejected rather than expanded.
Date tokens are rejected in queue mode. A queue subscription filters on a fixed key prefix, which cannot be re-expanded per cycle, so a prefix containing %Y and the like fails at apply time. Write %% where a literal percent is wanted.
IAM Permissions
When using service principal authentication, the following Azure RBAC roles are required:
| Azure Role | Scope | Purpose |
|---|---|---|
Storage Blob Data Reader | Storage Account or Container | Read blobs and blob properties |
Storage Queue Data Message Processor | Storage Account or Queue | Dequeue and delete queue messages |
The device validates connectivity at startup by reading blob service properties and queue metadata.
Queue-Based Notification
The device uses an Azure Storage Queue as a notification mechanism. The queue name is specified by container_name. When a blob lands in the storage account, Azure emits a queue message pointing to the blob. The device dequeues these messages and downloads the referenced blobs for processing.
Authentication
When connection_string is provided, the device builds both the blob and queue clients directly from the connection string. When account is provided instead, the device constructs clients using the service principal credentials and builds URLs of the form https://<account>.blob.core.windows.net/ and https://<account>.queue.core.windows.net/<container_name>/.
Replay
The azblobreplay device type is this collector permanently in poll mode. It lists the container for a bounded time window and ingests the blobs it finds, without an Event Grid storage queue.
Every property behaves as documented under Poll Mode above, including start_date, end_date, timezone, the date tokens in prefix, and the cursor. The only configuration difference is type: mode may be omitted, since a replay can only poll, and setting it to queue is rejected.
What the replay type adds is a reported run — live progress, a run history, and the pause, resume, cancel and run-now controls. See Replay Devices.
Examples
Connection String Authentication
Connecting with an Azure Storage connection string... | |
Service Principal Authentication
Connecting with service principal credentials... | |
Large Blob Processing
Increasing the blob size limit for large files... | |