Snowflake (Azure Blob Staging)

Data Warehouse Target

Synopsis

The Snowflake Azure Blob target stages telemetry files to Azure Blob Storage, then executes COPY INTO commands on Snowflake to load data into tables.

Schema

- name: <string>
  description: <string>
  type: azsnowflake
  properties:
    account: <string>
    username: <string>
    password: <string>
    database: <string>
    schema: <string>
    warehouse: <string>
    role: <string>
    storage_account: <string>
    staging_container: <string>
    staging_prefix: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    table: <string>
    data_schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    tables:
      - table: <string>
        schema: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
    batch_size: <integer>
    max_size: <integer>
    timeout: <integer>
    drop_unknown_stream_events: <boolean>
    field_format: <string>
    max_rows_per_rowgroup: <numeric>
    buffer_size: <numeric>
    data_page_version: <string>
    metadata: <key-value>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Base Target Fields

Field	Required	Default	Description
name	Y	-	Unique identifier for this target
description	N	-	Human-readable description
type	Y	-	Must be azsnowflake
pipelines	N	-	Pipeline names to apply before sending
status	N	true	Enable/disable the target

Snowflake Connection

Field	Required	Default	Description
account	Y	-	Snowflake account identifier (e.g., abc123.west-europe.azure)
username	Y	-	Snowflake username
password	Y	-	Snowflake password
database	Y	-	Snowflake database name
schema	N	PUBLIC	Snowflake schema name. Must be a valid SQL identifier — letters, digits, and underscores only, not starting with a digit.
warehouse	N	-	Snowflake virtual warehouse name
role	N	-	Snowflake role name

Azure Blob Staging Configuration

Field	Required	Default	Description
storage_account	Y	-	Azure storage account name
staging_container	Y	-	Azure Blob container name for staging files
staging_prefix	N	snowflake-staging/	Blob prefix path
tenant_id	Y	-	Azure AD tenant ID
client_id	Y	-	Service principal client ID
client_secret	Y	-	Service principal client secret

Table Configuration

Field	Required	Default	Description
table	Y*	-	Catch-all table name for all events
data_schema	Y*	-	Avro/Parquet schema for the catch-all table: a Library name, a built-in name, an inline JSON definition, or a field:type,... list. When it is empty, the top-level schema is read in its place, which still has to be a valid SQL identifier because it is also the Snowflake schema.
name	N	vmetric.{{.Timestamp}}.{{.Extension}}	File naming template
format	N	json	File format. Use json, avro, or parquet — see the warning below.
compression	N	zstd	Compression algorithm
extension	N	-	File extension override. Defaults to the resolved format.
tables	N	-	Multiple table configurations (see below)
tables.table	Y	-	Target table name
tables.schema	Y*	-	Avro/Parquet schema for this table: a Library name, a built-in name, an inline JSON definition, or a field:type,... list. This table does not read data_schema.
tables.name	Y	-	File naming template for this table
tables.format	N	-	File format for this table. Falls back to the catch-all format.
tables.compression	N	-	Compression algorithm for this table. Falls back to the catch-all compression.
tables.extension	N	-	File extension override for this table. Falls back to the catch-all extension.

warning

Only json, avro, and parquet are written correctly. csv, orc, and xml pass configuration validation, but the staged file is written as JSON while the COPY INTO command still declares the requested format, so the load fails.

Parquet Options

These apply only when format is parquet.

Field	Required	Default	Description
max_rows_per_rowgroup	N	10000	Maximum rows per Parquet row group
buffer_size	N	262144	Page buffer size in bytes (256KB) of the older Parquet writer only
data_page_version	N	-	V1, or no value, uses the column-by-column writer, which writes version 1 data pages, unless the schema or the compression keeps the older writer. V2 keeps the older Parquet writer, which writes version 2 data pages. Before, the default was V2
metadata	N	-	Key/value pairs written into the Parquet file metadata

How the staged files are written, which values they keep, and which outputs still use the older writer: see How Targets Write Parquet.

* = At least one of table (catch-all) or tables (multiple) must be configured. For Avro/Parquet formats, the catch-all table needs data_schema (or, failing that, schema) and every tables entry needs its own schema.

Batch Configuration

Field	Required	Default	Description
batch_size	N	100000	Maximum events per file before flush
max_size	N	33554432	Maximum file size in bytes before flush (32MB)
timeout	N	300	COPY INTO command timeout in seconds
drop_unknown_stream_events	N	true	Drop events that do not match any configured table

Normalization

Field	Required	Default	Description
field_format	N	-	Data normalization format. Any supported format is accepted; OCSF is a common choice for security data in a lakehouse. See applicable Normalization section

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging for this target
debug.dont_send_logs	N	false	Log events without sending to Snowflake

Details

Architecture Overview

The Snowflake Azure Blob target implements a two-stage loading pattern:

1. Stage Files to Azure Blob: Events are written to files in Azure Blob Storage using the configured format
2. Execute COPY INTO: SQL commands load data from Blob Storage into Snowflake tables using azure:// paths

Snowflake Connection

Account identifier:

• Format for Azure: <account_locator>.<region>.azure (e.g., abc123.west-europe.azure)
• Account locator is visible in your Snowflake URL
• Region is the Azure region where your Snowflake account is deployed

Authentication:

• Uses username/password authentication
• Credentials are used to connect to Snowflake SQL API v2
• Supports optional warehouse and role specification

Database and Schema:

• Database name is required and must be a valid SQL identifier
• Schema defaults to PUBLIC if not specified
• Both database and schema names are validated for SQL compliance

Snowflake Permissions

The Snowflake user requires permissions to:

• Execute SQL statements using the specified warehouse
• Write data to the target database and schema

Director sends the COPY INTO statement without a storage integration and without credentials, so Snowflake reads the staged file from the container with access of its own. Grant that access on the Snowflake side and on the storage account. There is no property on this target that names a storage integration.

Azure Blob Staging Operations

File Upload:

• Files are staged to https://{storage_account}.blob.core.windows.net/{container}/{prefix}/{table}/{filename} structure
• Uses Azure SDK for secure uploads with service principal authentication
• Supports Azure AD authentication through client credentials

Azure Path Construction:

• The target automatically constructs azure:// paths for COPY INTO commands
• Format: azure://{storage_account}.blob.core.windows.net/{container}/{prefix}/{table}/{filename}
• Azure protocol is used for direct Snowflake access to Azure Blob Storage

Cleanup:

• Staged files are automatically deleted after successful COPY INTO execution
• Failed uploads remain in Blob Storage for troubleshooting

Service Principal Authentication

Azure AD Integration:

• Uses service principal (client credentials) for Azure Blob Storage authentication
• Requires tenant_id, client_id, and client_secret configuration
• Service principal must have Storage Blob Data Contributor role on the container

Required Permissions:

• Storage Blob Data Contributor: Write and delete blobs in staging container
• Storage Blob Data Reader: for the identity Snowflake itself uses to read the staging container, not for the service principal above

Service Principal Permissions

The service principal writes the staged file into the container. It grants Snowflake nothing. Snowflake reads that same file with read access granted separately to its own identity, so both sides have to be set up before a load succeeds.

File Format Support

Valid Formats:

• CSV: Comma-separated values with optional headers
• JSON: Newline-delimited JSON objects
• AVRO: Schema-based binary format (requires schema)
• ORC: Optimized row columnar format
• PARQUET: Columnar storage format (requires schema)
• XML: XML document format

Schema Requirements:

• Avro and Parquet formats require a schema definition: data_schema for the catch-all table, schema on each tables entry
• Schema must match the expected table structure in Snowflake
• Other formats use schema inference from data

Multi-Table Routing

Catch-All Table:

• Use table field to send all events to a single table
• Simplest configuration for single-destination scenarios

Multiple Tables:

• Use tables array to route different event types to different tables
• Each table entry specifies table, schema, name, format fields
• Events routed based on SystemS3 field in pipeline

Example Configuration:

tables:
  - table: SECURITY_EVENTS
    schema: security_schema
    name: security.{{.Timestamp}}.parquet
    format: parquet
  - table: ACCESS_LOGS
    schema: access_schema
    name: access.{{.Timestamp}}.parquet
    format: parquet

Performance Considerations

Batch Processing:

• Events are buffered until batch_size or max_size limits are reached
• Larger batches reduce Blob API calls and COPY INTO operations
• Balance batch size against latency requirements

Upload Optimization:

• Azure SDK automatically handles large blob uploads
• Uses block blobs for efficient data transfer
• Connection pooling optimizes network performance

COPY INTO Performance:

• COPY INTO commands are executed with configurable timeout
• Failed COPY operations return errors for retry logic
• Warehouse must be running (resumed) for COPY INTO to succeed

Warehouse State

Ensure the virtual warehouse is running before sending data. COPY INTO commands will fail if the warehouse is suspended. Configure warehouse auto-resume or manual resume procedures.

Error Handling

Upload Failures:

• Failed Blob uploads are retried based on sender configuration
• Permanent failures prevent COPY INTO execution
• Check service principal permissions and network connectivity

COPY INTO Failures:

• Schema mismatches between files and tables cause failures
• Invalid SQL identifiers (database, schema, table names) are rejected at validation
• Check Snowflake query history for detailed error messages

Examples

Basic Configuration

Sending telemetry to Snowflake using Azure Blob staging with Parquet format...

targets: - name: snowflake-warehouse type: azsnowflake properties: account: abc123.west-europe.azure username: datastream_user password: "${SNOWFLAKE_PASSWORD}" database: PRODUCTION_DATA warehouse: COMPUTE_WH storage_account: datastreamstaging staging_container: snowflake-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: EVENTS data_schema: event_schema name: events.{{.Timestamp}}.parquet format: parquet

With Custom Staging Prefix

Using custom blob prefix for organized staging file structure...

targets: - name: snowflake-organized type: azsnowflake properties: account: xyz789.west-europe.azure username: security_user password: "${SNOWFLAKE_PASSWORD}" database: SECURITY_ANALYTICS warehouse: SECURITY_WH role: SECURITY_ADMIN storage_account: securitystorage staging_container: staging staging_prefix: datastream/snowflake/ tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: SECURITY_EVENTS data_schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet

Multi-Table Configuration

Routing different event types to separate Snowflake tables...

targets: - name: snowflake-multi-table type: azsnowflake properties: account: abc123.west-europe.azure username: analytics_user password: "${SNOWFLAKE_PASSWORD}" database: ANALYTICS warehouse: ANALYTICS_WH storage_account: analyticsstorage staging_container: staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" tables: - table: AUTHENTICATION_EVENTS schema: auth_schema name: auth.{{.Timestamp}}.parquet format: parquet - table: NETWORK_EVENTS schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet - table: APPLICATION_LOGS schema: app_schema name: app.{{.Timestamp}}.parquet format: parquet

High-Volume Configuration

Optimizing for high-volume ingestion with batch limits and compression...

targets: - name: snowflake-high-volume type: azsnowflake properties: account: abc123.west-europe.azure username: streaming_user password: "${SNOWFLAKE_PASSWORD}" database: HIGH_VOLUME_DATA warehouse: LARGE_WH storage_account: streamingstorage staging_container: high-volume-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" batch_size: 100000 max_size: 134217728 timeout: 600 table: STREAMING_EVENTS data_schema: streaming_schema name: stream.{{.Timestamp}}.parquet format: parquet compression: snappy

JSON Format

Using JSON format for flexible schema evolution and debugging...

targets: - name: snowflake-json type: azsnowflake properties: account: abc123.west-europe.azure username: dev_user password: "${SNOWFLAKE_PASSWORD}" database: DEVELOPMENT warehouse: DEV_WH storage_account: devstorage staging_container: dev-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" table: TEST_EVENTS name: test.{{.Timestamp}}.json format: json

With Normalization

Applying ASIM normalization before loading to Snowflake...

targets: - name: snowflake-normalized type: azsnowflake properties: account: abc123.west-europe.azure username: security_user password: "${SNOWFLAKE_PASSWORD}" database: SECURITY_DATA warehouse: SECURITY_WH storage_account: securitystorage staging_container: security-staging tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" field_format: ASIM table: ASIM_EVENTS data_schema: asim_schema name: asim.{{.Timestamp}}.parquet format: parquet

Production Configuration

Production-ready configuration with performance tuning and multi-table routing...

targets: - name: snowflake-production type: azsnowflake properties: account: production.west-europe.azure username: production_user password: "${SNOWFLAKE_PASSWORD}" database: PRODUCTION_ANALYTICS warehouse: PRODUCTION_WH role: DATA_ENGINEER storage_account: productionstorage staging_container: production-staging staging_prefix: datastream/snowflake/ tenant_id: "${AZURE_TENANT_ID}" client_id: "${AZURE_CLIENT_ID}" client_secret: "${AZURE_CLIENT_SECRET}" batch_size: 50000 max_size: 67108864 timeout: 300 field_format: ECS tables: - table: SECURITY_EVENTS schema: security_schema name: security.{{.Timestamp}}.parquet format: parquet compression: snappy - table: AUDIT_LOGS schema: audit_schema name: audit.{{.Timestamp}}.parquet format: parquet compression: snappy - table: NETWORK_FLOWS schema: network_schema name: network.{{.Timestamp}}.parquet format: parquet compression: snappy

Troubleshooting

This section covers the errors you are most likely to see with the azsnowflake target, what causes each one, and how to fix it.

Where to look:

• Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
• The target's connection status in the web interface. It shows the same reason as the log line, prefixed with connection failed for <target name>:.

See Target Delivery Errors for how Director logs and retries target failures.

Two facts shape most of the entries below. Director tests the Snowflake connection at startup but does not contact Azure until the first file is staged, so a wrong service principal or a missing storage role appears at the first flush while the connection status stays positive. And the COPY INTO statement names the staged file by its azure:// URL without a storage integration and without credentials, so Snowflake has to reach the container on its own.

Which permission is missing?

Two identities are involved, and the error text tells you which side to fix. The application in client_id writes the staged file into the container. Snowflake then reads that same container by itself. Granting one never covers the other.

Error text	Missing right	Where it is granted
ClientSecretCredential authentication failed ... AADSTS...	None. The service principal credential itself is wrong	The app registration in the tenant named by tenant_id
failed to upload to azure blob: ... ERROR CODE: AuthorizationPermissionMismatch	Storage Blob Data Contributor for the application in client_id	The storage account, or only the staging_container. Owner and Contributor do not carry blob write permission
failed to upload to azure blob: ... ERROR CODE: AuthorizationFailure or AuthorizationSourceIPMismatch	None. The storage account firewall rejects the Director host	The storage account's Networking settings
failed to test snowflake connection: snowflake API error (status 401)	None. The Snowflake user name or password is wrong	The Snowflake user in username
snowflake API error (status 422) with code 002003	USAGE on the warehouse, the database and the schema, and INSERT on the table	Snowflake, to the role in role if set, otherwise to the user's default role
failed to execute COPY command: snowflake error: 091003	Read access to the staging container for Snowflake's own identity	Snowflake and the storage account together. Nothing you set on this target grants it

note

Azure RBAC changes typically take several minutes to propagate. Director retries automatically, so no restart is needed once a role is assigned.

"Failed to reinitialize target ... (attempt N)" with a configuration reason

[Error] [director] [target-<target id>] [snowflake-warehouse] Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: invalid schema name: event_schema.avsc

Cause: the target could not start. Retries back off from 5 seconds to once a minute and continue until it does. Only the first failure is logged as an Error, and the count of failed attempts is in the Warning that follows at most every 15 minutes while the cause lasts. A high attempt count only means the cause has been present for a while.

Fix: correct the reason shown here.

Reason	Fix
account is required for azsnowflake target, and the same for username, password, database, storage_account, staging_container, tenant_id, client_id and client_secret	All nine are mandatory. A ${ENV} reference that is not exported to the Director service reads as empty. This target has no Managed Identity mode
invalid database name: ...	database must be an unquoted SQL identifier: letters, digits and underscores, not starting with a digit. Hyphens, dots and quotes are rejected. Table names follow the same rule
invalid schema name: ...	schema is the Snowflake schema and is validated as a SQL identifier, so a file name or a path is rejected. Put the catch-all table's Avro or Parquet schema in data_schema instead
invalid schema format: invalid field format: ...	The schema name resolved to nothing: it has to be a schema in your schema library, a built-in model, inline JSON, or a field:type list. For the catch-all table the reason starts with invalid default table data_schema:, or, when data_schema is empty and schema was read in its place, with invalid default table schema: data_schema is not set. Set data_schema in that case
at least one table must be configured (either 'table' or 'tables'), or invalid table configuration: name is required	Set a catch-all table or at least one tables entry, and give every tables entry its own name. Only the catch-all gets a default file name
invalid table configuration: tables[].schema is required for parquet format, or data_schema is required for parquet format, and the same for avro format	Set schema on that tables entry, or data_schema for the catch-all table
failed to generate unique file path after 10 attempts for thread 0	The name template is static, so parallel workers produce the same path. Keep {{.Timestamp}} in it

Nothing is sent while the target is failing. Incoming data waits in the Director queue and is delivered once the target starts, so no records are lost. Restarting Director is not required.

"failed to test snowflake connection: snowflake API error (status 401)"

Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: failed to test snowflake connection: snowflake API error (status 401)

Cause: Director runs a one-row test query against the Snowflake SQL API at startup, and Snowflake refused the sign-in. The body after the status carries a code. It is typically 390100 when username or password is wrong, and 390114 when the session credential has expired.

Fix: correct username and password. The target signs in with the user name and password only, so if that user is enrolled in multi-factor authentication, or your account requires key pair or OAuth sign-in for it, the request is typically refused whatever you put in password. Point the target at a service user that can sign in with a password alone. Nothing is sent while this persists and the target is retried indefinitely, so no records are lost.

"ClientSecretCredential authentication failed" with an AADSTS code

Sender worker 1 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to upload to azure blob: ClientSecretCredential authentication failed. POST https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/v2.0/token
RESPONSE 401: 401 Unauthorized
{"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. ..."}

Cause: the Azure service principal credentials are wrong. Building the storage client contacts nothing, so this is reported at the first flush rather than at startup, and the connection status shows the target as connected until then. The AADSTS code tells you which field:

Code	Meaning	Check
AADSTS7000215	Invalid client secret	The secret ID was pasted instead of the secret value, or the secret was rotated
AADSTS7000222	Client secret expired	Create a new secret on the app registration and update client_secret
AADSTS700016	Application not found in the directory	client_id is wrong, or tenant_id is a different tenant than the one the app registration lives in
AADSTS90002	Tenant not found	tenant_id is not a valid tenant ID

Fix: correct the field and save the target. The usual cause of AADSTS7000215 is a secret copied from the Secret ID column instead of the Value column. The batch is kept in the queue and redelivered once the token request succeeds, so nothing is lost.

"failed to upload to azure blob" with RESPONSE 403 or 404

Sender worker 2 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to upload to azure blob: PUT https://mystorageaccount.blob.core.windows.net/mycontainer/snowflake-staging/my_table/vmetric.1757590123456789000.json
RESPONSE 403: 403 ...
ERROR CODE: AuthorizationPermissionMismatch

Cause: the staged file could not be written. AuthorizationPermissionMismatch means the token was accepted but the application has no permission to write blobs there. Owner and Contributor on the storage account manage the account and do not include writing data. The other codes:

Error code	Cause	Fix
ContainerNotFound with 404	The staging_container does not exist. The target never creates it	Create the container under Data storage > Containers. InvalidResourceName with 400 instead means the name has capitals or underscores: Azure container names take lower-case letters, digits and hyphens only
AuthorizationFailure or AuthorizationSourceIPMismatch with 403	A storage firewall, virtual network rule or private endpoint rejects the Director host	Allow the egress address under Networking, or enable the trusted services exception
AccountIsDisabled with 403, or ServerBusy with 503	The account is disabled, or it is throttling ingress	Re-enable the account, or lower max_size. Throttling is already retried three times inside timeout before it reaches the log

Fix: for AuthorizationPermissionMismatch, open the storage account in the Azure Portal, or only the staging container, then Access control (IAM) > Add role assignment, and assign Storage Blob Data Contributor to the application whose ID is in client_id. That role also covers the delete of the staged file after the load. All of these are retried until fixed, and nothing is staged in the meantime, so no records are lost.

"091003 - Failure using stage area"

Sender worker 2 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to execute COPY command: snowflake error: 091003 - Failure using stage area. Cause: [Access Denied]

Cause: the file reached the container, but Snowflake could not read it. The statement points at azure://mystorageaccount.blob.core.windows.net/mycontainer/... and carries no storage integration and no credentials, so Snowflake must already be able to read that location. Adding roles to the application in client_id changes nothing here. That identity writes the file. It does not speak for Snowflake.

Fix: give Snowflake read access to the staging container in its own right, so that the access applies without being named in the statement. Set it up on the Snowflake side for that container, and grant the identity Snowflake uses Storage Blob Data Reader on it. If the storage account has a firewall, allow the Snowflake account's egress addresses as well as the Director host. Then confirm the container and the staging_prefix in the URL are the ones that access covers. Retried until fixed, so nothing is lost: the same batch is staged and loaded again on every attempt until Snowflake can read it.

"does not exist or not authorized", or a suspended warehouse

Sender worker 2 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to execute COPY command: snowflake API error (status 422)

Cause: Snowflake accepted the sign-in but refused the statement. The code in the body says why.

Code	Typical meaning	Fix
002003	The database, schema or table in the statement does not exist, or the role cannot see it	Create the table. The target never creates tables. Check schema as well: when data_schema is empty, schema is also read as the catch-all table's Avro or Parquet schema, so a value chosen for the file format sends the load to my_database.<that value>.my_table
000606	The warehouse is suspended and auto-resume is off	Resume the warehouse, or set AUTO_RESUME = TRUE on it

Fix: grant USAGE on the warehouse, the database and the schema, and INSERT on each table, to the role in role or to the user's default role. Retried until fixed, and data waits in the queue, so nothing is lost.

"record rejected by target" after COPY INTO

Sender worker 2 execute() failed for snowflake-warehouse: failed to execute COPY command: record rejected by target: snowflake API error (status 422): {"code":"100038","message":"Numeric value 'lots' is not recognized"}
Sender worker 2 deterministic failure for snowflake-warehouse after 4 attempts — dropping (giving up): failed to execute COPY command: record rejected by target: snowflake API error (status 422): {"code":"100038","message":"Numeric value 'lots' is not recognized"}

Cause: the file staged correctly, but its contents do not fit the table. Codes in the 1000xx and 1001xx range are data errors: a value that will not convert to the column type (100038), JSON the loader cannot parse (100069), a null in a NOT NULL column, or a column count that does not match. A 400 with code 000904 means the statement itself was rejected as invalid.

Fix: align the table definition with the staged file. A single VARIANT column takes JSON as it comes. For Parquet, the schema fields and the table columns have to match. If the message complains about JSON or a numeric value while your format is parquet, read the next entry before you change any column. These errors lose data, and so does a record routed to a table the target does not configure (see the is not configured on the target entry below). The batch is retried four times and then dropped, and the dropping line quotes the same reason.

The staged file is JSON although "format" says otherwise

Sender worker 2 execute() failed for snowflake-warehouse: failed to execute COPY command: record rejected by target: snowflake API error (status 422): {"code":"100069","message":"Error parsing JSON: ..."}

Cause: format is validated without regard to case, but the file writer recognizes lower-case names only. A value such as PARQUET, Parquet or CSV passes validation, the file is then written as JSON lines, and the COPY INTO statement still declares the format you asked for. Snowflake reads a JSON file as Parquet, CSV, ORC or XML and rejects it. The requirement to set a file schema for Avro and Parquet is skipped too, so format: PARQUET with no data_schema or schema starts without complaint.

Fix: write format in lower case, and use only json, avro or parquet. csv, orc and xml pass validation in any spelling but are never actually written.

warning

The message names the content, never the format, which is what makes this one hard to recognize. The batch is dropped after four delivery attempts like any other rejected batch, so everything staged while the spelling was wrong is lost.

"warehouse statement outcome indeterminate"

Sender worker 3 execute() failed for snowflake-warehouse: target broken: failed to finalize target cache: failed to execute COPY command: warehouse statement outcome indeterminate: statement 01b2-async-0001 still running: context deadline exceeded

Cause: Director started the load but never learned how it ended. The whole statement, including the polling that follows an asynchronous acceptance, has to finish within timeout seconds, 300 by default. A large file on a small warehouse, or a connection that drops mid-statement, ends here. The same wrapper appears with failed to execute request, failed to poll statement and failed to parse async response.

Fix: raise timeout, lower max_size so each staged file is smaller, or run the load on a larger warehouse. No records are dropped: the staged file is kept on purpose and the batch is redelivered. If the first statement did finish after Director stopped waiting, the redelivery stages the same records again under a new file name, so the same rows can be loaded twice. Reconcile the table after a run of these.

"destination ... is not configured on the target, and the target has no catch-all"

Sender worker 1 execute() failed for snowflake-warehouse: record rejected by target: destination "firewall_events" is not configured on the target, and the target has no catch-all

Cause: a pipeline routed a record to a table that is not in tables, there is no catch-all table, and drop_unknown_stream_events is set to false. A record routed to no table at all reports the record names no destination, and the target has no catch-all instead.

Fix: add the missing tables entry or a catch-all table, or make the pipeline set a table that is configured.

This is data loss. The record is tried four times, about 2 seconds apart. On the fourth attempt that record alone is dropped and counted on the target's dropped counter, logged as Target "<name>" rejected record <n> of <payload> on its last delivery attempt — dropping that record and delivering the rest of the payload: <reason>. The records before and after it in the same payload are delivered. Any other record the target refuses on that attempt is dropped the same way without a line of its own, and the payload's Completed processing line ends , rejected by the target and dropped: <count>. The target stays up, and other payloads keep flowing. At the default drop_unknown_stream_events: true the same records are dropped at once, without this line.

An Avro or Parquet encoder message after target broken: is a different failure: one record does not fit the schema. Make the schema fields nullable, or normalize the record with field_format. That record is not dropped. It is retried every few seconds and the target reinitializes on each attempt, so everything queued behind it on that worker stalls until you fix it.

"no such host", "context deadline exceeded", or certificate errors

Failed to reinitialize target "snowflake-warehouse" (attempt 1). Reason: failed to test snowflake connection: warehouse statement outcome indeterminate: failed to execute request: Post "https://my-account.snowflakecomputing.com/api/v2/statements": dial tcp: lookup my-account.snowflakecomputing.com: no such host

Cause: Director could not complete an HTTPS request. The host in the message tells you which endpoint failed.

• no such host on a snowflakecomputing.com name: account is wrong. It is the account locator with its region and cloud, such as my-account.west-europe.azure, not a URL and not the organization name. On a blob.core.windows.net name, storage_account is misspelled. Set it to the account name alone.
• tls: failed to verify certificate: x509: certificate signed by unknown authority: a TLS-intercepting proxy sits in the path. The target has no certificate options of its own and trusts the operating system store, so install the proxy's CA certificate at the OS level. proxyconnect tcp: ... connection refused means HTTPS_PROXY is set for the Director service but the proxy refused the connection.
• context deadline exceeded on the startup test: that test is bounded to 10 seconds, tighter than timeout. A warehouse that resumes slowly can miss it repeatedly.

Fix: allow outbound HTTPS (443) from the Director host to <account>.snowflakecomputing.com, <storage account>.blob.core.windows.net and login.microsoftonline.com. If you use a proxy, set HTTPS_PROXY for the Director service. Retried until fixed, and queued data is delivered once the endpoint answers, so nothing is lost.

note

Only the public Azure storage endpoint is used, so a staging account in the China or US Government cloud is not reachable from this target. A Snowflake PrivateLink account works only when the privatelink segment is already part of account.

The target is healthy but no rows arrive in Snowflake

Check these in order.

1. debug.dont_send_logs is enabled. In that mode records are accepted and then discarded before they are buffered. No file is staged, no load runs, and no counter moves. Only a single debug line at startup mentions it, and only when debug.status is also enabled. Remove the flag.

2. Records are dropped as unknown tables. With drop_unknown_stream_events at its default of true, a record whose routed table is not in tables is discarded without a log line when there is no catch-all table. Watch the target's dropped counter. Set the flag to false for a while to turn the silent drop into the is not configured on the target error above, which names the table. Those records are still dropped, after four attempts.

3. Every flush is failing. Azure is not contacted at startup, so the connection status can stay positive while every upload fails. Search the Director log for execute() failed or Finalize failed on flush with the target name.

4. The rows went to another schema. schema names the Snowflake schema. If you set it to the name of your Avro or Parquet schema instead of using data_schema, the load goes to my_database.<that value>.my_table, which may be a schema you never query.

5. The batch was dropped. A rejected batch is retried four times and then dropped with deterministic failure ... dropping (giving up). The format spelling above is the most common way to get there.

6. Staged files pile up in the container. The delete that follows a successful load is not reported when it fails, so an application that can write but not delete loads the rows and leaves every file behind. Assign Storage Blob Data Contributor, which covers both, or put a lifecycle rule on staging_prefix.

7. Compression is not what you configured. For json staging only gzip is applied. Any other value, the default zstd included, leaves the staged file uncompressed and logs nothing. No records are lost.