Avro

Apache Avro is a data serialization system that provides rich data structures and a compact, fast, binary data format. Originally developed within the Apache Hadoop ecosystem, Avro is designed for schema evolution and language-neutral data exchange.

Binary Layout

Section	Internal Name	Description	Possible Values / Format
File Header	magic	4-byte magic number identifying Avro files	ASCII: Obj followed by 1 byte (hex: 4F 62 6A 01)
	meta	Metadata map storing key-value pairs (e.g., schema, codec)	Map of string keys to byte values (e.g., "avro.schema" → JSON schema string)
	sync	16-byte random sync marker used between blocks	16 random bytes (unique per file)
Data Block	blockCount	Number of records in the block	Long (variable-length zigzag encoding)
	blockSize	Size in bytes of the serialized records (after compression, if any)	Long
	blockData	Serialized records (optionally compressed)	Binary-encoded data per schema
	sync	Sync marker repeated after each block	Same 16-byte value as in header

Schema Types (Stored in Metadata)

Type	Internal Name	Description	Example / Format
Primitive	null, boolean, int, long, float, double, bytes, string	Basic types	`"type": "string"
Record	record	Named collection of fields	{ "type": "record", "name": "Person", "fields": [...] }
Enum	enum	Named set of symbols	{ "type": "enum", "name": "Suit", "symbols": ["SPADES", "HEARTS"] }
Array	array	Ordered list of items	{ "type": "array", "items": "string" }
Map	map	Key-value pairs with string keys	{ "type": "map", "values": "int" }
Union	JSON array	Multiple possible types	[ "null", "string" ]
Fixed	fixed	Fixed-size byte array	{ "type": "fixed", "name": "md5", "size": 16 }

JSON Schema Definition

Avro schemas are themselves JSON. A schema authored in this format drives both .avro serialization and VirtualMetric's per-field type coercion before writing. Any schema conforming to the Apache Avro 1.11 specification serializes correctly; the tables below describe which parts VirtualMetric actively understands for validation.

A schema is a single JSON object representing an Avro record:

{
  "type": "record",
  "name": "MyEvent",
  "namespace": "org.example.events",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "created", "type": ["null", { "type": "long", "logicalType": "timestamp-millis" }], "default": null }
  ]
}

Property	Required	Description
type	Y	Must be record at the top level
name	Y	Record name (used by Avro for type identity)
namespace	N	Dotted namespace, e.g. org.example.events
fields	Y	Array of field definitions

Field Definition

Each entry in fields is an object:

Property	Required	Description
name	Y	Field name; used as the key in the message produced by VirtualMetric
type	Y	Field type — a string, a JSON object, or an array (union)
default	N	Default value when the field is absent; honored by the encoder

Other Avro field properties (doc, order, aliases) are accepted by the encoder and ignored by VirtualMetric.

Primitive Types

A primitive type is declared as a string, e.g. { "name": "user_id", "type": "string" }. Each maps to a VirtualMetric internal type:

Avro type	Internal type	Description
string	STRING	UTF-8 string
boolean	BOOLEAN	Boolean
int	INT32	32-bit signed integer
long	INT64	64-bit signed integer
float	FLOAT	32-bit floating point
double	DOUBLE	64-bit floating point
bytes	BYTE_ARRAY	Variable-length byte string. Dropped by the validator — see the warning below
null	—	Used inside unions only

warning

The schema validator — the stage that runs before encoding, both in the Enforce Schema processor and in the file sender when a target declares a schema — recognizes a narrower set of types than the schema parser accepts. It carries through string, boolean, int, long, float, double and the timestamp logical types, plus arrays of those scalars.

bytes is not among them: it maps to BYTE_ARRAY, which reaches the validator's default branch and removes the field from the record. The same happens to the decimal logical type, and to every array, map, record, enum and fixed field — an Avro type the mapper does not recognize is passed through as its own name, which matches no validator arm.

The schema is accepted, no error is raised, and the field is simply absent from the encoded output. array is included even for an array of strings: the validator's list arms are Parquet-side internal types and an Avro schema never produces them.

Nullable Fields

Declare an optional field as a union with null:

{ "name": "email", "type": ["null", "string"], "default": null }

VirtualMetric reads the union, skips the null branch, and treats the field as the non-null type. Member order does not matter for the validator, though encoders typically expect null first when default is null.

Logical Types

A logical type annotates a primitive with extra semantics, declared as a nested object:

{ "name": "created_at", "type": { "type": "long", "logicalType": "timestamp-millis" } }

VirtualMetric recognizes these logicalType values:

logicalType	Backing type	Internal type	Description
timestamp-millis	long	DATETIME	Milliseconds since the Unix epoch
timestamp-micros	long	DATETIME	Microseconds since the Unix epoch
date	int	DATETIME	Days since the Unix epoch
decimal	bytes / fixed	DECIMAL	Fixed-point decimal. Dropped by the validator — see the warning above

Other Avro logical types (uuid, time-millis, time-micros, local-timestamp-millis, duration) are accepted by the encoder but not coerced — they pass through as the underlying primitive. A DATETIME field accepts RFC 3339 timestamps, 2006-01-02 15:04:05 (UTC), or a numeric epoch (seconds, milliseconds, microseconds, or nanoseconds, auto-detected).

Validation Scope

The schema is consumed by two layers. The validator extracts the top-level field-name-to-type mapping and coerces each top-level field into its declared type before encoding (so a string "42" on an int field becomes a real integer). The encoder consumes the full Avro 1.11 specification. Automatic coercion applies only to top-level scalars, top-level nullable unions of primitives, and the three timestamp/date logical types. record, array, map, enum and fixed do not pass through — the validator deletes them, whether nested or at the top level. Only the types the validator recognizes reach the encoder at all.

Reference Example

{
  "type": "record",
  "name": "AuditLog",
  "namespace": "org.example.audit",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "received_at", "type": { "type": "long", "logicalType": "timestamp-millis" } },
    { "name": "actor_id", "type": ["null", "string"], "default": null },
    { "name": "severity", "type": "int" },
    { "name": "success", "type": "boolean" },
    { "name": "amount", "type": ["null", "double"], "default": null },
    { "name": "tags", "type": ["null", "string"], "default": null }
  ]
}

Schemas in this format can be registered as reusable Library entries and referenced by name from the Enforce Schema and Check Schema processors, or supplied inline.

Metadata Keys (in meta)

Key	Description	Example Value
avro.schema	JSON-encoded schema	JSON string defining the schema
avro.codec	Compression codec used	"zstandard" (default), "deflate", "snappy", "null"

Compression Codecs

Set with the compression field on a target. The following values are recognized:

Value	Codec	Best For
zstandard	Zstandard. Written to the file as zstandard	Best balance of speed and compression ratio
zstd	Zstandard. An alias for zstandard	Best balance of speed and compression ratio
deflate	DEFLATE, standard ZIP compression (RFC 1951)	General-purpose compression
snappy	Snappy, with a CRC32 checksum	Real-time streaming applications

Omitting the compression field applies zstandard.

warning

An unrecognized value is not an error. The file is written uncompressed and no warning is emitted, so a misspelled codec is silent. The four values above are the only ones that compress.