Microsoft Fabric Eventstream

Microsoft Azure Real-time Messaging

Synopsis

Creates a Microsoft Fabric Eventstream target that sends records to the custom endpoint of an eventstream, over the Event Hub, AMQP or Kafka protocol.

The target sends through the Azure Event Hubs target (Event Hub and AMQP) or the Apache Kafka target (Kafka). With Event Hub and AMQP, it sends exactly as the Azure Event Hubs target does.

Schema

- name: <string>
  description: <string>
  type: fabriceventstream
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    protocol: <string>
    client_connection_string: <string>
    event_hub: <string>
    namespace: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    batch_size: <integer>
    max_bytes: <integer>
    timeout: <integer>
    max_retry: <integer>
    retry_interval: <integer>
    partition_key: <string>
    field_format: <string>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be fabriceventstream
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
protocol	N	eventhub	The tab of the custom endpoint to send to: eventhub, amqp or kafka. In the console: Protocol (Event Hub, AMQP, Kafka)
client_connection_string	Y*	-	The connection string of the tab, which Fabric shows as "Connection string-primary key". Required with amqp and kafka, and with eventhub unless it signs in with Microsoft Entra ID. Plain text, an environment variable or a secret. In the console: Connection string; for Event Hub, with Authentication method set to SAS key
event_hub	Y*	-	The event hub. Required with eventhub and Microsoft Entra ID: the "Event hub" of the endpoint's Entra ID authentication page, such as es_1a2b3c4d. With a connection string, it is needed only when the string does not name the event hub; it is then the event hub, the entity name of the AMQP tab, or the Kafka topic. When the string names the event hub, event_hub may only repeat that name (see The Event Hub, Named Twice). In the console: Event hub (Event Hub with Entra ID) and Entity name (AMQP)
namespace	Y*	-	Only with eventhub and Microsoft Entra ID, and required then: the "Event hub namespace" of the Entra ID authentication page, such as eventstream-1a2b3c4d.servicebus.windows.net. In the console: Event hub namespace
tenant_id, client_id, client_secret	N	-	Only with eventhub and Microsoft Entra ID: the service principal to sign in as. Set all three, or none of them. With none, the target signs in through Azure's default credential chain, which on an Azure host is usually the managed identity of the director; give the workspace role to the identity it signs in as. Each takes plain text, an environment variable or a secret. In the console: Tenant ID, Client ID and Client secret, shown while Managed identity is off

* = Conditionally required (see the description)

Delivery Settings

Field	Required	Default	Description
batch_size	N	1000	The most records in one batch. In the console: Batch size
max_bytes	N	0; 1048576 (1 MiB) with kafka	The most bytes in one batch. With eventhub and amqp, 0 means the largest batch that Event Hubs takes. With kafka, 0 removes the target's own batch limit, but the producer still sends no message larger than 1000012 bytes, as for the Apache Kafka target. Not in the console
timeout	N	30	Only with eventhub and amqp: the seconds that sending one batch may take, retries included. In the console: Timeout
max_retry	N	3	Only with eventhub and amqp: how many times the target tries a failed send again. 0 means no retries. Not in the console
retry_interval	N	4	Only with eventhub and amqp: the seconds before the first retry. Each later wait is longer, up to 120 seconds. Not in the console
partition_key	N	-	Only with eventhub and amqp: records with the same key go to the same partition. In the console: Partition key
field_format	N	-	The format each record is converted to before the target sends it, such as ecs or ocsf. All three protocols use it. See applicable Normalization section. In the console: Field format

The target sends through the Azure Event Hubs target (eventhub, amqp) or the Kafka target (kafka). A setting that these tables do not list means what it means for that target.

Details

Before You Start, in Fabric

• Add a custom endpoint source to the eventstream, and publish the eventstream.
• The source's details pane then has one tab for each protocol: Event Hub, AMQP and Kafka. Each tab shows its "Connection string-primary key". The AMQP tab also shows the entity name, and the Event Hub tab's Entra ID authentication page shows the "Event hub namespace" and the "Event hub".

Which Protocol to Choose

• Event Hub (the default) sends over AMQP 1.0 and TLS, to port 5671, as the Azure Event Hubs target does. It is the only protocol that can sign in with Microsoft Entra ID.
• AMQP takes the AMQP tab's amqps:// connection string, and the director makes the same connection as for Event Hub. The target always checks the endpoint's certificate, so a verify=verify_none in the string makes no difference. A connection string in the Event Hubs format works too.
• Kafka sends over TLS to port 9093 of the namespace, with SASL PLAIN: the user name is $ConnectionString, and the password is the connection string. The event hub is the Kafka topic. Fabric gives Kafka a SAS key only.

How the Target Signs In

• With a SAS key, the target needs only the tab's connection string. With Event Hub and Kafka, Fabric's string already names the event hub.
• With Microsoft Entra ID (Event Hub only), the target needs the namespace and the event hub, and signs in as a service principal or, without one, through Azure's default credential chain, usually as the managed identity of the director. The identity needs the Contributor role on the Fabric workspace. A service principal also needs the Fabric tenant setting "Service principals can call Fabric public APIs" to be on.
• AMQP and Kafka sign in with the connection string alone. With them, the director refuses namespace, tenant_id, client_id and client_secret.

The AMQP Connection String

• It has the form amqps://<key name>:<key>@<host>/<event hub>. Write each / in the key as %2F.
• When the URL ends with the event hub, event_hub (the console's Entity name) can stay empty. When both name an event hub, they must name the same one.
• The director refuses amqp://: the endpoint takes TLS connections only.

The Event Hub, Named Twice

A connection string that names its event hub, and an event_hub that names another, is a configuration error for every protocol. Repeating the same name is fine.

Examples

Event Hub with a SAS Key

Event Hub, with the SAS key of the Event Hub tab:

targets:
  - name: fabric_eventstream
    type: fabriceventstream
    properties:
      client_connection_string: "$secret{id=1001}"

Event Hub with Microsoft Entra ID

Event Hub, with Microsoft Entra ID as the managed identity of the director:

targets:
  - name: fabric_eventstream_entra
    type: fabriceventstream
    properties:
      protocol: eventhub
      namespace: "eventstream-1a2b3c4d.servicebus.windows.net"
      event_hub: "es_1a2b3c4d"

AMQP with a Partition Key

AMQP, with the entity name apart from the connection string and a partition key:

targets:
  - name: fabric_eventstream_amqp
    type: fabriceventstream
    properties:
      protocol: amqp
      client_connection_string: "$secret{id=1002}"
      event_hub: "es_1a2b3c4d"
      partition_key: "firewall-logs"

Kafka with ECS Field Names

Kafka, with ECS field names and smaller batches:

targets:
  - name: fabric_eventstream_kafka
    type: fabriceventstream
    properties:
      protocol: kafka
      client_connection_string: "$secret{id=1003}"
      batch_size: 500
      field_format: ecs

Troubleshooting

Configuration the target refuses

Cause: the director refuses a configuration that it cannot use. A message may start with fabriceventstream:.

• protocol must be "eventhub", "amqp" or "kafka", got "…"
• client_connection_string is required when protocol is "amqp", and the same for "kafka"
• namespace cannot be used when protocol is "kafka", which signs in with client_connection_string only, and the same for tenant_id, client_id, client_secret and "amqp"
• event_hub "…" is not the event hub "…" that the connection string names
• event_hub is required when the connection string does not name an event hub. With Event Hub, the message reads event_hub is required for eventhubs target when the connection string does not name an event hub
• the connection string names no namespace
• For AMQP: the AMQP connection string must start with amqps://; the AMQP connection string starts with amqp://, but the endpoint accepts TLS connections only: use amqps://; the AMQP connection string is not a valid URL: …, with each / in the key written as %2F; the AMQP connection string has no host name; the AMQP connection string needs a key name and a key: …; the AMQP connection string names no event hub: put its name at the end of the URL or set event_hub
• For Event Hub with Microsoft Entra ID: namespace is required when not using connection string authentication for eventhubs target; event_hub is required for eventhubs target; tenant_id, client_id, and client_secret must all be provided together for service principal authentication

Fix: correct the setting the message names.

Fabric refuses the sign-in

Cause: at run time, a sign-in that Fabric refuses fails as it fails for the Azure Event Hubs target (Event Hub, AMQP) or the Apache Kafka target (Kafka). The usual causes: a wrong or regenerated key; with Entra ID, an identity without the Contributor role on the workspace, or a service principal while the tenant setting "Service principals can call Fabric public APIs" is off.

Fix: copy the current "Connection string-primary key" from the tab again, or give the identity the Contributor role on the workspace and turn the tenant setting on.