Skip to main content

Azure Logs

Microsoft Azure Pull

Synopsis​

Creates an Azure Logs client that queries Log Analytics workspaces using Kusto Query Language (KQL). Collects log data from specified tables with configurable batch sizes and collection frequencies.

Schema​

- id: <numeric>
  name: <string>
  description: <string>
  type: azlogs
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    workspace_id: <string>
    stream: <string[]>
    batch_size: <numeric>
    event_frequency: <numeric>

Configuration​

The following fields are used to define the device.

Device​

FieldRequiredDefaultDescription
idYUnique identifier
nameYDevice name
descriptionN-Optional description
typeYMust be azlogs
tagsN-Optional tags
pipelinesN-Optional pre-processor pipelines
statusNtrueEnable/disable the device

Authentication​

FieldRequiredDefaultDescription
tenant_idYAzure tenant ID
client_idYAzure client ID
client_secretYAzure client secret
workspace_idYLog Analytics workspace ID

Events​

FieldRequiredDefaultDescription
streamYArray of Log Analytics queries to collect
batch_sizeN1000Number of log entries to collect per batch
event_frequencyN300Collection frequency in seconds

Examples​

The following are commonly used configuration types.

Basic​

The minimum required configuration:

Creating a basic collector...

devices:
  - id: 1
    name: basic_azlogs
    type: azlogs
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "11111111-1111-1111-1111-111111111111"
      client_secret: "your-client-secret"
      workspace_id: "22222222-2222-2222-2222-222222222222"
      stream:
        - "SecurityEvent"

Multiple Streams​

The collecter can consume multiple log types with pre-processing:

Specifying multiple log streams...

devices:
  - id: 2
    name: multi_stream_azlogs
    type: azlogs
    pipelines:
      - security_events
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "11111111-1111-1111-1111-111111111111"
      client_secret: "your-client-secret"
      workspace_id: "22222222-2222-2222-2222-222222222222"
      stream:
        - "SecurityEvent"
        - "Syslog"
        - "AzureActivity"
      batch_size: 2000
note

The security_events pipeline can be used to process and enrich security-related log entries before ingestion.

High Volumes​

Performance can be enhanced for high log volumes:

Optimizing for high volumes...

devices:
  - id: 3
    name: high_volume_azlogs
    type: azlogs
    properties:
      tenant_id: "00000000-0000-0000-0000-000000000000"
      client_id: "11111111-1111-1111-1111-111111111111"
      client_secret: "your-client-secret"
      workspace_id: "22222222-2222-2222-2222-222222222222"
      stream:
        - "SecurityEvent | where Level == 'Critical' or Level == 'Error'"
        - "Syslog | where Facility == 'auth'"
      batch_size: 5000
warning

Large batch sizes may impact memory usage and processing time. Monitor system resources and adjust accordingly.