ClickHouse

Databases Analytics

Synopsis

Creates a ClickHouse target that sends log data to a ClickHouse database server for analytics and storage. Supports batch processing and field normalization.

Schema

- name: <string>
  description: <string>
  type: clickhouse
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <integer>
    username: <string>
    password: <string>
    database: <string>
    table: <string>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be clickhouse
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
address	Y	-	ClickHouse server address
port	N	9000	ClickHouse server port (native protocol)
username	Y	-	ClickHouse username
password	Y	-	ClickHouse password
database	Y	-	ClickHouse database name
table	Y	-	ClickHouse table name
max_bytes	N	67108864 (64 MB)	Maximum size of a single request body or batch, in bytes

note

max_bytes caps the size of a single request body or batch. It behaves differently from max_size on the file-writing targets, and the difference is easy to get wrong:

• An explicit max_bytes: 0 IS honoured and removes the ceiling. The value is read with Int64, which returns what you wrote — unlike max_size, which is read with MustInt64 and silently substitutes its default for a zero.
• A negative value is rejected at configuration time with max_bytes must not be negative.

The ceiling is validated when the target is built and enforced again on every batch, so a value the receiving service will refuse fails early rather than per request.

Processing

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section

note

batch_size is a Director service-level setting configured in vmetric.yml (DefaultWriteBatchSize = 1000) and cannot be overridden per target.

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The ClickHouse target uses the native ClickHouse protocol to efficiently send log data in batches. Logs are accumulated until the batch size is reached, then sent to the server. The default batch size is defined by the service configuration, but can be overridden.

The target supports field normalization to convert log data into standard formats like Elastic Common Schema (ECS), Common Information Model (CIM), or Advanced Security Information Model (ASIM) before sending it to ClickHouse.

Prerequisites

1. A running ClickHouse server
2. A database and table already created in ClickHouse
3. A user with write permissions to the specified table

note

The ClickHouse table should have a schema compatible with the log data being sent. At minimum, it should include columns for timestamp and message fields.

warning

For high-volume logging, ensure your ClickHouse server is properly configured for performance, including appropriate settings for inserts, memory usage, and disk I/O.

Examples

Basic

Minimum configuration for sending logs to ClickHouse:

targets:
  - name: basic_clickhouse
    type: clickhouse
    properties:
      address: "192.168.1.100"
      username: "default"
      password: "password"
      database: "logs"
      table: "system_logs"

Custom Port

Configuration with a non-default port:

targets:
  - name: custom_port_clickhouse
    type: clickhouse
    properties:
      address: "clickhouse.example.com"
      port: 9440
      username: "logger"
      password: "secure_password"
      database: "logs"
      table: "application_logs"

With Normalization

Configuration using field normalization:

targets:
  - name: normalized_clickhouse
    type: clickhouse
    properties:
      address: "clickhouse.example.com"
      username: "logger"
      password: "secure_password"
      database: "logs"
      table: "security_logs"
      field_format: "ecs"

With Pipeline

Using a pipeline for additional log processing:

targets:
  - name: pipeline_clickhouse
    type: clickhouse
    pipelines:
      - enrich_logs
    properties:
      address: "clickhouse.example.com"
      username: "logger"
      password: "secure_password"
      database: "logs"
      table: "enriched_logs"

Secure Configuration

Using environment variables for credentials:

targets:
  - name: secure_clickhouse
    type: clickhouse
    properties:
      address: "${CLICKHOUSE_ADDRESS}"
      username: "${CLICKHOUSE_USER}"
      password: "${CLICKHOUSE_PASSWORD}"
      database: "logs"
      table: "security_logs"