Google Cloud Pub/Sub

Google Cloud Message Queue

Synopsis

Creates a target that publishes messages to Google Cloud Pub/Sub topics with support for batch processing, message ordering, and service account authentication. Provides reliable message delivery to Google Cloud Pub/Sub for event-driven architectures and distributed systems.

Schema

- name: <string>
  description: <string>
  type: gcppubsub
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    project_id: <string>
    topic_id: <string>
    credentials_file: <string>
    credentials_json: <string>
    ordering_key: <string>
    max_messages: <numeric>
    max_bytes: <numeric>
    field_format: <string>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be gcppubsub
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Connection

Field	Required	Default	Description
project_id	Y	-	Google Cloud project ID
topic_id	Y	-	Pub/Sub topic ID
credentials_file	N*	-	Path to service account JSON key file
credentials_json	N*	-	Service account JSON key as string

* = Either credentials_file or credentials_json must be provided.

note

Service account must have roles/pubsub.publisher permission on the topic.

Message Configuration

Field	Required	Default	Description
ordering_key	N	-	Ordering key for message ordering within the topic
max_messages	N	1000	Maximum number of messages per batch
max_bytes	N	10485760	Maximum batch size in bytes (10 MB)
field_format	N	-	Data normalization format. See applicable Normalization section

Scheduler

Field	Required	Default	Description
interval	N	realtime	Execution frequency. See Interval for details
cron	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Google Cloud Pub/Sub target publishes messages to Pub/Sub topics for asynchronous, scalable message delivery. It supports batch processing for optimal performance and message ordering for maintaining event sequence.

Authentication

Authentication uses Google Cloud service account credentials. You can provide credentials either as a file path or as a JSON string. The service account must have the roles/pubsub.publisher role on the target topic.

Topic Management

The target automatically checks if the specified topic exists and creates it if needed. Topics are created in the specified Google Cloud project.

Message Ordering

When an ordering_key is set, Pub/Sub ensures that messages with the same ordering key are delivered in the order they were published. This is useful for maintaining event sequences for specific entities.

Example: Use device ID as ordering key to ensure events from the same device are processed in order.

Message Attributes

The target automatically adds the following attributes to each message:

• device_id - Source device identifier
• device_type - Type of source device
• device_name - Name of source device

These attributes can be used for message filtering and routing in subscriptions.

Batch Processing

Messages are published in batches for optimal throughput. The target accumulates messages until the batch size limit is reached or during finalization. Google Cloud Pub/Sub supports up to 10 MB per batch.

At-Least-Once Delivery

Pub/Sub guarantees at-least-once delivery. Messages may be delivered more than once in case of network issues or subscriber failures. Design your message handlers to be idempotent.

Message Retention

Messages are retained for 7 days by default. You can configure retention periods from 10 minutes to 7 days in the topic settings.

Dead Letter Topics

Google Cloud Pub/Sub supports dead letter topics for messages that cannot be processed after a configured number of delivery attempts. Configure this in the subscription settings.

Examples

The following are commonly used configuration types.

Basic with File Credentials

Creating a basic Pub/Sub target with credentials file...	- name: basic_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "application-logs" credentials_file: "/path/to/service-account-key.json" max_messages: 1000
Target publishes JSON messages to Pub/Sub topic...	{ "timestamp": "2024-01-15T10:30:00Z", "host": "server01", "message": "Application started successfully", "severity": "info" }

With JSON Credentials

Using credentials as JSON string (useful for secrets management)...

- name: json_creds_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "security-events" credentials_json: "${GCP_SERVICE_ACCOUNT_JSON}" max_messages: 500

With Message Ordering

Using ordering key to maintain message sequence...

- name: ordered_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "ordered-events" credentials_file: "/path/to/service-account-key.json" ordering_key: "device-001" max_messages: 250

High-Throughput Configuration

Optimizing for high-volume message publishing...

- name: high_volume_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "high-volume-logs" credentials_file: "/path/to/service-account-key.json" max_messages: 2000 max_bytes: 20971520

Multiple Topics

Publishing to different topics for different log types...

- name: application_logs_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "application-logs" credentials_file: "/path/to/service-account-key.json" - name: error_logs_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "error-logs" credentials_file: "/path/to/service-account-key.json"

Field Normalization

Using field normalization for standard format...

- name: normalized_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "normalized-logs" credentials_file: "/path/to/service-account-key.json" field_format: "ecs" max_messages: 1000

Pipeline Processing

Applying post-processing pipelines before publishing...

- name: pipeline_pubsub type: gcppubsub pipelines: - format_timestamp - add_metadata - validate_schema properties: project_id: "my-project-id" topic_id: "processed-events" credentials_file: "/path/to/service-account-key.json" max_messages: 750

Scheduled Batching

Configuration with scheduled batch delivery...

- name: scheduled_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "scheduled-logs" credentials_file: "/path/to/service-account-key.json" max_messages: 1000 interval: "5m"

Debug Configuration

Configuration with debugging enabled...

- name: debug_pubsub type: gcppubsub properties: project_id: "my-project-id" topic_id: "test-logs" credentials_file: "/path/to/service-account-key.json" debug: status: true dont_send_logs: true

Production Configuration

Configuration for production with optimal settings...

- name: production_pubsub type: gcppubsub pipelines: - checkpoint properties: project_id: "production-project" topic_id: "production-logs" credentials_json: "${GCP_SERVICE_ACCOUNT_JSON}" ordering_key: "production-cluster" max_messages: 1500 max_bytes: 15728640 field_format: "ecs"

Troubleshooting

Authentication Errors

If you encounter authentication errors:

• Verify service account JSON is valid and not expired
• Check the service account has roles/pubsub.publisher role
• Ensure the credentials file path is correct
• Verify the project ID matches the service account project

Topic Not Found

If topic errors occur:

• Verify topic ID is correct
• Check the topic exists in the specified project
• Ensure service account has permissions to create topics
• Verify project ID is correct

Permission Denied

If you see permission denied errors:

• Check service account has roles/pubsub.publisher role
• Verify IAM policies are correctly configured
• Ensure the service account is not disabled
• Check organization policies don't block Pub/Sub access

Message Size Exceeded

If messages exceed size limits:

• Google Cloud Pub/Sub supports maximum 10 MB per message
• Reduce message size before publishing
• Split large messages into smaller chunks
• Check max_bytes configuration

Ordering Key Errors

If message ordering fails:

• Verify ordering_key is set in configuration
• Ensure ordering is enabled on the topic
• Check subscription has ordering enabled
• Verify messages with same key go to same topic

Quota Exceeded

If you hit quota limits:

• Check your project's Pub/Sub quotas in GCP Console
• Request quota increases if needed
• Reduce publishing rate
• Monitor quota usage in Cloud Monitoring

Network Connectivity

If connection failures occur:

• Verify network connectivity to Google Cloud
• Check firewall rules allow outbound HTTPS
• Ensure DNS resolution works for pubsub.googleapis.com
• Review VPC network settings if using VPC Service Controls

Batch Limit Exceeded

If batch limits are reached:

• Reduce max_messages parameter
• Decrease max_bytes parameter
• Increase finalization frequency
• Monitor message sizes