Skip to main content

Processors: Catalog

Processors are fundamental components in log processing pipelines that perform specific operations on log data. They are responsible for transforming, enriching, and manipulating log entries as they flow through the system. Each processor is designed to handle a specific type of operation, from simple field modifications to complex data transformations.

🧠 AI​

AI processors harness the power of artificial intelligence APIs for sophisticated content analysis and processing. These processors utilize various AI services to perform advanced text analysis, classification, and generation tasks. They enable intelligent processing of content, making it possible to extract insights and meaning from complex data.

💹 Analytics​

Analytics processors gather and manipulate data to render the data points suitable for metrics and analyses. They select the data points that reveal critical information about the generators of data, and process them to make the relevant information contained in them more visible.

🧮 Arithmetic​

Arithmetic processors perform mathematical operations and calculations on numeric field values within log data. They support basic mathematical functions like addition, subtraction, multiplication, and division, as well as more complex operations such as calculating percentages, averages, and statistical computations. These processors enable quantitative analysis of log data by transforming raw numbers into meaningful metrics and derived values.

🔀 Flow Control​

Flow Control processors manage the execution paths and logic within processing pipelines. They direct how documents move through the system, handle conditional processing, filtering, and organize pipeline structure. These processors are essential for creating sophisticated processing logic and maintaining efficient pipeline organization.

🗄️ State Management​

State Management processors share state across pipelines and cluster nodes through a cluster-wide key-value cache. They write, read, and delete keys in a shared store, enabling patterns such as session correlation, deduplication, and short-lived enrichment caches that remain consistent on single-node and multi-node deployments.

⌚ Date and Time​

Date and Time processors handle temporal data operations including parsing, formatting, and manipulating date and time values. They convert between different date formats, extract time components, calculate time differences, and manage timezone conversions. These processors are essential for standardizing temporal data and performing time-based analysis on log entries.

💠 Enrich​

Enrichment processors enhance log data by incorporating additional context and information from external sources. They add value to existing data by integrating geographical information, performing DNS lookups, and adding domain intelligence. These processors connect with external databases and services to provide comprehensive context to your log data, making it more valuable for analysis and understanding.

🔧 Data Manipulation​

Data Manipulation processors modify existing data fields and values to ensure proper formatting and structure. They handle tasks such as appending values, converting data types, managing field structures, string manipulation, and data transformation. These processors are fundamental for maintaining data consistency and preparing information for further processing or analysis.

🔓 Decode​

Decode processors specialize in decoding and decrypting encoded data formats. They handle operations like JWT token decoding, ACL information extraction, hexadecimal to ASCII conversion, and other specialized decoding tasks. These processors are essential for converting encoded or encrypted data into readable formats for further processing.

🖧 Networking​

Networking processors handle network-related data operations and communications. They perform network protocol analysis, manage IP address operations, conduct DNS lookups, and handle network connectivity tasks. These processors are vital for processing network logs, analyzing network traffic patterns, and enriching data with network intelligence.

🔔 Notification​

Notification processors send alerts and notifications to external services when specific conditions are detected in the data pipeline. They integrate with incident management platforms, messaging services, and collaboration tools to enable real-time alerting on schema drift, validation failures, and other pipeline events.

📋 Parse​

Parsing processors transform raw data into structured formats by extracting meaningful information from various input types. They handle multiple data formats and message types, converting them into structured data. These processors excel at converting unstructured or semi-structured data into well-organized, usable formats by applying patterns and rules to extract relevant fields.

🛡️ Security​

Security processors focus on protecting sensitive information and managing data security. They implement encryption and decryption operations, generate document signatures, and handle data masking and redaction. These processors ensure that sensitive information is properly protected while maintaining the utility of the data for analysis.

📝 Text Processing​

Text Processing processors manipulate string values: changing case, trimming, padding, wrapping, slicing, repeating, counting occurrences and locating substrings. They operate on the characters of a field, with no language model behind them — for classification or analysis of meaning, use the AI processors.

🕵️ Threat Intelligence​

Threat Intelligence processors integrate with external security services to provide context about potential security threats. They connect with various threat intelligence providers to retrieve and incorporate security data. These processors are crucial for security analysis and threat detection, providing real-time intelligence about potential security risks.