Skip to main content

Splunk S2S

Splunk Observability

Synopsis​

Creates a target that sends events to a Splunk indexer over Splunk's native Splunk-to-Splunk (S2S) protocol on port 9997 — the wire protocol a Splunk universal forwarder speaks, as opposed to the HTTP Event Collector used by the splunk and splunksecurity targets. Supports weighted load balancing across an indexer tier, protocol v3/v4 framing, delivery acknowledgement, and compression.

Schema​

- name: <string>
  description: <string>
  type: splunks2s
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <numeric>
    servers: <string[]>
    destinations:
      - address: <string>
        port: <numeric>
        weight: <numeric>
    load_balance_seconds: <numeric>
    load_balance_bytes: <numeric>
    load_balance_stats_seconds: <numeric>
    dns_resolve_seconds: <numeric>
    destination_down_seconds: <numeric>
    protocol: <string>
    index: <string>
    source: <string>
    source_type: <string>
    host: <string>
    extract_metadata: <boolean>
    ack: <boolean>
    ack_timeout: <numeric>
    compression: <boolean>
    server_name: <string>
    mgmt_port: <string>
    batch_size: <numeric>
    timeout: <numeric>
    max_retries: <numeric>
    retry_delay: <numeric>
    disable_retries: <boolean>
    max_event_bytes: <numeric>
    nested_field_serialization: <string>
    throttle_bytes_per_second: <numeric>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>
    tls:
      status: <boolean>
      verify: <boolean>
      server_name: <string>
      ca_name: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
      min_tls_version: <string>
      max_tls_version: <string>

Configuration​

The following are the fields used to define the target:

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be splunks2s
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

Destinations​

FieldRequiredDefaultDescription
addressY*-Single indexer address (hostname or IP), paired with port
portN9997Port for address, and the default port for any destinations entry that omits one
serversN-Array of "host:port" strings, one indexer per entry
destinationsN-Weighted destination list. Each entry: address, port (default 9997), weight (default 1)

* = address, servers, and destinations are combinable and additive; at least one is required.

Load Balancing​

FieldRequiredDefaultDescription
load_balance_secondsN30Seconds a worker holds one connection before reconsidering the destination. 0 pins the connection
load_balance_bytesN0Bytes written on one connection before it is rotated. 0 rotates on time only
load_balance_stats_secondsN300Window over which per-destination volume is remembered when apportioning load; half of it carries forward each rollover
dns_resolve_secondsN600How often a configured hostname is re-resolved
destination_down_secondsN30How long a destination that refused a connection is passed over before being retried

Protocol​

FieldRequiredDefaultDescription
protocolN"v3"Wire framing: v3 (legacy cooked, default), v4 (compact framing), auto (negotiate v4, fall back to v3)
ackNfalseWait for indexer acknowledgement before reporting a batch delivered. Requires protocol: v4 or auto
ack_timeoutN60Seconds to wait for acknowledgement before a batch counts as undelivered. Only meaningful with ack: true
compressionNfalsezlib-compress the connection
server_nameNproduct nameIdentity presented to the indexer during the handshake
mgmt_portN"8089"Management port advertised during the handshake. Nothing connects to it

Event Metadata​

FieldRequiredDefaultDescription
indexN-Default Splunk index stamped on every event
sourceN-Default source
source_typeN-Default sourcetype
hostN-Default host
extract_metadataNtrueLift _raw/host/source/sourcetype/index/time out of a Splunk-shaped JSON envelope. Explicitly configured metadata above still wins
nested_field_serializationN"none"How an object/array-valued envelope field is handled: none (drop it) or json (forward its JSON text)

Batching and Delivery​

FieldRequiredDefaultDescription
batch_sizeN1000Events buffered before a flush
timeoutN30Connection, handshake, and write timeout in seconds
max_retriesN3Retries after a failed delivery attempt. The default of 3 makes four attempts in all
retry_delayN1Base for the wait between retry attempts, in seconds. Each wait is a random interval that doubles per attempt, capped at 30 seconds
disable_retriesNfalsetrue makes one attempt per batch and never waits to retry it, whatever max_retries holds. The payload that attempt fails is still handed back and redelivered by the sender, as any failed send is. Absent or false, nothing changes. In the console this is the Disable retries switch
max_event_bytesN66060288 (~63 MB)Ceiling on one event's payload. A larger event has its tail cut to this size and the shortened event is sent
throttle_bytes_per_secondN0Ceiling on outbound bytes per second per sender worker, measured post-compression. 0 disables it

TLS​

FieldRequiredDefaultDescription
tls.statusNfalseEnable TLS
tls.verifyNtrueVerify the server certificate
tls.server_nameN-SNI and expected server hostname. With tls.verify enabled, an address is verified directly when the certificate carries that address. Set this when it does not, or when the indexer expects a hostname for SNI
tls.ca_nameN-Custom CA to trust. Empty uses the OS trust store
tls.cert_nameN*-Client certificate for mutual TLS
tls.key_nameN*-Client key for mutual TLS
tls.passphraseN-Passphrase for an encrypted client key
tls.min_tls_versionNtls1.2Minimum negotiated TLS version
tls.max_tls_versionNtls1.3Maximum negotiated TLS version

* = Conditionally required. tls.cert_name and tls.key_name must be supplied together; supplying only one fails configuration validation.

note

TLS material fields (cert_name, key_name, ca_name, client_ca_name) accept any of the following:

  • File name — resolved relative to the service root directory. Nested paths such as certs/prod/server.pem are supported.
  • Absolute path — honored only if it resolves inside the service root. Any path that escapes the root is refused.
  • Inline PEM content — used verbatim when the value contains -----BEGIN.
  • Environment variable — ${ENV_VAR}.
  • Vault reference — $secret{id=...} or $secret{store=...,ref=...}.

Normalization​

FieldRequiredDefaultDescription
field_formatN-Data normalization format. See applicable Normalization section

Scheduling​

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options​

FieldRequiredDefaultDescription
debug.statusNfalseEnable debug logging
debug.dont_send_logsNfalseProcess logs but don't send to target (testing)

Details​

The Splunk S2S target speaks the same wire protocol a Splunk universal forwarder uses on port 9997, so an indexer accepts the data as cooked, pre-attributed events with no HEC token and no ingest-time parsing. It is a single-type target — splunks2s is its only registered type, unlike splunk/splunksecurity/crowdstrike, which share one HEC implementation.

Indexer Load Balancing​

address/port configure a single destination, servers a plain list, and destinations a weighted list — the three forms are combinable and additive; each host:port is kept once, so to weight a servers member, move it into destinations. A weight is a share of traffic, not an absolute value: keep weights within the same order of magnitude (1/5/4 works; 10/120/9 does not). Weight 0 parks a destination — configured and reachable as a last resort, but never chosen while any weighted destination is up.

A configured hostname is resolved to every address it has, and each resolved address becomes its own destination — so a single name fronting an indexer tier spreads traffic across the whole tier instead of pinning one member. TLS is still verified against the configured hostname, not the resolved address.

Two settings trigger rotation to a new destination:

  • load_balance_seconds is how long a worker holds one connection before reconsidering, mirroring Splunk's own autoLBFrequency default of 30. Without it, the first indexer a worker reaches keeps every batch that worker ever sends — an indexer added to the tier never receives anything, and one that was briefly down never gets its share back. 0 pins the connection.
  • load_balance_bytes is the volume trigger, and the one that matters for bursty input: a batch that arrives all at once fits inside any sensible time interval, so a time-only rule sends the whole burst to whichever destination the first batch picked. Measured against two indexers with time-only rotation, a 500-event burst went 500/0. Splunk carries the same pair of levers as autoLBFrequency and autoLBVolume. 0 rotates on time only.

load_balance_stats_seconds is the window over which per-destination volume is remembered when apportioning load; half of it carries forward on each rollover, so short-term imbalance decays instead of either persisting indefinitely or being forgotten between windows. dns_resolve_seconds controls how often a configured hostname is re-resolved — a resolution that fails keeps the last known address set rather than emptying it. destination_down_seconds is how long a destination that refused a connection is passed over; the cooldown is shared across every worker of the target, so one worker's discovery is not re-learned by the rest.

Protocol Versions​

protocol: v3 (the default) sends legacy cooked messages, where each message carries its own _done terminator — the indexer takes the event exactly as sent, so the original timestamp is preserved and a multi-line event stays one event. v4 (or auto, which negotiates v4 and falls back to v3 if the receiver refuses it) sends the compact framing a Universal Forwarder uses, which hands event-boundary detection and timestamp extraction back to the indexer. ack: true requires protocol: v4 or auto — acknowledgement rides on v4 message IDs, so it cannot be combined with an explicitly configured v3.

Event Metadata​

index, source, source_type, and host set the attribution stamped on every event; leaving them empty lets the source's own attribution through. When extract_metadata is enabled (the default), the target lifts _raw/host/source/sourcetype/index/time out of a Splunk-shaped JSON envelope, so data received from another Splunk source keeps its original attribution instead of arriving as an opaque JSON blob — any of the fields above that are explicitly set still take priority. Every other envelope member is forwarded on as an indexed field; nested_field_serialization decides what happens when one of those fields is an object or array, which has no representation in Splunk's flat, text-only indexed fields: none (the default) drops it, json forwards its JSON text.

If an event has no host attribution after metadata resolution, the target falls back to the originating device's name rather than letting Splunk misattribute it to the receiving indexer. Setting schema, source, or log_type in a pipeline overrides source_type for that event, the same last-write-wins mechanism the splunk target uses for its sourcetype override.

Compression​

compression: true zlib-compresses the connection, as a forwarder's compressed = true setting does. It is worth enabling on a slow or metered link — a 211 KB batch of repetitive log text went out in 3.7 KB in testing — at the cost of CPU on both ends.

Payload Limits​

max_event_bytes bounds a single event's payload before it is framed. Splunk's receiver enforces a hard ceiling near 64 MB and does not report it as an error — it resets the TCP connection instead. An event that exceeds max_event_bytes has its tail cut to that size and is sent shortened, so the batch still goes through and the record is still indexed, minus its tail. Without a ceiling, an oversized event that keeps getting retried resets the connection every time and wedges the destination in a reconnect loop that looks like a network fault.

Delivery and Retry​

max_retries and retry_delay only absorb a brief blip in reaching a destination, and disable_retries: true skips them: each batch is sent once. Once retries are exhausted, or with retries disabled on the first failure, the target returns the send error and the sender pool NAKs the source records for redelivery — which is what actually prevents loss, and which requires persistent_storage to be enabled on the Director.

Throttling​

throttle_bytes_per_second caps outbound bytes per second on each sender worker, measured on what is actually written to the wire — so on a compressed connection the ceiling applies to the post-compression byte count. 0 disables it.

Examples​

Basic​

Sending events to a single indexer over the S2S protocol...

targets:
  - name: splunks2s_basic
    type: splunks2s
    properties:
      address: "idx1.example.com"
      port: 9997

Indexer Tier with Failover​

Spreading events across an indexer tier with automatic failover...

targets:
  - name: splunks2s_tier
    type: splunks2s
    properties:
      servers:
        - "idx1.example.com:9997"
        - "idx2.example.com:9997"
        - "idx3.example.com:9997"

Weighted Destinations​

Favoring a larger indexer over two smaller ones with weighted destinations...

targets:
  - name: splunks2s_weighted
    type: splunks2s
    properties:
      destinations:
        - address: "idx1.example.com"
          port: 9997
          weight: 5
        - address: "idx2.example.com"
          port: 9997
          weight: 1
        - address: "idx3.example.com"
          port: 9997
          weight: 1

Acknowledged Delivery​

Waiting for indexer acknowledgement on the compact v4 framing...

targets:
  - name: splunks2s_acked
    type: splunks2s
    properties:
      address: "idx1.example.com"
      port: 9997
      protocol: auto
      ack: true
      ack_timeout: 60

Secure (TLS)​

Connecting to an indexer tier over TLS, verified by hostname...

targets:
  - name: splunks2s_tls
    type: splunks2s
    properties:
      address: "indexers.example.com"
      port: 9997
      tls:
        status: true
        verify: true
        server_name: "indexers.example.com"
        ca_name: "splunk-ca.pem"
        min_tls_version: "tls1.2"

High-Volume with Compression​

Compressing the connection and capping outbound bandwidth for a bursty, high-volume source...

targets:
  - name: splunks2s_highvolume
    type: splunks2s
    properties:
      address: "idx1.example.com"
      port: 9997
      batch_size: 5000
      compression: true
      load_balance_bytes: 10485760
      throttle_bytes_per_second: 5242880

Troubleshooting​

This section covers the errors you are most likely to see with the splunks2s target, what causes each one, and how to fix it.

Where to look:

  • Director logs. Target errors are tagged with the target name and carry "Section":"SenderPool". The part after Reason: or after the last colon is the actual cause.
  • The target's connection status in the web interface. It shows the same reason as the log line.

See Target Delivery Errors for how Director logs and retries target failures.

This target does not open a connection when it starts. The first connection to an indexer is made when the first batch is ready to send. An indexer that is down, a closed port, and a port that belongs to something else therefore never show as a failed connection: the connection status in the web interface stays positive, and the only sign of trouble is the delivery errors below. The status turns negative only for faults Director can see without the network, such as certificate material it cannot read.

Every failure on this page is retried, and this target never gives up on a batch. A permanent mistake such as the wrong port is redelivered for as long as it lasts. Nothing is dropped, but nothing moves either until you fix the cause.

What the indexer needs​

This protocol carries no token and no password. What the target needs is a receiving port it can reach and, with TLS, certificates both sides accept.

PrerequisiteWhen it appliesError when it is missing
A receiving input on the port, [splunktcp://9997] in the indexer's inputs.confAlwaysconnect: connection refused
An SSL receiving input, [splunktcp-ssl:9997] with an [SSL] stanzatls.status: truefirst record does not look like a TLS handshake, or EOF when the mismatch runs the other way
Outbound TCP from the Director host to every indexer on the configured port, and DNS for every configured hostnameAlwaysi/o timeout, no such host
The indexer accepts a connection from the Director host, for example through its acceptFrom settingAlwaysreceiver refused the connection: ..., typically
The CA that signed the indexer certificate, in tls.ca_name or in the Director host's trust storetls.status: true with tls.verify: truex509: certificate signed by unknown authority
A client certificate and key in tls.cert_name and tls.key_name, signed by a CA the indexer trusts through sslRootCAPathOnly when the indexer sets requireClientCert = trueremote error: tls: certificate required
Read permission for the Director service account on the certificate, key, and CA filesTLS material given as file pathsread key "...": ... permission denied
The index named in index exists on the indexerWhen index is set, or an envelope carries oneNone. The indexer typically discards events for an index that does not exist

"connect: connection refused"​

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 3 execute() failed for splunks2s_basic: failed to send after 3 retries: failed to connect to indexer.example.com:9997: dial tcp indexer.example.com:9997: connect: connection refused

Cause: nothing is listening on that port. On a Splunk indexer this almost always means receiving is not enabled, because an indexer does not listen on 9997 until you turn it on. The other causes are a port that is not the receiving port, and an indexer whose splunkd is stopped.

Fix: on the indexer, open Settings > Forwarding and receiving > Configure receiving and add the port, or add a [splunktcp://9997] stanza to inputs.conf. splunk btool inputs list splunktcp shows what is configured. Then check the port in address, servers, and destinations. When a tier has several destinations and all of them refuse, the message names only one of them, so check every indexer in the tier rather than the one in the log line. No data is lost: the target makes four attempts, then the payload returns to the queue and is redelivered until an indexer accepts it.

"x509: certificate signed by unknown authority"​

[Error] [director] [target-<target id>] [splunks2s_tls] Sender worker 1 execute() failed for splunks2s_tls: failed to send after 3 retries: failed to connect to indexer.example.com:9997: tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause: tls.status is on, and the certificate the indexer presents is not signed by a CA Director trusts. A Splunk installation ships with its own self-signed certificate, so this is the normal result of pointing TLS at a default indexer while tls.ca_name is empty and the host trust store is used.

Fix: set tls.ca_name to the CA chain that signed the indexer certificate. On a default installation that is Splunk's own cacert.pem under $SPLUNK_HOME/etc/auth. Copy it where the Director service can read it, or paste it inline as PEM. Nothing is lost while Director keeps running, within the queue retention limit, and it is retried until you fix the cause.

warning

tls.verify: false also makes the error go away, and takes the protection with it. Use it only in a lab.

"certificate is valid for ..., not ..." or "doesn't contain any IP SANs"​

[Error] [director] [target-<target id>] [splunks2s_tls] Sender worker 1 execute() failed for splunks2s_tls: failed to send after 3 retries: failed to connect to indexer.example.com:9997: tls: failed to verify certificate: x509: certificate is valid for indexer.example.com, not indexers.example.com

Cause: the certificate is trusted, but the name being verified is not on it. tls.server_name is the name that is checked, and with no tls.server_name the configured hostname is used instead.

Fix: set tls.server_name to a name the certificate carries, or configure the destination under a name the certificate carries. The second form of the error is what a destination configured as an address gives you instead:

... failed to connect to <indexer address>:9997: tls: failed to verify certificate: x509: cannot validate certificate for <indexer address> because it doesn't contain any IP SANs

An address has no hostname to fall back on, so verification needs the certificate to carry that address as a subject alternative name. Set tls.server_name whenever a destination is an address and tls.verify is on. A configured hostname is resolved to all of its addresses and each one is dialled separately, but verification still uses the hostname, so a name-based destination does not need this.

"stream desynchronised: v3 body length 1213486160"​

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 4 execute() failed for splunks2s_basic: failed to send after 3 retries: handshake with indexer.example.com:8088: splunks2s: stream desynchronised: v3 body length 1213486160

Cause: the port answered, but not with S2S. The first four bytes of the reply are read as the length of the message that follows, and a length outside the 4 byte to 64 MiB range produces this message. The number is the giveaway. 1213486160 is the four letters HTTP read as a length, so the port is serving HTTP: the HTTP Event Collector, the web interface, or the management port.

Fix: point the target at the indexer's receiving port, 9997 by default. This target cannot use an HTTP Event Collector port. If what you have is a HEC token rather than a receiving port, use the splunk target instead, which speaks HTTP to that endpoint. mgmt_port is not involved: it is only a number the target announces about itself, and changing it never fixes this error.

"first record does not look like a TLS handshake", or a handshake that ends in "EOF"​

[Error] [director] [target-<target id>] [splunks2s_tls] Sender worker 2 execute() failed for splunks2s_tls: failed to send after 3 retries: failed to connect to indexer.example.com:9997: tls: first record does not look like a TLS handshake
[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 2 execute() failed for splunks2s_basic: failed to send after 3 retries: handshake with indexer.example.com:9997: EOF

Cause: the two sides disagree about TLS. The first message means tls.status: true was used against a plaintext receiving input. The second means the indexer accepted the connection and then closed it without answering, which usually means the opposite mismatch: the input is SSL only while tls.status is off. compression: true against a receiver that cannot read compressed messages ends the same way, because there is nothing that negotiates it.

Fix: make both ends agree. Either configure [splunktcp-ssl:9997] with an [SSL] stanza on the indexer and set tls.status: true, or leave both sides plaintext. If TLS already matches, turn compression off and try again.

"receiver refused the connection"​

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 1 execute() failed for splunks2s_basic: failed to send after 3 retries: handshake with indexer.example.com:9997: receiver refused the connection: <the reply the indexer sent>

Cause: the connection was established, and the indexer answered the capability exchange with a refusal instead of an acceptance. The text after the last colon is that reply. It typically carries a cap_response=failure field with a reason= value naming the refusal, so read that value first.

Fix: the decision was made on the indexer, so its own logs hold the detail. Look in splunkd.log for TcpInputProc entries at the same time, then review the [splunktcp] settings that can refuse a sender, acceptFrom first. Nothing is lost, and every batch is retried until the indexer accepts the connection.

"i/o timeout" or "no such host" while connecting​

[Error] [director] [target-<target id>] [splunks2s_tier] Sender worker 0 execute() failed for splunks2s_tier: failed to send after 3 retries: failed to connect to indexer.example.com:9997: dial tcp indexer.example.com:9997: i/o timeout

Cause: i/o timeout means the connection attempt went unanswered for the whole of timeout, 30 seconds by default, which is what a firewall that drops packets rather than refusing them looks like. The other form of this error, dial tcp: lookup indexer.example.com: no such host, means the name does not resolve on the Director host.

Fix: allow outbound TCP to every indexer on the configured port, and confirm the address. For the name, fix DNS or the hosts file, or configure addresses instead. Nothing is lost while either lasts. Note that a hostname is re-resolved every dns_resolve_seconds, and a re-resolution that fails keeps the last known addresses rather than emptying the list, so a tier that has moved keeps being dialled at its old addresses until those refuse.

"write: i/o timeout", or an acknowledgement that never arrives​

[Error] [director] [target-<target id>] [splunks2s_basic] Sender worker 3 execute() failed for splunks2s_basic: failed to send after 3 retries: write tcp ...->indexer.example.com:9997: write: i/o timeout

Cause: the indexer took the connection but stopped keeping up. This message means it stopped reading from the socket before timeout expired, which on an indexer points at blocked parsing or indexing queues. With ack: true the same stall arrives as waiting for acknowledgement: read tcp ...: i/o timeout, meaning the indexer accepted the bytes and did not acknowledge them within ack_timeout, 60 seconds by default.

Fix: look at the indexer's queue health in metrics.log, and at blocked-queue messages in splunkd.log, before changing anything here. Raise timeout or ack_timeout only when the link itself is slow. The batch counts as undelivered and is sent again, so events the indexer did write are indexed twice.

"connection reset by peer" or "broken pipe"​

[Error] [director] [target-<target id>] [splunks2s_highvolume] Sender worker 2 execute() failed for splunks2s_highvolume: failed to send after 3 retries: write tcp ...->indexer.example.com:9997: write: connection reset by peer

Cause: the indexer closed the connection in the middle of the batch. The usual reasons are an indexer restart, a sender the indexer will not accept, and a single event above Splunk's receive ceiling near 64 MB. Splunk resets the connection for an oversized event instead of reporting one, so a raised max_event_bytes shows up here looking like a network fault.

Fix: check whether the indexer restarted, and keep max_event_bytes at or below its default. The destination is passed over for destination_down_seconds and the whole batch is retried on a fresh connection, so whatever was written before the reset may be indexed twice.

"acknowledgement was requested but the receiver refused v4"​

[Error] [director] [target-<target id>] [splunks2s_acked] Sender worker 3 execute() failed for splunks2s_acked: failed to send after 3 retries: acknowledgement was requested but the receiver refused v4, so no message ids exist to acknowledge

Cause: acknowledgement rides on v4 message IDs, and the receiver did not grant v4. It either held the connection on v3, or never answered the capability exchange at all, which is treated the same way. A Splunk indexer grants v4, so in practice this means the port belongs to something that is not one.

Fix: confirm the port is a Splunk receiving input. If the receiver really is a third-party sink that speaks only v3, remove ack. Every batch fails this way until the configuration or the receiver changes, and each one is redelivered rather than dropped, so the queue grows in the meantime.

Configuration rejected before anything is sent​

These arrive as ValidateConfig failed for target "splunks2s_basic": ... and are re-checked until the configuration changes. Nothing is sent in the meantime.

Reason textFix
invalid server "...": expected host:portEvery servers entry needs a port, and an IPv6 literal needs brackets, [...]:9997. Only address may be written without one
address (or servers/destinations) is required for splunks2s targetSet at least one of address, servers, and destinations
destination 0: address is requiredEvery destinations row needs an address. The number is the row's position in the list
destination "...": weight must be zero or greater, got -5Weights cannot be negative. 0 parks a destination rather than removing it
port must be between 1 and 65535, got 70000Correct the port
invalid protocol "V4" (must be "v3", "v4" or "auto")The value is case-sensitive. Write it in lower case
ack is only supported with protocol v4 or autoack: true needs protocol: auto or protocol: v4. The default is v3
invalid nested_field_serialization "flatten" (must be "none" or "json")Use none or json
batch_size must be greater than 0, got -1Remove the setting to take the default, or set a positive value
both tls.cert_name and tls.key_name must be provided together or omitted togetherSupply both, or neither

"failed to build TLS config"​

TLS material is loaded before any connection is attempted, so these are the one family of faults that does reach the connection status: Failed to reinitialize target "splunks2s_tls" (attempt 1). Reason: failed to build TLS config: .... Retries back off from 5 seconds to once a minute and continue until the material loads. Incoming data waits in the queue, and no restart is needed once the file is in place.

Reason after failed to build TLS config:Fix
ca_name: "..." could not be resolved (env/vault token, inline PEM, or a path under the service root)The file is missing, unreadable, or outside the Director service root. Move it under the service root and reference it relatively, or paste the PEM inline
ca_name: credential: env variable "..." is not setThe ${...} reference names a variable the Director service does not have. Export it for the service itself, not only in your own shell
ca_name "..." contains no valid PEM certificate(s)The file is not PEM. Convert it, so that it starts with -----BEGIN CERTIFICATE-----
client certificate: read key "...": ... permission deniedGrant the Director service account read access to the key
client certificate: decrypt PKCS#8 private key (wrong passphrase?): ...Correct tls.passphrase, or supply a key that is not encrypted
client certificate: tls: private key does not match public keytls.cert_name and tls.key_name come from different pairs
min TLS version "tls1.3" is greater than max TLS version "tls1.2"Correct the tls.min_tls_version and tls.max_tls_version range

The target is healthy but nothing arrives in the index​

Nothing fails here, so there is no error to search for. Check the following in order.

  1. debug.dont_send_logs is enabled. Records are processed by the pipeline and then discarded before anything is buffered. No counter moves, nothing reaches the indexer, and the target reports healthy. The only trace is one line at startup, and only when debug.status is enabled as well.

    Log sending is disabled for this target (splunks2s_basic). Logs will be processed by the pipeline but will not be sent to the target.

  2. The index does not exist on the indexer. Delivery succeeds because the indexer accepts the events first and decides where to put them afterwards. An index that does not exist typically means they are discarded on the indexer, with nothing reported back and nothing in the target's counters, unless the indexer is configured with a last-chance index. Create the index, or set index to one that exists. An event sent with no index at all typically lands in the indexer's default index, main.

  3. The port is not a Splunk receiving input. A plain TCP listener that accepts the connection and never answers the capability exchange is treated as a receiver that speaks only v3. Events are written to it and counted as delivered while the receiver makes nothing of them. Each new connection also waits the whole of timeout for the answer that never comes, which shows up as a stall every load_balance_seconds. Confirm the port, and set ack: true with protocol: auto to turn the silence into the hard error above.

  4. Delivery is not acknowledged by default. With ack off, a batch counts as delivered once it has been written to the socket. An indexer that closes, or that drops the data just after accepting it, loses that batch with no error anywhere. Set ack: true with protocol: auto for at-least-once delivery.

  5. Fields are missing from the events that do arrive. With nested_field_serialization at its default of none, an envelope field whose value is an object or an array is dropped from the event, as are null and empty values. The event itself is delivered. Set nested_field_serialization: json to forward them as JSON text. An index, source, source_type, or host configured on the target also replaces whatever the envelope carried, silently.

  6. One indexer in the tier is dead. When one destination refuses a connection and another accepts, the batch succeeds and nothing is logged. The failed destination is passed over for destination_down_seconds and tried again later. A destination with weight: 0 is never chosen at all while a weighted one is available. Watch the per-destination event counts in the target's Indexers summary for a member that has stopped receiving its share.

  7. Events are shorter than they were. Events above max_event_bytes have their tails cut, and the shortened versions are delivered. One warning per batch reports it, carrying the number of events it shortened. No target counter moves.

    [SplunkS2S] truncated file=<payload> target=splunks2s_basic events=2 limit=66060288