This connector allows Kafka Connect to emulate a Splunk Http Event Collector. This connector support receiving data and writing data to Splunk.

Configuration

SplunkHttpSinkConnector

The Sink Connector will transform data from a Kafka topic into a batch of json messages that will be written via HTTP to a configured Splunk Http Event Collector.

name=connector1
tasks.max=1
connector.class=com.github.jcustenborder.kafka.connect.splunk.SplunkHttpSinkConnector

# Set these required values
splunk.remote.host=
splunk.auth.token=

SplunkHttpSourceConnector

The Splunk Source connector allows emulates a Splunk Http Event Collector to allow application that normally log to Splunk to instead write to Kafka. The goal of this plugin is to make the change nearly transparent to the user. This plugin currently has support for X-Forwarded-For so it will sit behind a load balancer nicely.

name=connector1
tasks.max=1
connector.class=com.github.jcustenborder.kafka.connect.splunk.SplunkHttpSourceConnector

# Set these required values
splunk.ssl.key.store.password=
splunk.collector.index.default=
splunk.ssl.key.store.path=
kafka.topic=

Schemas

com.github.jcustenborder.kafka.connect.splunk.EventKey

This schema represents the key for the data received from the Splunk listener.

com.github.jcustenborder.kafka.connect.splunk.Event

This schema represents the data received from the Splunk listener.

Example Config

This configuration will write to Splunk over SSL but will not verify the certificate.

name=splunk-http-sink
topics=syslog-udp
tasks.max=1
connector.class=com.github.jcustenborder.kafka.connect.splunk.SplunkHttpSinkConnector
splunk.remote.host=192.168.99.100
splunk.remote.port=8088
splunk.ssl.enabled=true
splunk.ssl.validate.certs=false
splunk.auth.token=**********

Writing data to Splunk.

The Sink Connector uses the Splunk Http Event Collector as it's target to write data to Splunk. To use this plugin you will need to configure an endpoint.

The Sink Connector will pull over all of the fields that are in the incoming schema. If there is a timestamp field named date or time it will be converted to a Splunk timestamp and moved to the time field. The host or hostname if it exists will be placed in the host field. All other fields will be copied to the event object.

Here is an example of an event generated by Kafka Connect Syslog written to Splunk.

{
  "host": "vpn.example.com",
  "time": 1472342182,
  "event": {
    "charset": "UTF-8",
    "level": "6",
    "remote_address": "\/10.10.0.1:514",
    "message": "filterlog: 9,16777216,,1000000103,igb2,match,block,in,4,0x0,,64,5581,0,none,6,tcp,40,10.10.1.22,72.21.194.87,55450,443,0,A,,2551909476,8192,,",
    "facility": "16"
  }
}

Running in development

Run the connector

./bin/debug.sh

Suspend waiting on the debugger to attach.

export SUSPEND='y'
./bin/debug.sh