ETS Kafka Topic Config Manager

Kafka Topic Config Manager helps you manage the topic configurations of your Kafka clusters. This is done by providing your configurations in a XML format, the XSD can be found here.

XSD version compatibility

Supplying Kafka configurations

You have to supply the XML configurations to the manager. This enabled the manager to check differences between the configuration itself and the Kafka cluster and optionally updates the Kafka cluster configurations.

This is best done by creating a Java / Scala project that allows you to provide the configurations. We will go into this in more detail in this README.

There are four things you have to supply (3 and 4 are optional):

1. The manager needs to know how to retrieve the XML configuration files (e.g. from a folder, the classpath, an url,...). This is done by providing a TopicConfigurationServiceProvider.
2. The manager needs to know the Kafka server addresses. This is done by providing a HOCON style config. The keys are environements[].name and environments[].bootstrapServers. The configuration will be explained later on.
3. The manager calls zero to n TopicXmlConfigurationValidationServiceProvider to validate XML configurations.
4. The manager calls zero to n TopicUpdateListenerServiceProvider if topic configurations of the cluster are updated.

Create a project that depends on ets-kafka-topic-config-manager-api

The project must provide a de.kaufhof.ets.kafkatopicconfigmanager.TopicConfigurationServiceProvider via the Java ServiceLoader interface.

The TopicConfigurationServiceProvider is located in the de.kaufhof.ets:ets-kafka-topic-config-manager-api JAR. Use the provided scope, because the api JAR will be provided by the manager JAR later.

Provide topic configurations

You can use the just created project or another one or just package the topics however you like. They must be provided to the manager later as well.

The example uses the same project for the TopicConfigurationServiceProvider and the configurations.

Provide the manager config

You can provide the manager config via an application.conf file in HOCON format by specifying, e.g.

environments = [
  {
    name: env1
    bootstrapServers: kafka1-env1:9092,kafka2-env1:9092
  }
  {
    name: env2
    bootstrapServers: kafka1-env2:9092,kafka2-env2:9092
  }
]

Alternatively you can also specify the configuration via Java system properties like -Denvironments.0.name=env1 -Denvironments.0.bootstrapServers=kafka1-env1:9092,kafka2-env1:9092.

Provide the TopicConfigurationsServiceProvider and the configs to the manager

The manager JARs, configuration provider and topic configuration files must be visible to the classpath.

You can then execute the de.kaufhof.ets.kafkatopicconfigmanager.Main class.

java -cp "manager/*:providers/*:configs/*" de.kaufhof.ets.kafkatopicconfigmanager.Main

Example

For an example see the ets-kafka-topic-config-manager-example module. It packages all JAR files to a single tar.gz. The configurations are packaged in the same JAR as the TopicConfigurationsServiceProvider.

You have to provide the JARs packaged in the tar.gz file of the manager and the ones from your configuration JAR(s) and execute the de.kaufhof.ets.kafkatopicconfigmanager.Main Main class.

Example

$ tar -xf "manager.tar.gz" -C "manager"
$ tree manager
manager
├── config-1.3.2.jar
├── kafka-clients-1.1.1.jar
├── ets-kafka-topic-config-manager-api-1.0-SNAPSHOT.jar
├── ets-kafka-topic-config-manager-core-1.0-SNAPSHOT.jar
├── logback-classic-1.2.3.jar
├── logback-core-1.2.3.jar
├── lz4-java-1.4.1.jar
├── scala-library-2.12.4.jar
├── slf4j-api-1.7.25.jar
└── snappy-java-1.1.7.1.jar
$ tar -xf "configs.tar.gz" -C "configs"
$ tree configs
  configs
  ├── ets-kafka-topic-config-manager-example-1.0-SNAPSHOT.jar
  ├── spring-core-5.0.1.RELEASE.jar
  └── spring-jcl-5.0.1.RELEASE.jar

$ java -cp "manager/*:configs/*" de.kaufhof.ets.kafkatopicconfigmanager.Main

Attention: The VERSION must be the same across the dependency you use in your configurations JAR (de.kaufhof.ets:ets-kafka-topic-config-manager-api) and the ets-kafka-topic-config-manager-core JAR / Docker image you use to manage your topic configurations.

FolderTopicConfigurationProvider (file-folder dependency)

There is a predefined TopicConfigurationProvider that reads files from a folder.

You can use the file-folder dependency in your project or use the appropriate Docker Image galeriakaufhof/ets-kafka-topic-config-manager:file-folder.

The FolderTopicConfigurationServiceProvider expects a Java Property named file.folder.path with the configuration folder path and optionally accepts file.folder.recursive true or false (default) that enabled recursive lookup.

Example:

docker run --rm -e JAVA_OPTS="-Dfile.folder.path=/tmp/topic-configs ..." -v /path/to/your/topics/folder:/tmp/topic-configs galeriakaufhof/ets-kafka-topic-config-manager:file-folder

Create topics in your docker-compose setup

You may want to use the Kafka Topic Config Manager to create topics on production system and your local Docker or docker-compose setup.

To accomplish this, use the file-folder Docker image, e.g.

version: '2'

services:

  kafka:
    ...
    
  zookeeper:
    ...
    
  kafkatopicconfigmanager:
    image: galeriakaufhof/ets-kafka-topic-config-manager:file-folder
    depends_on:
      - kafka
    environment:
      JAVA_OPTS: -Denvironments.0.name=test -Denvironments.0.bootstrapServers=172.16.241.10:9092 -Dfile.folder.path=/tmp/topics -DautoApply=true
    volumes:
      - /path/to/topics/folder/:/tmp/topics/
    ...

...

.

Do not forget to use test as the environment in your XML files then.

Example: https://github.com/Galeria-Kaufhof/ets-kafka-clients/blob/master/.travis/docker-compose.yml

Auto apply

By default, config differences are only printed without changing any configurations and topics of the Kafka cluster.

You can enable auto apply by supplying -DautoApply=true as Java system property.

Run docker image

The Docker container utilizes all JARs in /opt/topic-configs and /opt/config-providers by adding them to the classpath. This allows you to supply topic configs by mounting them to /opt/topic-configs and config providers to /opt/config-providers.

Example:

docker run --rm -v /path/to/your/topics/jar:/opt/topic-configs/topics.jar galeriakaufhof/ets-kafka-topic-config-manager

You can also provide Java Opts via the JAVA_OPTS env var via -e JAVA_OPTS=....

This allows you to enable auto apply via

docker run --rm -v /path/to/your/topics/jar:/opt/topic-configs/topics.jar -e JAVA_OPTS="-DautoApply=true" galeriakaufhof/ets-kafka-topic-config-manager

License

The Kafka Topic Config Manager is released under the MIT License.