JarCasting
JarCasting JarCasting

kafka-connect-bigtable

Kafka Sink Connect GCP Bigtable

License

License

MIT
GroupId

GroupId

com.sanjuthomas
ArtifactId

ArtifactId

kafka-connect-gcp-bigtable
Last Version

Last Version

1.0.7
Release Date

Release Date
Type

Type

jar
Description

Description

kafka-connect-bigtable
Kafka Sink Connect GCP Bigtable
Project URL

Project URL

http://sanjuthomas.com
Source Code Management

Source Code Management

https://github.com/sanjuthomas/kafka-connect-gcp-bigtable

Download kafka-connect-gcp-bigtable

Filename Size
kafka-connect-gcp-bigtable-1.0.7.pom
kafka-connect-gcp-bigtable-1.0.7.jar 41 KB
kafka-connect-gcp-bigtable-1.0.7-sources.jar 25 KB
kafka-connect-gcp-bigtable-1.0.7-shaded.jar 31 MB
kafka-connect-gcp-bigtable-1.0.7-javadoc.jar 185 KB
Browse

How to add to project

<!-- https://jarcasting.com/artifacts/com.sanjuthomas/kafka-connect-gcp-bigtable/ -->
<dependency>
    <groupId>com.sanjuthomas</groupId>
    <artifactId>kafka-connect-gcp-bigtable</artifactId>
    <version>1.0.7</version>
</dependency>
// https://jarcasting.com/artifacts/com.sanjuthomas/kafka-connect-gcp-bigtable/
implementation 'com.sanjuthomas:kafka-connect-gcp-bigtable:1.0.7'
// https://jarcasting.com/artifacts/com.sanjuthomas/kafka-connect-gcp-bigtable/
implementation ("com.sanjuthomas:kafka-connect-gcp-bigtable:1.0.7")
'com.sanjuthomas:kafka-connect-gcp-bigtable:jar:1.0.7'
<dependency org="com.sanjuthomas" name="kafka-connect-gcp-bigtable" rev="1.0.7">
  <artifact name="kafka-connect-gcp-bigtable" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.sanjuthomas', module='kafka-connect-gcp-bigtable', version='1.0.7')
)
libraryDependencies += "com.sanjuthomas" % "kafka-connect-gcp-bigtable" % "1.0.7"
[com.sanjuthomas/kafka-connect-gcp-bigtable "1.0.7"]

Dependencies

compile (5)

Group / Artifact Type Version
com.fasterxml.jackson.core : jackson-databind jar 2.9.10
com.fasterxml.jackson.dataformat : jackson-dataformat-yaml jar 2.9.10
com.google.cloud.bigtable : bigtable-client-core jar 1.12.1
io.grpc : grpc-netty-shaded jar 1.23.0
org.apache.kafka : connect-api jar 2.3.0

provided (1)

Group / Artifact Type Version
org.slf4j : slf4j-api jar 1.7.21

test (6)

Group / Artifact Type Version
log4j : log4j jar 1.2.17
org.mockito : mockito-junit-jupiter jar 2.23.0
org.junit.jupiter : junit-jupiter-api jar 5.3.2
org.junit.jupiter : junit-jupiter-engine jar 5.3.2
org.junit.jupiter : junit-jupiter-params jar 5.3.2
org.apache.kafka : connect-json jar 2.3.0

Project Modules

There are no modules declared in this project.

Build Status codecov Codacy Badge Maintainability codebeat badge Maven Central

Kafka Sink Connect Google Cloud (GCP) Bigtable

Apache Kafka Sink only Connect can be used to stream messages from Apache Kafka to Google Cloud Platform (GCP) wide column store Bigtable.

What is Apache Kafka

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. For more details, please refer to Apache Kafka home page.

What is Google Cloud Bigtable

Bigtable is a compressed, high performance, proprietary data storage system built on Google File System, Chubby Lock Service, SSTable and a few other Google technologies. On May 6, 2015, a public version of Bigtable was made available as a service in the Google Cloud Platform. For more details, please refer to GCP Bigtable home page.

High Level Architecture

This project leverage bigtable-client-core library (NO HBase) to stream data to GCP Bigtable. bigtable-client-core internally use the gRPC framework to talk to GCP Bigtable.

Kafka Connect GCP Bigtable

Prerequisites

Apache ZooKeeper and Apache Kafka installed and running in your machine. Please refer to respective sites to download and start ZooKeeper and Kafka. You would also need Java version 8 or above.

Tested Software Versions

Software Version Note
Java 1.8.0_161 You may use java 8 or above. Tested using Java 8.
Kafka >= 2.11-1.1.1 Please refer. Tested using kafka_2.11-1.1.1, may work with older versions.
Zookeeper >= 3.4.13 Please refer. Tested using zookeeper-3.4.13.
bigtable-client-core 1.12.1 Please refer.
Kafka connect-api 2.3.0 Please refer.
grpc-netty-shaded 1.23.0 Please refer.

Configurations

Please refer to project Wiki

Constraints

The current configuration system supports streaming messages from a given topic to a given table. You can subscribe any number of topics, but a topic can be pointed to one and only table. Say for example, if you subscribed from topic named demo-topic, you should have yml file named demo-topic.yml. That yml file contains all the configuration requires to transform and write data into Bigtable.

As of today, we have transformer support for JSON Messages. I'm planning to add the Avro Messages transformer in the next version.

How to build the artifact

Please refer to project Wiki

How to deploy the connector

Please refer to project Wiki

How to start connector in stand-alone mode

Please refer to project Wiki

Questions

Either create an issues in this project or send it to bt@sanju.org. Thanks!

License

FOSSA Status

Versions

Version
1.0.7
1.0.6
1.0.5
1.0.3