JarCasting
JarCasting JarCasting

kafka-connect-bigtable

Kafka Sink Connect GCP Bigtable

Лицензия

Лицензия

MIT
Группа

Группа

com.sanjuthomas
Идентификатор

Идентификатор

kafka-connect-gcp-bigtable
Последняя версия

Последняя версия

1.0.7
Дата

Дата
Тип

Тип

jar
Описание

Описание

kafka-connect-bigtable
Kafka Sink Connect GCP Bigtable
Ссылка на сайт

Ссылка на сайт

http://sanjuthomas.com
Система контроля версий

Система контроля версий

https://github.com/sanjuthomas/kafka-connect-gcp-bigtable

Скачать kafka-connect-gcp-bigtable

Имя Файла Размер
kafka-connect-gcp-bigtable-1.0.7.pom
kafka-connect-gcp-bigtable-1.0.7.jar 41 KB
kafka-connect-gcp-bigtable-1.0.7-sources.jar 25 KB
kafka-connect-gcp-bigtable-1.0.7-shaded.jar 31 MB
kafka-connect-gcp-bigtable-1.0.7-javadoc.jar 185 KB
Обзор

Как подключить последнюю версию

<!-- https://jarcasting.com/artifacts/com.sanjuthomas/kafka-connect-gcp-bigtable/ -->
<dependency>
    <groupId>com.sanjuthomas</groupId>
    <artifactId>kafka-connect-gcp-bigtable</artifactId>
    <version>1.0.7</version>
</dependency>
// https://jarcasting.com/artifacts/com.sanjuthomas/kafka-connect-gcp-bigtable/
implementation 'com.sanjuthomas:kafka-connect-gcp-bigtable:1.0.7'
// https://jarcasting.com/artifacts/com.sanjuthomas/kafka-connect-gcp-bigtable/
implementation ("com.sanjuthomas:kafka-connect-gcp-bigtable:1.0.7")
'com.sanjuthomas:kafka-connect-gcp-bigtable:jar:1.0.7'
<dependency org="com.sanjuthomas" name="kafka-connect-gcp-bigtable" rev="1.0.7">
  <artifact name="kafka-connect-gcp-bigtable" type="jar" />
</dependency>
@Grapes(
@Grab(group='com.sanjuthomas', module='kafka-connect-gcp-bigtable', version='1.0.7')
)
libraryDependencies += "com.sanjuthomas" % "kafka-connect-gcp-bigtable" % "1.0.7"
[com.sanjuthomas/kafka-connect-gcp-bigtable "1.0.7"]

Зависимости

compile (5)

Идентификатор библиотеки Тип Версия
com.fasterxml.jackson.core : jackson-databind jar 2.9.10
com.fasterxml.jackson.dataformat : jackson-dataformat-yaml jar 2.9.10
com.google.cloud.bigtable : bigtable-client-core jar 1.12.1
io.grpc : grpc-netty-shaded jar 1.23.0
org.apache.kafka : connect-api jar 2.3.0

provided (1)

Идентификатор библиотеки Тип Версия
org.slf4j : slf4j-api jar 1.7.21

test (6)

Идентификатор библиотеки Тип Версия
log4j : log4j jar 1.2.17
org.mockito : mockito-junit-jupiter jar 2.23.0
org.junit.jupiter : junit-jupiter-api jar 5.3.2
org.junit.jupiter : junit-jupiter-engine jar 5.3.2
org.junit.jupiter : junit-jupiter-params jar 5.3.2
org.apache.kafka : connect-json jar 2.3.0

Модули Проекта

Данный проект не имеет модулей.

Build Status codecov Codacy Badge Maintainability codebeat badge Maven Central

Kafka Sink Connect Google Cloud (GCP) Bigtable

Apache Kafka Sink only Connect can be used to stream messages from Apache Kafka to Google Cloud Platform (GCP) wide column store Bigtable.

What is Apache Kafka

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. For more details, please refer to Apache Kafka home page.

What is Google Cloud Bigtable

Bigtable is a compressed, high performance, proprietary data storage system built on Google File System, Chubby Lock Service, SSTable and a few other Google technologies. On May 6, 2015, a public version of Bigtable was made available as a service in the Google Cloud Platform. For more details, please refer to GCP Bigtable home page.

High Level Architecture

This project leverage bigtable-client-core library (NO HBase) to stream data to GCP Bigtable. bigtable-client-core internally use the gRPC framework to talk to GCP Bigtable.

Kafka Connect GCP Bigtable

Prerequisites

Apache ZooKeeper and Apache Kafka installed and running in your machine. Please refer to respective sites to download and start ZooKeeper and Kafka. You would also need Java version 8 or above.

Tested Software Versions

Software Version Note
Java 1.8.0_161 You may use java 8 or above. Tested using Java 8.
Kafka >= 2.11-1.1.1 Please refer. Tested using kafka_2.11-1.1.1, may work with older versions.
Zookeeper >= 3.4.13 Please refer. Tested using zookeeper-3.4.13.
bigtable-client-core 1.12.1 Please refer.
Kafka connect-api 2.3.0 Please refer.
grpc-netty-shaded 1.23.0 Please refer.

Configurations

Please refer to project Wiki

Constraints

The current configuration system supports streaming messages from a given topic to a given table. You can subscribe any number of topics, but a topic can be pointed to one and only table. Say for example, if you subscribed from topic named demo-topic, you should have yml file named demo-topic.yml. That yml file contains all the configuration requires to transform and write data into Bigtable.

As of today, we have transformer support for JSON Messages. I'm planning to add the Avro Messages transformer in the next version.

How to build the artifact

Please refer to project Wiki

How to deploy the connector

Please refer to project Wiki

How to start connector in stand-alone mode

Please refer to project Wiki

Questions

Either create an issues in this project or send it to bt@sanju.org. Thanks!

License

FOSSA Status

Версии библиотеки

Версия
1.0.7
1.0.6
1.0.5
1.0.3