JarCasting
JarCasting JarCasting

Kafka Provision Spring Boot Starter

Spring Boot Starter to enable decentralized Kafka provisioning

Лицензия

Лицензия
Категории

Категории

Spring Boot Контейнер Микросервисы
Группа

Группа

io.github.zghurskyi.kafka
Идентификатор

Идентификатор

kafka-provision-spring-boot-starter
Последняя версия

Последняя версия

0.0.1
Дата

Дата
Тип

Тип

jar
Описание

Описание

Kafka Provision Spring Boot Starter
Spring Boot Starter to enable decentralized Kafka provisioning
Ссылка на сайт

Ссылка на сайт

https://github.com/zghurskyi/kafka-provision-spring-boot-starter
Система контроля версий

Система контроля версий

http://github.com/zghurskyi/kafka-provision-spring-boot-starter/tree/master

Скачать kafka-provision-spring-boot-starter

Имя Файла Размер
kafka-provision-spring-boot-starter-0.0.1.pom
kafka-provision-spring-boot-starter-0.0.1.jar 19 KB
kafka-provision-spring-boot-starter-0.0.1-sources.jar 9 KB
kafka-provision-spring-boot-starter-0.0.1-javadoc.jar 59 KB
Обзор

Как подключить последнюю версию

<!-- https://jarcasting.com/artifacts/io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter/ -->
<dependency>
    <groupId>io.github.zghurskyi.kafka</groupId>
    <artifactId>kafka-provision-spring-boot-starter</artifactId>
    <version>0.0.1</version>
</dependency>
// https://jarcasting.com/artifacts/io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter/
implementation 'io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:0.0.1'
// https://jarcasting.com/artifacts/io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter/
implementation ("io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:0.0.1")
'io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:jar:0.0.1'
<dependency org="io.github.zghurskyi.kafka" name="kafka-provision-spring-boot-starter" rev="0.0.1">
  <artifact name="kafka-provision-spring-boot-starter" type="jar" />
</dependency>
@Grapes(
@Grab(group='io.github.zghurskyi.kafka', module='kafka-provision-spring-boot-starter', version='0.0.1')
)
libraryDependencies += "io.github.zghurskyi.kafka" % "kafka-provision-spring-boot-starter" % "0.0.1"
[io.github.zghurskyi.kafka/kafka-provision-spring-boot-starter "0.0.1"]

Зависимости

runtime (7)

Идентификатор библиотеки Тип Версия
com.fasterxml.jackson.core : jackson-core jar 2.9.7
com.fasterxml.jackson.core : jackson-annotations jar 2.9.7
com.fasterxml.jackson.core : jackson-databind jar 2.9.7
org.apache.commons : commons-lang3 jar 3.7
org.hibernate : hibernate-validator jar 6.0.10.Final
org.springframework.boot : spring-boot-starter jar 2.1.4.RELEASE
org.springframework.kafka : spring-kafka jar 2.1.10.RELEASE

test (3)

Идентификатор библиотеки Тип Версия
org.springframework.boot : spring-boot-starter-test jar 2.1.4.RELEASE
org.springframework.boot : spring-boot-test jar 2.1.4.RELEASE
org.springframework.kafka : spring-kafka-test jar 2.1.10.RELEASE

Модули Проекта

Данный проект не имеет модулей.

Kafka Provision Spring Boot Starter

Build Status Quality Gate Status Coverage Maven Central

This Spring Boot Starter enables distributed Kafka topics provisioning and centralized topic configs management.

Following featurs are supported:

Getting Started

Note: Introduction article with more details can be found here and usage example here.

To enable automatic topic creation and configuration follow the steps:

  1. Add dependency
implementation 'io.github.zghurskyi.kafka:kafka-provision-spring-boot-starter:0.0.1'
  1. Add @EnableTopicProvisioning
@SpringBootApplication
@EnableTopicProvisioning
public class Application {
 
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}
  1. Configure topics:
  • application.yml:
# disable topic creation by Spring Cloud Stream
spring.cloud.stream.kafka.binder:
  auto-create-topics: false

# configure topic provisioning
kafka.provision:
  brokers: ${kafka.brokers}
  topics:
  - name: ${topic.one}
    numPartitions: ${topic.one.partitions}
    replicationFactor: ${topic.one.replication.factor}
    configs:
      cleanup.policy: ${topic.one.cleanup.policy}
      retention.ms: ${topic.one.retention.ms}
  - name: ${topic.two}
    numPartitions: ${topic.two.partitions}
    replicationFactor: ${topic.two.replication.factor}
    configs:
      cleanup.policy: ${topic.two.cleanup.policy}
      retention.ms: ${topic.two.retention.ms}

Note: Any valid Kafka topic config can be used in configs section.

  • application.properties:
kafka.provision.brokers=${kafka.brokers}
kafka.provision.topics[0].name=${topic.one}
kafka.provision.topics[0].numPartitions=${topic.one.partitions}
kafka.provision.topics[0].replicationFactor=${topic.one.replication.factor}
kafka.provision.topics[0].configs.cleanup.policy=${topic.one.cleanup.policy}
kafka.provision.topics[0].configs.retention.ms=${topic.one.retention.ms}
kafka.provision.topics[1].name=${topic.two}
kafka.provision.topics[1].numPartitions=${topic.two.partitions}
kafka.provision.topics[1].replicationFactor=${topic.two.replication.factor}
kafka.provision.topics[1].configs.cleanup.policy=${topic.two.cleanup.policy}
kafka.provision.topics[1].configs.retention.ms=${topic.two.retention.ms}
  1. (Optional) Disable provisioning during tests in application-test.yml:
kafka.provision.enabled: false

Externalized configuration

The starter allows centralized management of topic configurations and decentralized topic provisioning.

This is possible, because provisioning is specified through application properties. So, simply manage Kafka topics state in your system through environment variables, that are collected together in centralized place:

  • Dockerfile:
ENV KAFKA_BROKERS kafka-broker-0:9092,kafka-broker-1:9092
ENV TOPIC_ONE topic_one
ENV TOPIC_ONE_PARTITIONS 32
ENV TOPIC_ONE_REPLICATION_FACTOR 3
ENV TOPIC_ONE_CLEANUP_POLICY delete
ENV TOPIC_ONE_RETENTION_MS 7776000000
ENV TOPIC_TWO topic_one
ENV TOPIC_TWO_PARTITIONS 32
ENV TOPIC_TWO_REPLICATION_FACTOR 3
ENV TOPIC_TWO_CLEANUP_POLICY delete
ENV TOPIC_TWO_RETENTION_MS 7776000000

  • Docker Swarm config

  • Kubernetes configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: 2019-04-14T18:38:05Z
  name: kafka-topics-config
  namespace: default
  selfLink: /api/v1/namespaces/default/configmaps/kafka-topics-config
data:
  KAFKA_BROKERS: kafka-broker-0:9092,kafka-broker-1:9092
  TOPIC_ONE: topic_one
  TOPIC_ONE_PARTITIONS: 32
  TOPIC_ONE_REPLICATION_FACTOR: 3
  TOPIC_ONE_CLEANUP_POLICY: delete
  TOPIC_ONE_RETENTION_MS: 7776000000
  TOPIC_TWO: topic_one
  TOPIC_TWO_PARTITIONS: 32
  TOPIC_TWO_REPLICATION_FACTOR: 3
  TOPIC_TWO_CLEANUP_POLICY: delete
  TOPIC_TWO_RETENTION_MS: 7776000000

Версии библиотеки

Версия
0.0.1