[Kafka] 10장. Kafka CI Tools 실습해보기

Kafka 기초 다지기

Kafka CI Tools 실습해보기


  1. Kafka CI Tools

  2. Kafka ElasticSearch Connector

1. Kafka CI Tools

경로 (macOS)


kafka-topics Create, alter, list, and describe topics

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

kafka-topics --list --zookeeper localhost:2181


kafka-console-producer Read data from standard output and write it to a Kafka topic

kafka-console-producer --broker-list localhost:9092 --topic test

> Hello Kafka

kafka-console-consumer Read data from a Kafka topic and write it to standard output

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

Hello Kafka

kafka-consumer-groups Check the number of messages read and written, as well as the lag for each consumer in a specific consumer group.

kafka-consumer-groups  --bootstrap-server localhost:9092 --list

kafka-consumer-groups  --bootstrap-server localhost:9092 --describe --group logstash

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
mall_change     0          2               2               0               logstash-0-c1d2cf12-8efe-47cd-8dfb-8d51b4f54e53 /      logstash-0

2. Kafka ElasticSearch Connector

커넥터 종류 확인

./confluent list connectors

커넥터 실행

./confluent load elasticsearch-sink

리스트 확인 List the connector plugins available on this worker

curl localhost:8083/connector-plugins | jq

실행 중인 리스트 확인 Listing active connectors on a worker

curl localhost:8083/connectors
./confluent status connectors

상태 확인

./confluent status elasticsearch-sink