Setup Kafka

Download Kafka from https://kafka.apache.org/downloads

At the time of this article, I chose Kafka 3.6.1 with Scala 2.13 . Untar the file by

tar -xzvf kafka_2.13-3.6.1.tgz

Go into the folder after you untar it.

Setup Kafka cluster ID

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

Format the Kafka storage

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

Start the Kafka service

# run in terminal
bin/kafka-server-start.sh config/kraft/server.properties
# OR
# run in background
bin/kafka-server-start.sh config/kraft/server.properties &

Stop the Kafka service

If you ran it in the terminal, you can just go back to the same terminal and press Ctrl+C else run

bin/kafka-server-stop.sh

Create a topic with 3 partitions

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic topic_name --partitions 3

List all topics

bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Describe a topic

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic topic_name

Altering partition count

bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic topic_name --partitions 5

Subscribe to a topic

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_name --group consumer_group_name

View consumer group state

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer_group_name --state

View consumer group members

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer_group_name --members

View consumer group details

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer_group_name

Publishing message to topic

bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic topic_name

Since the consumer was still listening. Whenever I publish a message it will be shown on the consumer window.

Deleting a topic

bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic topic_name

Summary of all the above commands

# create a topic
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic topic_name --partitions 3

# list all topics
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# describe a topic
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic topic_name

# alter partitions count
bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic topic_name --partitions 5

# subscribe to topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_name --group consumer_group_name

# view consumer group state
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer_group_name --state

# view consumer group members
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer_group_name --members

# describe consumer group details
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer_group_name

# publishing message to topic
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic topic_name

# deleting a topic
bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete -topic topic_name

# producer adding key with separator :
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic topic_name --property parse.key=true --property key.separator=:

# consumer printing key with separator :
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_name --group consumer_group_name --property print.key=true --property key.separator=:

< Back to all the stories I had written