To create a Kafka consumer and consume messages from a Kafka topic
Category : Apache Kafka
| Sub Category : Apache Kafka | By Prasad Bonam Last updated: 2023-08-05 14:47:45
Viewed : 56
To create a Kafka consumer and consume messages from a Kafka topic:
To create a Kafka consumer and consume messages from a Kafka topic, you can use the
kafka-console-consumer.sh script (or
kafka-console-consumer.bat on Windows) provided by Apache Kafka. This script allows you to consume and read messages from a specific Kafka topic using the command-line.
The basic command to start a Kafka consumer is as follows:
kafka-console-consumer.sh --bootstrap-server <BROKER_LIST> --topic <TOPIC_NAME> [--from-beginning]
--bootstrap-server: Specifies the Kafka broker(s) to connect to in the format
host:port. You need to provide at least one broker to connect the consumer.
--topic: Specifies the name of the Kafka topic from which you want to consume messages.
--from-beginning (optional): If provided, the consumer will start reading messages from the beginning of the topic. Otherwise, it will read only new messages published after the consumer started.
Lets say you have a Kafka cluster running with a broker on
localhost:9092, and you want to consume messages from a topic named "my_topic."
Using the command-line tool on Unix/Linux/Mac:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic --from-beginning
Using the command-line tool on Windows:
binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my_topic --from-beginning
In the above example, we are using the
kafka-console-consumer.bat) script to create a Kafka consumer. We specify the Kafka broker to connect to using the
--bootstrap-server option with the broker address
We also specify the
--topic option with the value "my_topic" to indicate that we want to consume messages from the "my_topic" Kafka topic.
--from-beginning flag is used here to instruct the consumer to start reading messages from the beginning of the topic. This is useful when you want to consume all messages from the topic, including those published before the consumer started running.
Once you run the consumer command, it will start reading messages from the specified Kafka topic and display them in the console. As new messages are published to the topic, the consumer will continue to consume and display them in real-time.
This is a test message
Welcome to Kafka messaging
The consumer will keep displaying messages as long as new messages are published to the "my_topic" topic. If you want to stop the consumer, you can use
Ctrl+C in the console.