To create a Kafka consumer and consume messages from a Kafka topic

Category : Apache Kafka | Sub Category : Apache Kafka | By Prasad Bonam Last updated: 2023-08-05 09:17:45 Viewed : 320

To create a Kafka consumer and consume messages from a Kafka topic:

To create a Kafka consumer and consume messages from a Kafka topic, you can use the kafka-console-consumer.sh script (or kafka-console-consumer.bat on Windows) provided by Apache Kafka. This script allows you to consume and read messages from a specific Kafka topic using the command-line.

The basic command to start a Kafka consumer is as follows:

cssCopy code
kafka-console-consumer.sh --bootstrap-server <BROKER_LIST> --topic <TOPIC_NAME> [--from-beginning]

• --bootstrap-server: Specifies the Kafka broker(s) to connect to in the format host:port. You need to provide at least one broker to connect the consumer.

• --topic: Specifies the name of the Kafka topic from which you want to consume messages.

• --from-beginning (optional): If provided, the consumer will start reading messages from the beginning of the topic. Otherwise, it will read only new messages published after the consumer started.

Example: Lets say you have a Kafka cluster running with a broker on localhost:9092, and you want to consume messages from a topic named "my_topic."

Using the command-line tool on Unix/Linux/Mac:

bashCopy code
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic --from-beginning

Using the command-line tool on Windows:

batchCopy code
binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my_topic --from-beginning

Explanation: In the above example, we are using the kafka-console-consumer.sh (or kafka-console-consumer.bat) script to create a Kafka consumer. We specify the Kafka broker to connect to using the --bootstrap-server option with the broker address localhost:9092.

We also specify the --topic option with the value "my_topic" to indicate that we want to consume messages from the "my_topic" Kafka topic.

The optional --from-beginning flag is used here to instruct the consumer to start reading messages from the beginning of the topic. This is useful when you want to consume all messages from the topic, including those published before the consumer started running.

Once you run the consumer command, it will start reading messages from the specified Kafka topic and display them in the console. As new messages are published to the topic, the consumer will continue to consume and display them in real-time.

Example output:

cssCopy code
Hello Kafka
This is a test message
Welcome to Kafka messaging

The consumer will keep displaying messages as long as new messages are published to the "my_topic" topic. If you want to stop the consumer, you can use Ctrl+C in the console.