Kafka Offsets example

Category : Apache Kafka | Sub Category : Apache Kafka | By Prasad Bonam Last updated: 2023-08-05 09:42:18 Viewed : 250


Kafka Offsets example:

In Apache Kafka, offsets are used to represent the position of a consumer within a partition. Offsets are a critical aspect of Kafka`s design, as they allow consumers to keep track of their progress in consuming messages from a topic. Each partition maintains its own offset for each consumer group.

Lets go through an example to understand Kafka offsets:

  1. Setting Up Kafka: For this example, assume you have set up a Kafka cluster with a single broker: localhost:9092.

  2. Topic Creation: Create a Kafka topic named "my_topic" with three partitions and a replication factor of 1. This means there will be three partitions, and each partition will have a single replica.

    Using the command-line tool on Unix/Linux/Mac:

    bash
    bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my_topic --partitions 3 --replication-factor 1

    Using the command-line tool on Windows:

    batch
    binwindowskafka-topics.bat --bootstrap-server localhost:9092 --create --topic my_topic --partitions 3 --replication-factor 1
  3. Producing Messages: Start a producer to send messages to the "my_topic" topic.

    Using the command-line tool on Unix/Linux/Mac:

    bash
    bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic

    Using the command-line tool on Windows:

    batch
    binwindowskafka-console-producer.bat --broker-list localhost:9092 --topic my_topic

    Now, you can enter messages in the console, and Kafka will publish them to one of the three partitions in the "my_topic" topic.

  4. Consuming Messages: Start a consumer to read messages from the "my_topic" topic.

    Using the command-line tool on Unix/Linux/Mac:

    bash
    bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic --group my_group

    Using the command-line tool on Windows:

    batch
    binwindowskafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my_topic --group my_group

    The consumer will read messages from one of the partitions in the "my_topic" topic. Kafka will automatically assign a partition to the consumer within the consumer group and start reading from the offset where it left off.

  5. Offset Tracking: Kafka maintains the offset for each consumer group and partition. As the consumer reads messages, it updates its current offset. If the consumer is stopped and restarted, it will resume reading from the last committed offset for each partition.

  6. Manual Offset Control: In some cases, you might want to manually control the offset position. Kafka allows you to commit offsets manually, which can be useful for scenarios like at-least-once message processing. You can commit offsets after successfully processing a batch of messages to ensure that they are not reprocessed in case of failures.

    In code (Java) for manual offset control with the KafkaConsumer:

    java
    // Assuming you have a KafkaConsumer instance consumer.subscribe(Arrays.asList("my_topic")); try { while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord<String, String> record : records) { // Process the record System.out.printf("Offset: %d, Key: %s, Value: %s%n", record.offset(), record.key(), record.value()); } // Manually commit the offsets after successful processing consumer.commitSync(); } } finally { consumer.close(); }

Kafka offsets play a crucial role in ensuring reliable message processing and data integrity. By maintaining the offset position, Kafka allows consumers to consume messages efficiently and handle various scenarios, including restarts, failures, and load balancing within consumer groups.

Search
Sub-Categories
Related Articles

Leave a Comment: