Example of cluster apache ZooKeeper for maintaining cluster metadata

Category : Apache Kafka | Sub Category : Apache Kafka | By Prasad Bonam Last updated: 2023-08-03 01:16:55 Viewed : 338

Example of cluster apache ZooKeeper for maintaining cluster metadata 

Apache ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and group services in a distributed environment. It is commonly used in Apache Kafka to manage cluster metadata and perform leader election. Lets walk through a simple example of setting up a ZooKeeper ensemble for maintaining cluster metadata.

Prerequisites: Ensure you have Java installed, as both ZooKeeper and Kafka are Java-based applications.

Step 1: Download and Extract ZooKeeper: Download the latest stable release of Apache ZooKeeper from the official website (https://zookeeper.apache.org/) and extract it to a directory of your choice.

Step 2: Configuration: Create a configuration file for ZooKeeper, typically named zoo.cfg. This file will define various properties for the ZooKeeper ensemble. Below is a simple example:

propertiesCopy code
# zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/path/to/zookeeper-data
clientPort=2181
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890

In this example, we have defined three ZooKeeper servers (nodes) with unique IDs (1, 2, and 3) and specified their respective ports for leader election (2888) and quorum communication (3888).

Step 3: Data Directory: Create the data directory specified in the dataDir property in the zoo.cfg file. This is where ZooKeeper will store its data and transaction logs.

Step 4: Start ZooKeeper Ensemble: Now, start the ZooKeeper ensemble by running the ZooKeeper server on each of the nodes. Open three terminal windows (one for each node) and execute the following commands:

For Node 1:

bashCopy code
$ cd /path/to/zookeeper
$ bin/zkServer.sh start zoo.cfg

For Node 2:

bashCopy code
$ cd /path/to/zookeeper
$ bin/zkServer.sh start zoo.cfg

For Node 3:

bashCopy code
$ cd /path/to/zookeeper
$ bin/zkServer.sh start zoo.cfg

Step 5: Verify Ensemble Status: You can check the status of the ZooKeeper ensemble by running the following command on any of the nodes:

bashCopy code
$ cd /path/to/zookeeper
$ bin/zkServer.sh status

If everything is set up correctly, you should see output similar to the following:

makefileCopy code
Mode: leader

This indicates that the ZooKeeper node you executed the command on has been elected as the leader.

Step 6: Connect Kafka to ZooKeeper: With the ZooKeeper ensemble up and running, you can now configure Apache Kafka to use it. In the Kafka configuration file (server.properties), set the zookeeper.connect property to point to the ZooKeeper ensemble:

propertiesCopy code
# server.properties
...
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183 
....

Now, when you start your Kafka brokers, they will connect to the ZooKeeper ensemble for coordination and metadata management.

This example demonstrates a simple ZooKeeper ensemble with three nodes, but in a production environment, you may have more nodes to achieve better fault tolerance and higher availability. Apache Kafka uses ZooKeeper for managing cluster metadata, leader election, and coordination, making it a critical component in Kafkas distributed architecture.