Apache Kafka Without ZooKeeper

Previous Article related to Kafka

• Introduction to Apache Kafka with .NET
• Apache Kafka with Dapr Bindings in .NET
• Raft Consensus Algorithm

Introduction

Apache Kafka is an open-source Distributed Streaming platform that collects, processes, stores, and manages real-time data that are streaming continuously into Kafka servers. Zookeeper is an open-source coordination service for managing distributed applications. In the realm of modern data processing and real-time event streaming, Apache Kafka stands out as a robust and adaptable tool. With its capability for constructing scalable, fault-tolerant, and high-throughput data pipelines, Kafka has become the preferred solution for organizations grappling with extensive data volumes.

If you're seeking to establish a Kafka cluster to cater to your data streaming requirements, you've come to the right resource. In this article, we will walk you through deploying a Kafka cluster. We'll delve into the utilization of Docker Compose without using ZooKeeper to initiate and administer your Kafka cluster, each presenting distinct advantages and suitable applications.

In this article, you will learn about Kafka, Zookeeper, and running Apache Kafka without Zookeeper. You will also learn how to install Apache Kafka without Zookeeper.

What is a Zookeeper?

Zookeeper functions as a service designed to manage configurations, naming, and distributed synchronization within distributed systems. Through nodes, distributed processes coordinate with each other. Its performance capabilities make it suitable for large distributed systems, while its reliability prevents it from being a single point of failure. Zookeeper's strict ordering enables the implementation of sophisticated synchronization primitives at the client level.

Unlike standard file systems, each node in Zookeeper can have associated data. Primarily intended for coordination data storage, Zookeeper handles information such as status, configuration, and location data, often in kilobytes or bytes.

Zookeeper offers various features tailored for:

• distributed systems
• including distributed configuration management
• self-election/consensus building
• coordination and locks
• a key-value store.

The service itself is replicated over a set of machines that comprise it. These machines maintain an in-memory image of the data tree along with transaction logs and snapshots in a persistent store. Because the data is kept in memory, Zookeeper is able to get very high throughput and low latency numbers. The downside to an in-memory database is that the size of the database that Zookeeper can manage is limited by memory.

As long as a majority of the servers are available, the Zookeeper service will be available.

Role of Zookeeper in Kafka

Here are the things Zookeeper does for Kafka; actually, for any distributed system.

• Brokers registration, with a heartbeat mechanism to keep the broker list updated
• Maintaining metadata of topics
  • Topic configuration (partitions, replication factor, additional configs, etc.)
  • The list of in-sync replicas for partitions
  • Performs leader election for all partitions
• Store metadata of Access Control Lists (ACLs), if security is enabled
  • Topics
  • Consumer Groups
  • Users
• Store the Kafka cluster-id (randomly generated while the Broker registers the first time)

ZookeeperLimitations and Challenges

Zookeeper seems an inseparable part of running a distributed system like Kafka but it has some limitations which are preventing Kafka from working to its full potential.

Let’s discuss some of those challenges here.

Challenges and drawbacks of Kafka

Having Zookeeper as an external system alongside Kafka introduces several challenges and drawbacks.

1. Configuration and Management Overhead: Administrators need to handle separate configuration syntaxes and management tools for both Kafka and Zookeeper, increasing complexity and the likelihood of errors.
2. Increased Complexity: Managing two distributed systems with different configurations raises the potential for errors due to the added complexity.
3. Duplication of Work: Enabling security features, for instance, requires applying configurations to both Kafka and Zookeeper, resulting in duplicated efforts.
4. Resource Inefficiency: Running an external metadata store like Zookeeper consumes additional resources, leading to inefficiencies.
5. Scalability Constraints: Zookeeper's involvement limits Kafka's scalability. During startup, the Kafka controller must load cluster state from Zookeeper synchronously, leading to potential scalability limitations.
6. Impact on Cluster Availability: Loading metadata synchronously during startup makes the cluster temporarily unavailable, impacting availability. Additionally, as metadata grows over time, the process becomes increasingly inefficient, constraining the number of partitions the cluster can handle.
7. Metadata Distribution Overhead: The Zookeeper-based Kafka controller is responsible for distributing metadata to all brokers, which can lead to additional overhead and potential performance issues.

Why does Kafka depend on Zookeeper?

Kafka uses Zookeeper as a Service Discovery tool, meaning each Kafka node that wants to find other nodes needs to discover their addresses and status through service discovery. Kafka also uses Zookeeper for other purposes such as Leader Election, Broker Discovery, Configuration Management, and Health Monitoring.

In the ZooKeeper setup, there exists a separate ZooKeeper cluster utilized by the Kafka cluster for member discovery, storage of essential configurations, and coordination among brokers and their associated data. Upon startup, each broker initiates a controller process. The first broker to register in ZooKeeper assumes the role of an active controller, thereby gaining eligibility to oversee the cluster and disseminate updates to all brokers. Concurrently, other controllers continuously monitor ZooKeeper for any indication of the active controller's disappearance. In the event of such a scenario, the first controller to detect the absence endeavors to register and ascend to the role of active controller for the cluster.

What is KRaft?

The Kafka cluster setup without ZooKeeper has a special type of server - controller. The controller servers form a cluster quorum. The cluster uses the KRaft algorithm.

Kraft is a new algorithm developed by Kafka based on the Raft_Consensus_algorithm, which is suitable for achieving consensus between trusted parties. Kafka no longer needs Zookeeper because it can achieve its goals using the Raft consensus algorithm.

Kafka employs the Raft consensus algorithm for leader election within its Kafka Raft Metadata mode, which obviates the necessity for ZooKeeper in managing cluster metadata. Raft is a consensus protocol designed to ensure fault-tolerant replication of state machines across a distributed system.

In Kafka's Raft implementation, each broker serves as a node in the Raft cluster, with one designated leader responsible for managing the cluster metadata's state. Upon startup, a broker sends a join request to the existing leader to join the Raft cluster. If no leader exists, the broker initiates an election process and becomes a candidate. The election process commences with a randomized timer known as the election timeout, initiated upon a candidate's activation. Should a candidate fail to garner a majority of votes from other brokers within the election timeout, a new election ensues, featuring an extended timeout. This iterative process persists until a candidate secures a majority of votes, thereby assuming leadership. Once elected, the leader replicates metadata to other brokers. In case of leader failure or unresponsiveness detected by a broker, a new election is triggered to elect a successor leader. This new leader assumes responsibility for managing the cluster metadata.

Raft ensures strong consistency guarantees for cluster metadata, ensuring sole active leadership and consistent metadata views across all brokers. This setup ensures Kafka cluster reliability and fault tolerance, eliminating the need for a separate ZooKeeper cluster for metadata management.

The KRaft controllers collectively form a Kraft quorum, which stores all the metadata information regarding Kafka clusters. With this method, you eradicate the dependency of Zookeeper within the Kafka environment architecture. Besides, you can achieve various benefits like eliminating system complexities and data redundancy while running Kafka without Zookeeper. As Kafka plans to discontinue Zookeeper as a centralized configuration service, you will have a simplified Kafka architecture without any third-party service dependencies.

Benefits of Kafka’s New Quorum Controller

1. Kraft enables right-sized clusters, meaning clusters that are sized with the appropriate number of brokers and compute to satisfy a use case’s throughput and latency requirements, with the potential to scale up to millions of partitions.
2. Improves stability, simplifies the software, and makes it easier to monitor, administer, and support Kafka.
3. Allows Kafka to have a single security model for the whole system.
4. Unified management model for configuration, networking setup, and communication protocols.
5. Provides a lightweight, single-process way to start Kafka.
6. Makes controller failover near-instantaneous.

Raft Algorithm for Kafka Leader election

Since version 2.8.0, Kafka offers the option to operate without Apache ZooKeeper by introducing Kafka Raft Metadata mode. In this mode, brokers communicate and synchronize through the Raft consensus protocol. This approach enables them to elect a leader and replicate metadata across the cluster independently, dispensing with the necessity for an external system like ZooKeeper. By doing so, Kafka Raft Metadata mode presents a more straightforward and streamlined deployment for Kafka clusters, as it removes the requirement for a separate ZooKeeper cluster and simplifies configuration.

Apache Kafka has officially deprecatedZooKeeperin version 3.5.

Kafka version	State
2.8	Kraft early access
3.3	Kraft production-ready
3.4	Migration scripts early access
3.5	ZooKeeper deprecated
4.0	ZooKeeper not supported

To run Kafka in Raft Metadata mode, you need to set a few configuration parameters in the broker properties file.

1. Set the broker. id parameter to a unique numeric ID for each broker in the cluster.
2. Set the metadata. replication. factor parameter to the number of brokers that should replicate metadata. This should be set to at least three for production environments.
3. Set the listener parameter to the network interface and port that the broker should listen on.

In KRaftmode each Kafka server can be configured as a controller, a broker, or both using the process. rolesproperty. This property can have the following values.

• If process. roles are set to broker, and the server acts as a broker.
• If process. roles are set to the controller, and the server acts as a controller.
• If the process. roles are set to broker, and controller, the server acts as both a broker and a controller.
• If process. roles are not set at all, it is assumed to be in ZooKeeper mode.

The following image provides a simple illustration of the difference between running with ZooKeeper and KRaft for managing metadata.

How it works?

The quorum controllers utilize the new KRaft protocol to ensure accurate replication of metadata across the quorum. Each quorum controller maintains its state using an event-sourced storage model, guaranteeing the precise recreation of internal state machines. To prevent the indefinite growth of the event log (also known as the metadata topic), periodic snapshots are taken to abridge the log. Other controllers within the quorum keep pace with the active controller by processing the events it generates and stores in its log. Therefore, in the event of a node pausing due to partitioning or other factors, it can swiftly catch up on missed events by accessing the log upon rejoining. This significantly reduces the window of unavailability, thereby enhancing the worst-case recovery time of the system.

In contrast to the ZooKeeper-based controller, the event-driven nature of the KRaft protocol eliminates the necessity for the quorum controller to load state from ZooKeeper before assuming activity. Upon a leadership change, the new active controller possesses all committed metadata records in memory. Additionally, the same event-driven mechanism employed in the KRaft protocol tracks metadata across the cluster, replacing the RPCs used previously. This transition to an event-driven approach, coupled with utilizing an actual log for communication, enhances the efficiency and reliability of the system.

How does it overcome the challenges with Zookeeper?

Migration Strategy: Zookeeper to KRaft

Starting with KRaft is easy and one can set up the Kafka cluster in no time, but what about when the existing cluster runs with Zookeeper? How will the migration happen? What would be the operational changes required?

Answers to all these questions are available in KIP-866. This Kafka Improvement Plan lists the approach to migrate from Zookeeper quorum to Raft quorum without any impact on partition availability and with minimal impacts on operators and client applications. The implementation will be available as early access starting from Apache Kafka version 3.4.0.

Steps to migrate from Zookeeper to Raft quorum

Migration itself is a vast topic and still under development but the following are the steps to migrate from Zookeeper to Raft quorum.

1. Preparing the cluster: Upgrade the cluster to the version where migration-related implementation is available. This release will have additional configurations required by Zookeeper and Kraft Controllers in order to communicate with each other during the migration phase.
2. Controller migration: The migration only supports dedicated KRaft controllers as the target deployment. A new set of nodes will be provisioned to host the controller quorum.
3. Broker Migration: The brokers are restarted one by one in KRaft mode. During this rolling restart, if anything is not correct there will be an option to roll back the brokers in Zookeeper mode.
4. Finalizing the Migration: Once the cluster has been fully upgraded to KRaft mode, the controller will still be running in migration mode and making dual writes to KRaft and ZK. Once the operator has decided to commit to KRaft mode, the final step is to restart the controller quorum and take it out of migration mode by setting zookeeper.metadata.migration.enable "false" (or unsetting it).

Installation and Setup of Kafka

Step 1. Docker is one of the most popular container engines used in the software industry to create, package, and deploy applications.

Set up Docker Desktop on your machine and make sure the Docker engine is up and running.

Step 2. Deploy with Docker Compose. Here’s a sample Kafka cluster configuration that you can run using Docker Compose as a YAML file(attached docker-compose.yaml file) and save inside a folder:

version: '3'
services:
  
  kafka-1:
    image: 'bitnami/kafka:latest'
    container_name: kafka-1
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-1:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
      - KAFKA_CFG_NODE_ID=1
    ports:
      - 9192:9092

  kafka-2:
    image: 'bitnami/kafka:latest'
    container_name: kafka-2
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=2
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-2:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
      - KAFKA_CFG_NODE_ID=2
    ports:
      - 9292:9092
      
  kafka-3:
    image: 'bitnami/kafka:latest'
    container_name: kafka-3
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=3
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-3:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
      - KAFKA_CFG_NODE_ID=3
    ports:
      - 9392:9092

  kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka-1:9092
      - KAFKA_CLUSTERS_0_NAME=r4zt_wrqTRuT7W2NJsB_GA

Let me explain what these properties do.

• KAFKA_CFG_NODE_ID: This will act as the node ID in the cluster. This will help us identify which broker this is. It will also help us identify which Kraft controller node this is.
• KAFKA_CFG_PROCESS_ROLES: A node can act as an abrokercontrollerorboth. Here we are indicating that this node can be both a kafka broker and a kraft controller node.
• KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT: Here the broker listener name is set toPLAINTEXT.
• KAFKA_CFG_CONTROLLER_LISTENER_NAMES: Here the controller listener name is set toCONTROLLER.
• KAFKA_CFG_LISTENERS: Here we indicate that the broker will use port 9092 and the kraft controller will use port 19092.
• KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: Here the connection security details are added.
• KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: This is used to indicate all the Kraft controllers that are available. Here we are indicating that we will have 3 kraft controller nodes running on ports 19092, 19093, and 19094.

Step 3. Open Windows PowerShell -> navigate to the folder where the above docker-compose.yml file is present and Execute the below command.

docker-compose up -d

Step 4. Verify the same in the Docker desktop. After configuring the brokers, you can start them up as usual, using the Kafka-server-start script. The brokers will automatically elect a leader and replicate metadata across the cluster using the Raft consensus protocol.

Step 5. Now browse Kafka-ui in localhost with port number 8080 in the browser.

http://localhost:8080/

Step 6. Select Brokers from the menu. We can see the configured Kafka brokers.

Step 7. You will have a basic Kafka environment running and ready to use. From there, you’ll set up your topics, and write/read events into/from the topics.

Step 8. Open Powershell in admin mode and execute the below command to create a topic in Kafka and observe Topic is created successfully.

docker exec -it kafka-1 kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 1 --topic test-topic-KRaft

Once the Topic is created, the same can be verified in Kafka UI which is loaded in the browser with the created topic name.

Step 9. Now let's try to publish messages with the topic created above. Open another PowerShell in admin mode and execute the below command to publish the message.

docker exec -it kafka-1 kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic-KRaft

Now, let's try publishing some messages with the above Powershell window.

The above-published messages can be seen in Kafka which is loaded in the browser with the created topic name.

Image 1. An overview of the topic

Image 2. Messages available on the topic

Step 10. Now let's try to create a consumer to consume the published messages with the topic created above. Open another PowerShell in admin mode and execute the below command to publish the message.

The above-created Consumercan be seen in Kafka which is loaded in the browser with the created topic name.

The same consumer is available in the Consumers menu as well.

Once a Consumer is created and available in KafkaUI, try selecting the Consumer by clicking on the ID available in the Group ID column. This will show us the topic for which the consumer is subscribed.

Summary

The removal of Zookeeper dependency is a huge step forward for Kafka. The new KRaft mode feature will extend the scalability capabilities of Apache Kafka and also shorten the learning curve since now teams won’t have to worry about ZooKeeper any longer. Now, system administrators will find it easier to monitor, administer, and support Kafka. Developers will have to deal with a single security model for the whole system. Moreover, we have a lightweight single-process deployment to get started with Kafka.