Step-By-Step Installation And Configuration Guide Of Apache Kafka On Windows Operating System

What is Kafka?

Kafka, licensed by Apache, is an open-source distributed stream-processing platform that is capable of handling more than trillions of events in a day. This massive platform has been developed by LinkedIn Team, written in Java and Scala, and donated to the Apache.

Step-By-Step Installation And Configuration Guide Of Apache Kafka On Windows Operating System

Real-world Applications by LinkedIn

LinkedIn is using Kafka for the scalable messaging system. About six years ago, LinkedIn was facing some problem in their messaging platform and they decided to redesign the infrastructure; that way, they introduced Kafka. That time Kafka was served only 1.4 trillion data over the year of an experiment by the LinkedIn team. Now, Kafka is smart enough to handle way more records.

What do you need on your development computer?

JDK 1.8
(Download Server JRE according to your OS and CPU from here.

Single node Kafka cluster
(Download Kafka 2.12-2.2.0 from here.)

Step By Step Kafka Configuration and Installation Process

Here, I am going to explain how to configure and install the Kafka server; please follow the below steps.

Install JDK 1.8 and set installation path into the Environment Variables on your Windows computer.

Please check the version of JDK.

Download the Kafka single cluster by using the following link.

http://kafka.apache.org/downloads

Configure the Zookeeper Properties

After downloading the Kafka single cluster, extract the folder and put it into your local drive. In here, we need to configure the Zookeeper. Zookeeper is basically used to manage the Kafka cluster and provides synchronization within distributed systems.

Create the folder into the Kafka folder with the name of zookeeper to keep the file log.
Go to your Kafka config directory. For me it’s D:\kafka\kafka_2.12-2.2.0\config, edit the zookeeper.properties file and change the following line,
1. dataDir=D:\Kafka\kafka_2.12-2.2.0\zookeeper

Configure the Kafka Server Properties

Create the folder into the Kafka folder with the name of kafka_log to keep the log file.
Go to your Kafka config directory. For me it’s D:\kafka\kafka_2.12-2.2.0\config, edit the server.properties file and change the following line.
1. # A comma separated list of directories under which to store log files
2. log.dirs=D:\Kafka\kafka_2.12-2.2.0\kafka_log
3. # The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
4. # For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
5. offsets.topic.num.partitions=1
6. offsets.topic.replication.factor=1
7. transaction.state.log.replication.factor=1
8. transaction.state.log.min.isr=1
9. min.inaync.replicas=1
10. default.replication.factor=1

How to start ZooKeeper and Kafka Server

You need to make sure your ZooKeeper instance is up and running before starting the Kafka server.

So, to run the ZooKeeper and Kafka instance, we need some commands to be available globally. Go to the Windows folder of Kafka folder > copy the path and set the path into the Environment Variable.

Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows
Open a command prompt and run the following command,
1. zookeeper-server-start.bat D:\Kafka\kafka_2.12-2.2.0\config\zookeeper.properties

After running the zookeeper follow the below steps.

Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows.
Open a command prompt and run the following command,
1. kafka-server-start.bat D:\Kafka\kafka_2.12-2.2.0\config\server.properties

Creating Topics

The Kafka topic has been divided into the number of partitions; you can say it is an anatomy of Kafka. It is the most essential part of this distributed messaging system. Topic should have a name to understand the purpose of the message that is stored and published into the server.

Now the time to create a topic with the name of “chat-message”. We are running it into only one Kafka server.

Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows
Open a command prompt and run the following command,
1. kafka-topics.bat --create --zookeeper localhost:2181 -replication-factor 1 --partitions 1 --topic chat-message

Creating a Producer and Consumer the Topic from the Kafka Server

First of all, we need to consume the topic from the server, so for that follow the below steps

Go to your Kafka installation directory: For me, it’s D:\kafka\kafka_2.12-2.2.0\bin\windows
Open a command prompt and run the following command,
1. kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic chat-message --from-beginning

After consuming the topic we need to produce the message by the following command.

kafka-console-producer.bat --broker-list localhost:9092 --topic chat-message

In here, the message is coming from the Kafka server through the chat-message topic and in the second window, we can see the message that we have produced through the Kafka server.

10 Comments

Join the conversation! Your thoughts help the community grow.

Bhanu Singh6y ago
D:\kafka\kafka_2.13-2.6.0\bin\windows>kafka-server-start.bat D:\kafka\kafka_2.13-2.6.0\config\server.properties is not working and throwing error "The system cannot find the path specified."
- Satyaki Chakraborty6y ago
  Please reinstall your kafka engine, and try again
Rouse Rouse6y ago
Great article. thanks for the time to put that up.
Teofil BOCANCEA6y ago
Great article! very clear and easy to follow - thanks.
- Satyaki Chakraborty6y ago
  Thanks for the feedback
Taha Mubarak6y ago
Thank you very much .
Sourav Kumar Das7y ago
Liked this Article..
- Satyaki Chakraborty7y ago
  Thanks for your feedback Sourav.
Sourav Kumar Das7y ago
Great Article Sir.
- Satyaki Chakraborty7y ago
  Thank you.