Apache Kafka Quick Setup Guide

Apache Kafka Setup

If you have ever used LinkedIn, Uber, PayPal, Netflix, Twitter, Pinterest etc., then congratulations! you have experienced Apache Kafka service. Yes, these all big firms are using this open source system to tackle the complexity of their messaging model. Kafka is capable of tolerating millions of messages per secind and in a very efficient way. And it's very easy to set up Apache Kafka on a Windows machine.

Ingredients

  1. Download Java
  2. Setup Java environment
  3. Download Apache Kafka

Prerequisite

  1. Download and unzip Apache Kafka library.

    Apache Kafka library
  2. Configure Apache Kafka.

    1. Set environment variables for Kafka.

    2. Open system properties.

      Apache Kafka library
    3. Click "Environment Variables" and add new "System Variable" as follow. 

    4. Click "New" and create "KAFKA_HOME" variable

      Apache Kafka library
    5. Add "%KAFKA_HOME%\bin" in "Path" variable. 

      Apache Kafka library 
Recipe

To set up the basic Kafka messaging model, we need to understand a couple of terms. This article gives you an extremely basic overview of Apache Kafka.

A couple of the most important parts of Apache Kafka.
  1. Zookeeper
  2. Kafka Server
  3. Topic (Channel)
  4. Producer (Sender)
  5. Consumer (Receiver)
Start Zookeeper Server

zookeeper-server.bat ../../config/zookeeper.properties

Apache Kafka library 
Apache Kafka library 
Start Kafka Server

kafka-server.bat ../../config/server.properties

Apache Kafka library 
Apache Kafka library
Create Topic

kafka-topics.bat --create --topic KafkaDemo --zookeeper localhost:2181 --replication-factor 1 --partitions 1 

Apache Kafka library
We can see an entry of created topics in the running Kafka server.

Apache Kafka library 
Create Producer (Sender)

kafka-console-producer.bat --broker-list localhost:9092 --topic KafkaDemo

Apache Kafka library
Create Consumer(Receiver)

kafka-console-consumer.bat --zookeeper localhost:2181 --topic KafkaDemo --from-beginning

Apache Kafka library 
Write a message in Producer console, hit enter and we will receive it in the consumer console.

Apache Kafka library
And here, we are done with an extremely basic Apache Kafka messaging setup. I have not covered Partition, Broker, Cluster and many more topics in this blog. The intention was just to explain how quickly we can set up the Kafka messaging system.