Kafka Fundamentals And Architecture

To understand Kafka in simple terms we can say that Kafka is a very fast and highly performant fault-tolerant publish and subscribe messaging system.

Kafka works with the stream of events and is basically an event streaming platform. When we say the term Event, then it becomes necessary to understand the real meaning behind the Event.

So, an Event can be understood as an atomic piece of data, for example when we go to any web site and we register or sign up to that website then that signup is an event that has some data associated with it also, which contains all the information which is really required to sign up.

The event can also think upon as a message with some data and Kafka is a platform that works with the stream of events.

Now after this it becomes imperative that we understand some of the concepts.

There are some fundamentals concepts that are really important to understand before we discuss the Kafka Cluster architecture.

Producers write events to Kafka. Producers don’t wait for the acknowledgment from brokers, instead they simply push data to the broker.

Consumers as the name implies, consume the events that are produced by the producers. It is also possible that the same application can behave both as producer and consumer but they will be producing different topics other than what the application is consuming.

But generally, entities like databases or data analytics applications acts as consumers as they need to store the data that has been generated by some other external systems.

Kafka is a middle system or layer which sits in the middle of Producers and Consumers.

Each consumer group has a unique Id. If there are multiple consumers inside the consumer group then the same message will not be read multiple times by the multiple consumers within the same group.

The topic is a channel or a place in Kafka inside which the producers produces messages and from where the Consumers consume the messages. Messages inside the topics are well structured and organized and a specific type of message is produced inside a specific topic only.

So in the flow the producer writes the data or message inside a topic and then the consumer consumes the message from that topic. Inside Kafka, the topic name has to be unique.