Apache Kafka allows applications to process event streams across multiple consumers. A service might consume order events, process payments, update search indexes, or send notifications. When the rate of incoming messages becomes higher than the rate of processing, consumers begin falling behind.
This delay is known as consumer lag.
A growing lag does not automatically mean that Kafka needs more brokers or that the consumer group needs more instances. The problem may be a slow database query, an external API timeout, insufficient partitions, repeated message processing, or an inefficient consumer configuration.
Scaling without identifying the bottleneck can increase infrastructure costs without improving throughput. Adding consumers beyond the number of available partitions may not increase parallelism at all.
This article explains how Kafka consumer lag works, how to measure it, and how to identify the cause before changing your deployment.
What Is Kafka Consumer Lag?
Kafka stores messages in partitions. Each partition has an ordered sequence of records, and consumers track their progress using offsets.
A consumer group maintains committed offsets that indicate how far it has progressed through the partitions assigned to it. Kafka's lag measurement compares the consumer's progress with the latest available offset.
Conceptually:
Consumer lag = latest available offset - consumer's current position
The exact value shown by monitoring tools depends on whether they use committed offsets, consumer positions, and the way they calculate the log-end offset.
For example, suppose a partition has messages through offset 10,000, while the consumer group has committed offset 9,200. The difference is approximately 800 messages, depending on the offset convention used by the monitoring tool.
That means the consumer group has roughly 800 records of backlog on that partition.
A lag of 800 is not necessarily a problem. If the consumer is processing messages quickly and catching up, the backlog may disappear shortly. A smaller lag that continues growing for hours can be more serious.
The important metrics are lag, message arrival rate, processing throughput, and how quickly the consumer catches up after a traffic spike.
Check Consumer Lag with Kafka Consumer Groups
Kafka's command-line tools can help you inspect consumer group progress.
For a standard Kafka installation, run:
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group order-processing-group
Replace the broker address and consumer group name with your environment's values. For secured clusters, you may also need the appropriate client configuration.
The output typically includes columns such as:
TOPIC: The topic being consumed.PARTITION: The partition assigned to the consumer.CURRENT-OFFSET: The committed consumer offset.LOG-END-OFFSET: The latest offset represented by the tool's log-end measurement.LAG: The estimated difference between the log end and committed offset.CONSUMER-ID: The consumer instance identifier.HOST: The consumer's host.CLIENT-ID: The Kafka client identifier.
For example, you might see output similar to this:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
order-processing-group orders 0 9200 10000 800
order-processing-group orders 1 9800 10000 200
order-processing-group orders 2 10000 10000 0
These values are illustrative. They show that partition 0 has a larger backlog than the other partitions.
That difference is useful. A high lag concentrated on one partition may indicate an uneven workload or a slow consumer assigned to that partition. Similar lag across every partition may suggest that the whole consumer group is processing too slowly.
Take repeated measurements rather than relying on a single snapshot. A growing lag trend is usually more useful than an isolated number.
Find Out Whether the Consumer Is Falling Behind
Lag must be interpreted alongside message production and consumption rates.
Consider these situations:
- Producers publish 500 messages per second, while consumers process 700. The backlog can shrink if these rates continue.
- Producers publish 1,000 messages per second, while consumers process 700. The backlog grows by approximately 300 messages per second.
- Producers and consumers both process 1,000 messages per second. The backlog may remain stable, although the consumer might still be behind by an existing amount.
These examples assume comparable measurements and ignore other factors such as retries, filtering, and changes in partition activity.
Monitor the following metrics together:
| Metric | What it tells you |
|---|
| Consumer lag | How far the consumer group is behind |
| Records consumed per second | Consumer throughput |
| Records produced per second | Incoming workload |
| Processing duration | Time spent handling each record |
| Error and retry rate | Whether failures are slowing progress |
| Rebalance frequency | Whether partition assignments are changing repeatedly |
| Database or API latency | Whether downstream dependencies are slowing consumers |
A stable lag may be acceptable for some asynchronous workloads. For time-sensitive processing, such as payment notifications or inventory updates, the acceptable delay may be much lower.
Define a lag threshold based on business requirements rather than choosing an arbitrary number for every topic.
Check Whether Partitions Limit Consumer Parallelism
Kafka assigns partitions to consumers within a consumer group. A partition is assigned to at most one consumer in that group at a time.
Suppose a topic has four partitions and the consumer group has four active consumers. Each consumer can receive one partition.
If you increase the group to eight consumers, the additional consumers cannot receive more partitions from that topic while those four partitions remain the only available partitions. Some consumers will remain idle.
This is why adding instances does not always reduce lag.
Inspect the topic's partition count:
kafka-topics.sh \
--bootstrap-server localhost:9092 \
--describe \
--topic orders
Then inspect the consumer group:
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group order-processing-group
Check how many partitions exist, how they are distributed across consumers, and whether some consumers are idle.
Increasing the partition count can provide more parallelism, but it requires planning. It can affect ordering assumptions, key distribution, and how future records are assigned. More partitions also add metadata and operational overhead.
If the current partition count is already sufficient, investigate processing speed before changing the topic.
Find Slow Processing in the Consumer
A consumer may spend most of its time processing each record rather than fetching it from Kafka.
For example, an order consumer might perform several operations for every message:
- Deserialize the event.
- Validate the order.
- Query a database.
- Call an external service.
- Save the result.
- Commit the offset.
If the database or external service is slow, the consumer's throughput may fall even when Kafka itself is healthy.
Measure the duration of each processing stage. Application metrics and distributed tracing can help identify whether time is spent in database calls, HTTP requests, serialization, or business logic.
For .NET consumers using a Kafka client, record processing duration and failures around the message handler. Avoid logging every message at a high volume if it creates additional overhead. Aggregated metrics are often more useful for identifying persistent bottlenecks.
If processing one message takes 100 milliseconds, a strictly sequential consumer can handle roughly ten such operations per second under idealized conditions. Real throughput depends on scheduling, I/O behavior, batching, and other work performed by the application.
Before adding consumers, determine whether each record is doing unnecessary work or waiting for a slow dependency.
Improve Consumer Throughput with Batching
Processing records individually can create unnecessary overhead, especially when the downstream system supports bulk operations.
For example, a database may handle one batch insert more efficiently than many individual inserts. A consumer can collect a bounded number of records or wait for a short batching interval, then process them together.
Batching has trade-offs. Larger batches may improve throughput but increase memory usage and the time before an individual record is processed. If one record fails, the application also needs a clear strategy for handling partial success.
Use a bounded batch size and measure the effect on throughput and latency. Do not allow a consumer to accumulate an unlimited number of records in memory while waiting for a batch to fill.
For workloads that require low latency, smaller batches or immediate processing may be more appropriate.
Handle Offset Commits and Retries Carefully
Consumer lag can be affected by how the application commits offsets.
With automatic offset commits, the client commits progress according to its configuration. Depending on the processing flow, an offset may be committed before the application has safely completed the corresponding work.
Manual offset management can provide more control, but it also requires careful handling of failures and ordering.
A common design is to commit progress only after the message has been processed successfully. This reduces the risk of losing work when a consumer fails before completing processing, although a failure between completing the work and committing the offset can cause the message to be processed again.
That means consumers should be designed for duplicate delivery when the processing model permits it. Idempotent operations, unique event identifiers, and database constraints can help prevent duplicate side effects.
Retries also need limits and a strategy. If a single failing message is retried indefinitely, it can prevent progress on a partition, depending on the consumer's processing design. A dead-letter topic or another controlled failure path may be appropriate for messages that cannot be processed after the allowed attempts.
Do not simply skip failed messages without considering the data consistency requirements of the application.
Watch for Frequent Consumer Rebalances
Kafka rebalances occur when partition assignments change, such as when consumers join or leave a group.
Repeated rebalances can interrupt processing and reduce throughput. If consumers frequently exceed their configured processing limits or become unresponsive, the group may repeatedly redistribute partitions.
Investigate consumer logs and rebalance metrics. Check for application restarts, long-running message handlers, resource exhaustion, and consumer timeout configuration.
Increasing timeout values may help in specific cases, but it does not fix a consumer that is consistently overloaded. First identify why the consumer is failing to make progress or maintain its group membership.
For long-running processing, consider separating message retrieval from expensive work while preserving bounded queues, correct offset management, and partition-ordering requirements.
Build a Reliable Consumer Lag Monitoring Strategy
A useful monitoring system should show both the current backlog and whether it is improving.
Track lag by consumer group, topic, and partition. Add alerts for sustained growth, unexpected consumer inactivity, and processing latency that exceeds the application's service-level objectives.
Also monitor downstream dependencies. A growing Kafka lag accompanied by rising database latency suggests a different problem from lag that grows while database and API response times remain normal.
Avoid using one universal lag threshold for all topics. A low-volume administrative topic and a high-volume event stream have different traffic patterns and business requirements.
For important workflows, alert on the estimated time needed to process the backlog, not just the number of records. The time estimate should be based on observed throughput and current incoming traffic, and it becomes unreliable when those rates change significantly.
Common Mistakes When Troubleshooting Lag
Adding consumers without checking partitions
Consumers beyond the available partition count may remain idle. Verify the partition count and current assignments before scaling the group.
Scaling Kafka brokers before checking the application
A healthy broker cluster cannot compensate for a consumer that spends most of its time waiting for a database or external API.
Looking at only one lag measurement
A single snapshot does not show whether the backlog is growing, shrinking, or stable. Monitor trends and compare them with incoming message rates.
Increasing batch sizes without measuring latency
Larger batches can improve throughput but may delay individual messages and consume more memory. Test against both throughput and latency requirements.
Committing offsets before processing completes
This can cause work to be lost if the consumer fails after committing progress but before completing the operation. Design offset management around the application's delivery and consistency requirements.
Conclusion
Kafka consumer lag is a useful signal, but it does not identify the root cause by itself. The backlog may grow because the consumer is underpowered, the topic has too few partitions, a downstream dependency is slow, retries are excessive, or rebalances interrupt processing.
Start by measuring lag over time and comparing production rates with consumption throughput. Inspect partition assignments, profile message processing, review offset handling, and investigate downstream latency before increasing resources.
Scale only after identifying the bottleneck. That approach improves the chance that each additional consumer or partition will deliver a measurable benefit rather than simply increasing operational cost.
Join the conversation! Your thoughts help the community grow.