A Kafka consumer can be running without processing messages fast enough to keep up with incoming traffic. At first, the delay may be small. As producers continue publishing messages, the backlog grows, and applications begin processing increasingly old data.
This problem is known as consumer lag.
Consumer lag matters in systems that process orders, payment events, application logs, notifications, analytics events, and other streaming data. A growing backlog can delay business workflows, make dashboards outdated, and cause time-sensitive events to be processed too late.
However, consumer lag does not always mean that a consumer is broken. It can increase because of a traffic spike, slow database operations, insufficient consumer capacity, partition imbalance, or a problem with the downstream service.
The important task is to identify why lag is growing and determine whether the consumer can recover without intervention.
In this article, we will examine how Kafka consumer lag works, how to measure it, how to investigate common causes, and how to improve consumer throughput without introducing duplicate processing or data loss.
What Is Kafka Consumer Lag?
Kafka stores messages in partitions. Each partition maintains an ordered sequence of records identified by offsets.
A consumer group reads records from these partitions and tracks its progress by committing offsets. Consumer lag measures how far the consumer group's committed progress is behind the latest available records.
For a simplified example, suppose a partition has the following values:
- Latest log end offset:
15000 - Consumer group's committed offset:
12000
The difference is 3,000 offsets.
In this example, the consumer group has 3,000 offsets of committed lag for that partition.
The calculation is commonly represented as:
Consumer lag = Log end offset − Committed offset
Kafka offsets are zero-based, and the log end offset represents the next offset that would be assigned to a new record. Under the usual offset conventions, subtracting the committed next offset from the log end offset gives the number of offsets between them.
Actual monitoring tools may use different offset definitions, so confirm how your tool calculates lag before comparing values.
Also, offsets are not the same as business events successfully processed. A consumer can read records but fail while writing them to a database, or it can process records but delay committing its progress.
For that reason, committed-offset lag is an important metric, but it should not be the only measure of application health.
Why Consumer Lag Matters
A growing backlog can affect several parts of a system.
Delayed business processing
An order-processing service may receive an order event immediately but process it several minutes later. The customer sees a delay even though the producer successfully published the message.
Stale analytics and dashboards
A reporting pipeline that falls behind can display outdated metrics. During a traffic spike, dashboards may show a business state that no longer reflects current activity.
Increased recovery time
If a consumer remains slower than its producer for an extended period, the backlog may continue growing. Even after the underlying problem is fixed, the consumer needs enough spare capacity to process the accumulated records.
Risk of exceeding retention limits
Kafka retains records according to topic and broker configuration. If consumers remain behind for too long and older records are removed by retention, the consumers may no longer be able to read the entire backlog from the original topic.
The actual impact depends on the topic's retention policy, the consumer's position, and whether the required data remains available.
Step 1: Measure Consumer Lag
Start by checking the consumer group's committed offsets and the latest offsets in the topic.
Kafka's command-line tools include kafka-consumer-groups.sh, which can display consumer-group information.
For example:
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group order-processing-group
A typical result contains columns similar to these:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
order-processing-group orders 0 12000 15000 3000
order-processing-group orders 1 18000 18100 100
order-processing-group orders 2 9000 9000 0
The exact output depends on the Kafka version and command options.
This example shows that partition 0 has substantially more lag than the other partitions. Partition 1 has a small backlog, while partition 2 has no committed-offset lag.
That pattern is useful because it suggests the problem may not affect the entire consumer group equally.
Look at the following values:
CURRENT-OFFSET: The consumer group's committed progress.LOG-END-OFFSET: The latest offset boundary for the partition.LAG: The reported difference between those offsets.
If the command reports missing or unavailable offsets for an inactive group, check whether the group has committed offsets and whether the group is currently active.
For production systems, use your existing Kafka monitoring platform as well. A single command gives you a snapshot, while time-series metrics show whether lag is increasing, stable, or decreasing.
Step 2: Determine Whether Lag Is Growing
A lag value of 50,000 may be normal for a high-throughput system. A lag value of 500 may be serious if the application has a strict processing deadline.
The trend is often more useful than the absolute number.
For example, consider this simplified monitoring history:
| Time | Consumer lag |
|---|
| 10:00 | 1,000 |
| 10:05 | 2,500 |
| 10:10 | 5,000 |
| 10:15 | 8,500 |
The backlog is growing, so the consumer group is not keeping up with incoming records.
Now consider a different pattern:
| Time | Consumer lag |
|---|
| 10:00 | 20,000 |
| 10:05 | 18,000 |
| 10:10 | 14,000 |
| 10:15 | 9,000 |
The backlog is large, but the consumer group is catching up.
For a simple queue model:
Change in backlog = Incoming records − Processed records
If consumers process fewer records per second than producers publish, lag tends to increase. If processing capacity exceeds the incoming rate, lag can decrease.
In a real Kafka deployment, partition reassignment, retries, offset commits, retention, and changes in traffic can also affect the observed values.
Monitor both the current backlog and its rate of change. Where processing deadlines matter, also measure the age of the oldest unprocessed record or another meaningful end-to-end processing delay.
Step 3: Check Whether Consumers Are Active
The next step is to determine whether the consumer group has enough active consumers.
Run:
kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group order-processing-group \
--members
Depending on the Kafka version, this can show active members and their assignments.
Investigate these questions:
- Are all expected consumer instances running?
- Did an instance crash or stop polling?
- Are consumers repeatedly joining and leaving the group?
- Are partitions assigned unevenly?
- Is the number of consumers greater than the number of partitions?
Consumer groups assign each partition to one consumer within the group at a time. If a topic has six partitions and the group has ten consumers, at most six consumers can actively own those partitions under the standard assignment model. The remaining consumers may be idle.
Adding consumers beyond the available partition count does not automatically increase parallel processing for that topic.
If lag grows across all partitions, the group may need more processing capacity or more efficient work per message. If only one partition has high lag, adding consumers may not help unless the partitioning or processing design changes.
Step 4: Identify Slow Message Processing
A consumer can receive messages quickly but process them slowly.
For example, an order consumer might perform the following operations for every record:
- Deserialize the event.
- Validate the order.
- Call an external service.
- Write the result to a database.
- Commit the Kafka offset.
If each record takes 100 milliseconds to process sequentially, the consumer can process only about ten records per second under those assumptions.
If the producer sends 100 records per second to the same processing path, the backlog will grow.
The exact throughput depends on batching, concurrency, database latency, retries, and other implementation details, but the example shows why processing time matters.
Instrument the consumer to measure:
- Time spent deserializing and validating records.
- Database query and transaction duration.
- External HTTP request latency.
- Retry counts and retry delays.
- Time spent waiting for locks or connection-pool resources.
- Records processed per second.
- Errors and failed records.
Do not assume that Kafka itself is slow simply because lag is increasing. In many systems, the bottleneck is a downstream dependency.
Step 5: Check Partition-Level Lag
Consumer-group lag should be examined per partition, not only as a total.
Suppose a topic has eight partitions. Seven partitions have almost no lag, while one partition has 100,000 offsets of lag.
This can indicate a partition-specific problem.
Possible causes include:
- A hot key sends too much traffic to one partition.
- A single consumer assignment is processing unusually expensive records.
- One consumer instance has limited CPU or memory.
- A downstream operation is repeatedly failing for records in that partition.
- A particular partition receives substantially more records than the others.
Kafka preserves ordering within a partition. If the application requires ordering for a particular key, all records for that key generally need to follow the same partitioning strategy.
That requirement can create an imbalance when a small number of keys account for a large share of traffic.
How to investigate an uneven partition
First, inspect lag by partition. Then correlate the affected partition with consumer assignments, processing latency, record volume, and error logs.
If one partition consistently receives much more traffic than others, review the producer's partitioning strategy and key distribution.
Changing the partitioning strategy can improve distribution, but it may affect ordering guarantees and application behavior. Do not change keys or partition counts without understanding those consequences.
Step 6: Check Rebalances and Offset Commits
Consumer rebalances occur when group membership or partition assignments change. They are a normal part of Kafka consumer-group behavior, but frequent rebalances can reduce useful processing time.
Common triggers include:
- Consumer instances repeatedly restarting.
- Network interruptions.
- Processing that exceeds the configured poll interval.
- Deployment changes.
- Consumer-group membership changes.
Review consumer logs for repeated rebalances, lost assignments, and errors around polling or committing offsets.
Avoid blocking the consumer loop
If a consumer performs long-running work before polling again, it may exceed max.poll.interval.ms. Kafka can consider the consumer unresponsive and initiate a rebalance.
One option is to make each processing batch smaller or reduce the time spent on work between polls.
Another option is to move processing into a carefully designed worker model. However, this requires additional control over in-flight records, partition ownership, ordering, and offset commits.
Simply increasing max.poll.interval.ms may be appropriate for legitimately long processing operations, but it does not fix slow processing itself.
Commit offsets only after successful processing
Offset commits determine where the group resumes after a restart. If an application commits an offset before completing the corresponding business operation, a failure can cause a record to be skipped after restart.
If it processes a record but fails before committing the offset, Kafka may deliver that record again.
For a database-backed consumer, a common design is to commit the database transaction successfully before committing the Kafka offset. This supports at-least-once processing, but it can still produce duplicates if the process fails between those two steps.
Make downstream operations idempotent, or use an appropriate deduplication or transactional design where necessary.
Step 7: Improve Consumer Throughput
Once the bottleneck is known, choose a change that addresses the actual cause.
Increase consumer capacity where parallelism is available
If multiple partitions have growing lag and consumers are saturated, additional consumer instances may help.
For example, a topic with twelve partitions can potentially distribute its work across up to twelve active consumers in a group, subject to assignment and operational constraints.
More consumers will not help when the topic has only one partition or when the bottleneck is a shared database that cannot handle additional load.
Reduce per-record overhead
Processing messages individually can introduce unnecessary network and database overhead.
Where business requirements allow it, batch database writes or process records in controlled batches.
For example, writing 100 records in one database operation may be more efficient than executing 100 independent transactions. The actual benefit depends on the database, transaction size, and application logic.
Batching also introduces tradeoffs. Large batches can increase latency, memory use, and retry costs. Choose a batch size that fits the workload.
Use concurrency carefully
Independent records may be processed concurrently to increase throughput. However, concurrent processing can change completion order.
If a partition contains events that must be processed in order, unrestricted parallel processing can violate application-level ordering even though Kafka delivers records in partition order.
A safe design needs explicit control over the relationship between polling, processing, and offset commits.
Fix downstream bottlenecks
If the consumer spends most of its time waiting for a database, investigate database indexes, connection-pool saturation, transaction duration, and query execution plans.
If external HTTP calls dominate processing time, check service latency, rate limits, retry policies, and connection reuse.
Increasing Kafka consumer count without fixing a saturated downstream dependency can make the problem worse.
Step 8: Set Useful Lag Alerts
A production system should alert on lag trends and business impact rather than one arbitrary threshold.
Useful signals include:
- Total consumer-group lag.
- Lag by topic and partition.
- Rate of lag growth.
- Oldest-record processing delay.
- Records processed per second.
- Consumer-group membership changes.
- Processing errors and retry rates.
- Consumer instance health.
- Downstream database and API latency.
For example, a payment-processing pipeline may require an alert when processing delay exceeds a business-defined limit. A log-processing pipeline may tolerate a larger backlog if the backlog is shrinking and retention remains sufficient.
The right alert threshold depends on the service-level objective, traffic pattern, and retention policy.
Whenever possible, combine lag alerts with an indicator of whether the consumer group is making progress. A large but decreasing backlog may need monitoring, while a smaller backlog that has stopped moving may require immediate investigation.
Common Mistakes to Avoid
- Adding consumers without checking partitions: Consumers beyond the available partition parallelism may remain idle.
- Treating total lag as the only metric: One overloaded partition can be hidden by a healthy group-wide average.
- Increasing poll intervals without investigating processing time: A longer interval may reduce rebalances but leave the underlying bottleneck untouched.
- Committing offsets before successful processing: This can cause messages to be skipped after a failure.
- Assuming every retry is harmless: Reprocessing can create duplicate database writes or repeated external side effects.
- Increasing concurrency without considering ordering: Messages from the same partition may finish processing out of order.
- Ignoring retention: A sufficiently old backlog can outlast the topic's record-retention window.
- Alerting only on a fixed lag value: Lag must be interpreted alongside its growth rate and the application's processing deadline.
Best Practices for Managing Kafka Consumer Lag
- Monitor lag at the consumer-group and partition levels.
- Track processing throughput and end-to-end record age alongside committed-offset lag.
- Inspect consumer membership, assignments, and rebalance frequency.
- Measure database and external-service latency before scaling consumers.
- Tune batch size and concurrency based on measured workload behavior.
- Preserve ordering where required by the application's business rules.
- Commit offsets only after the associated processing succeeds.
- Make processing idempotent when duplicate delivery is possible.
- Configure retention with realistic recovery and outage scenarios in mind.
- Test recovery by simulating consumer restarts, downstream failures, and traffic spikes.
Summary
Kafka consumer lag measures how far a consumer group's committed progress is behind the topic's latest available offsets. A growing backlog can indicate insufficient processing capacity, slow downstream operations, partition imbalance, frequent rebalances, or other problems in the consumer pipeline.
Start by measuring lag per partition and observing whether it is growing or shrinking. Then inspect consumer membership, processing latency, offset commits, and downstream dependencies before deciding how to scale or tune the system.
Adding consumers is useful only when the workload can be distributed across available partitions and downstream services have enough capacity. Reliable offset handling, idempotent processing, and appropriate retention policies are equally important.
With the right metrics and a structured investigation, teams can distinguish a temporary backlog from a genuine processing bottleneck and restore the consumer group's ability to keep up with incoming traffic.
Join the conversation! Your thoughts help the community grow.