Kafka Consumer Recovery: How to Resume Processing After a Crash

A Kafka consumer can process messages successfully for hours and then suddenly stop because of a server restart, application crash, network failure, or unexpected exception.

When the consumer starts again, it needs to continue from the correct position without skipping messages or repeating business operations unnecessarily.

This is where Kafka offsets, consumer groups, commits, and rebalancing become important. A consumer's recovery behavior depends on when it commits an offset, whether its work has completed, and how the application handles messages that are delivered again.

In this article, we will examine how Kafka consumer recovery works, how to handle failures in a C# application, and how to avoid common problems when processing messages in production.

How Kafka Consumer Recovery Works

Kafka stores messages in partitions. Each message has an offset that identifies its position within a partition.

A consumer reads messages by polling a partition and advances through its offsets as it processes records. Kafka uses committed offsets to help a consumer group determine where to resume after a restart or rebalance.

For example, suppose a partition contains these messages:

Message	Offset	Processing status
Order A	100	Processed
Order B	101	Processed
Order C	102	Processing
Order D	103	Not processed

If the consumer crashes while processing Order C, the next consumer's starting position depends on the committed offset.

If the committed offset indicates that the next record to read is 102, the consumer can read Order C again after recovery.

This is normal Kafka behavior. Kafka does not automatically know whether your database update or external API call completed before the consumer crashed.

Your application must coordinate message processing with offset commits to achieve the delivery and processing guarantees it needs.

Step 1: Understand Kafka Offsets

An offset is a record's position within a specific partition. Offsets are not global message IDs, and the same numeric offset can exist in multiple partitions.

A committed consumer offset represents the next position the consumer group should read from that partition.

For example, if offset 102 has been processed and the consumer commits 103, the consumer group can resume from offset 103.

This distinction matters because committing the offset of the last processed message instead of the next position can cause recovery logic to start from an incorrect location.

Kafka also distinguishes between the consumer's current position and its committed position:

Current position: the next record the consumer will fetch.
Committed position: the next record the group should resume from after recovery.

The current position can advance before a record's business operation has completed. That is why offset management must be designed carefully.

Step 2: Configure a Kafka Consumer in C#

The following example uses the Confluent.Kafka .NET client.

Install the package in your application:

dotnet add package Confluent.Kafka

Create a consumer configuration:

using Confluent.Kafka;

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "order-processing-group",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = false
};

The important settings are:

BootstrapServers identifies the Kafka brokers.
GroupId identifies the consumer group.
AutoOffsetReset determines where to start when there is no valid committed offset for a partition.
EnableAutoCommit disables periodic automatic offset commits so the application can control when offsets are committed.

AutoOffsetReset does not normally override a valid committed offset. It applies when the group has no valid committed offset for the partition, such as during an initial run or when the previous offset is no longer available.

Using Earliest can help a new consumer read from the earliest available offset. It does not mean that every restart begins at the beginning of the topic.

Step 3: Process Messages Before Committing Offsets

For important business operations, a common approach is to process a record and commit its offset only after the processing succeeds.

The following example demonstrates this pattern:

using Confluent.Kafka;

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "order-processing-group",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = false,
    EnableAutoOffsetStore = false
};

using var consumer =
    new ConsumerBuilder<string, string>(config).Build();

consumer.Subscribe("orders");

while (true)
{
    try
    {
        var result = consumer.Consume(
            CancellationToken.None);

        await ProcessOrderAsync(result.Message.Value);

        consumer.Commit(result);
    }
    catch (ConsumeException ex)
    {
        Console.Error.WriteLine(
            $"Kafka consume error: {ex.Error.Reason}");
    }
    catch (Exception ex)
    {
        Console.Error.WriteLine(
            $"Order processing failed: {ex.Message}");
    }
}

This example illustrates manual offset control, but production code needs a deliberate failure policy. In particular, it must not keep consuming and silently skipping records when processing fails.

The configuration disables automatic offset storage as well as automatic commits. That makes it easier to control which offsets become eligible for commitment.

In this simple sequential loop, the consumer commits the record's offset only after ProcessOrderAsync completes. If the process crashes before the commit, the record can be delivered again after recovery.

If processing succeeds but the commit fails or its result is unknown, the record may also be delivered again. Your business operation should therefore be idempotent.

For production applications, use a cancellation token, handle shutdown gracefully, and distinguish Kafka client exceptions from application-processing failures.

Step 4: Understand What Happens When Processing Fails

Suppose the consumer receives a message at offset 102, but the database is temporarily unavailable.

The consumer has several possible choices:

Retry the database operation.
Stop processing and allow the application to recover.
Send the failed record to a dead-letter topic after applying a defined retry policy.
Record the failure for later investigation and continue only when doing so preserves the required ordering and business rules.

What it should not do is commit the offset as though the record had succeeded when the business operation has not completed.

Consider this sequence:

The consumer reads offset 102.
The application attempts to save an order.
The database operation fails.
The application commits offset 103 anyway.

On recovery, Kafka starts at offset 103. The failed record at 102 will not be replayed as part of normal group recovery.

For critical workloads, make sure that failed records are retried or durably recorded before advancing past them.

Step 5: Make Database Processing Idempotent

Committing offsets after processing reduces unnecessary duplicate work, but it cannot eliminate every duplicate.

Consider this sequence:

The consumer reads an order event.
The database transaction commits successfully.
The consumer crashes before committing the Kafka offset.
Kafka delivers the event again after recovery.

The database update may now be attempted twice.

One way to handle this is to store the Kafka event ID in a database table with a unique constraint.

For example:

CREATE TABLE dbo.ProcessedKafkaMessages
(
    EventId NVARCHAR(200) NOT NULL PRIMARY KEY,
    ProcessedAt DATETIME2 NOT NULL
        DEFAULT SYSUTCDATETIME()
);

The application can use this table to prevent the same logical event from applying its business effects more than once.

However, checking the table and updating the business data must be coordinated correctly. If the application records an event as processed separately from the business update, a crash between the two operations can cause inconsistent results.

When both operations use the same relational database, perform the processed-event insert and the business changes in one transaction. The unique constraint then helps protect against concurrent duplicate processing.

If processing includes an external service, such as a payment provider, use that service's idempotency support when available. A database transaction cannot roll back an external operation that has already succeeded.

Step 6: Handle Consumer Group Rebalancing

Kafka distributes partitions among consumers within a consumer group. When a consumer joins, leaves, or fails, Kafka may rebalance the group and assign partitions to different consumers.

A rebalance can occur during recovery after a crash. It can also happen when a consumer is considered unresponsive or when the group membership changes.

During a rebalance, the consumer may lose ownership of a partition. Any uncommitted work needs careful handling because another consumer may receive that partition.

For reliable processing:

Commit offsets only for work that has completed successfully.
Stop accepting new work from partitions that are being revoked.
Finish or safely abandon in-progress work according to the application's design.
Avoid committing offsets for records whose business operations are still running.
Monitor rebalance frequency and processing duration.

The Confluent.Kafka client provides partition-assignment and revocation callbacks through its consumer builder. Use these callbacks when your application needs explicit control over work during rebalances.

If processing is slow, review consumer polling behavior and the configured maximum polling interval. A consumer that spends too long processing records without polling can be removed from the group, depending on the configuration and client behavior.

Step 7: Understand the Limits of Automatic Commits

Kafka consumers can use automatic offset commits, but automatic commits do not guarantee that the application has finished processing the corresponding records.

For example, a consumer might fetch several messages, begin processing them, and then commit offsets automatically. If the process crashes before all business operations finish, recovery may resume after some records whose processing was incomplete.

This creates a risk of lost work from the application's perspective.

Automatic commits can be suitable for applications where occasional reprocessing or loss of in-flight work is acceptable. For critical business processing, manual offset control is often easier to reason about.

Manual commits still require care. If you process records concurrently, do not commit an offset beyond a record that has not completed successfully. Kafka offsets advance through a partition in order, so committing a later position can cause earlier unfinished records to be skipped during recovery.

Step 8: Recover from a Poison Message

A poison message is a record that repeatedly fails because of invalid data, an unsupported schema, or a business rule that cannot be satisfied.

Retrying such a record forever can stop progress on its partition.

A common strategy is to use bounded retries and then publish the failed record to a dead-letter topic. The dead-letter record should include enough information to investigate the failure, such as:

Original topic and partition.
Original offset.
Event identifier, if available.
Failure reason.
Retry count or processing-attempt information.
Relevant timestamps.

Only advance the original consumer offset after the failed record has been handled according to the application's policy. If publishing to the dead-letter topic succeeds but committing the original offset fails, the dead-letter record might be published more than once.

For that reason, dead-letter processing should also tolerate duplicates or use an appropriate transactional design.

Kafka transactions can coordinate consumed offsets with produced Kafka records in supported processing patterns, but they do not automatically make arbitrary external database or API operations part of the same atomic transaction.

Step 9: Monitor Recovery and Consumer Health

A consumer can appear healthy while falling behind or repeatedly failing on the same records.

Monitor the following metrics:

Metric	Why it matters
Consumer lag	Shows how far the consumer group is behind the latest available records
Processing duration	Helps identify slow handlers or overloaded dependencies
Commit failures	Indicates problems recording recovery positions
Rebalance frequency	Can reveal unstable consumer membership
Error and retry counts	Helps identify failing records or dependencies
Dead-letter volume	Shows how often records cannot be processed normally
Database latency	Helps identify a downstream bottleneck
Consumer restarts	Helps detect crashes or deployment-related interruptions

Consumer lag alone does not prove that processing is broken. A high incoming message rate can produce lag even when consumers are operating correctly.

Compare lag trends with processing throughput, error rates, and the rate at which new messages arrive.

Common Mistakes to Avoid

Committing before business processing completes

This can cause recovery to skip work that never finished. Commit only after the required operation has succeeded or the record has been durably handled according to your policy.

Assuming manual commits prevent duplicates

A crash after a database commit but before the Kafka offset commit can cause the same record to be processed again. Make business operations idempotent.

Treating AutoOffsetReset as a restart setting

AutoOffsetReset applies when a valid committed offset is unavailable. It does not normally replace an existing committed position.

Ignoring records that fail repeatedly

A poison message can block progress, particularly when processing must preserve order within a partition. Use bounded retries and a defined dead-letter strategy.

Committing offsets incorrectly during concurrent processing

Do not commit past records that have not completed. Track completion carefully when processing multiple records concurrently.

Assuming Kafka transactions cover a database

Kafka transactions can provide atomicity for supported Kafka-to-Kafka processing flows. A SQL Server transaction or external API call is not automatically included.

Best Practices for Production Applications

Use manual offset control for critical business operations.
Disable automatic offset storage when explicit control over eligible offsets is required.
Commit offsets only after successful processing or durable failure handling.
Use stable event identifiers and database uniqueness constraints.
Keep database updates and processed-event records in one transaction when possible.
Use bounded retries and dead-letter topics for repeatedly failing records.
Handle consumer shutdown and partition rebalancing deliberately.
Monitor consumer lag, processing latency, commit failures, and rebalances.
Avoid committing past unfinished records when using concurrent processing.
Test crashes before and after database commits, and before and after offset commits.

Summary

Kafka consumer recovery depends on the relationship between message processing and committed offsets. If a consumer crashes before committing an offset, the record can be delivered again. If an offset is committed before the business operation completes, the application can lose work.

For reliable recovery, control offsets deliberately, make database operations idempotent, and define how failed records are retried or moved to a dead-letter topic.

Also account for consumer group rebalancing and concurrent processing. These practices help a Kafka consumer resume safely after crashes without silently skipping important work or creating unnecessary duplicate business effects.