Setting up Kafka replication across AWS Regions is only the first part of a disaster recovery strategy.
The more important question is whether the application can actually recover when the primary Region becomes unavailable.
A replication configuration can appear healthy while the failover process still has problems. Consumer applications may not connect to the secondary MSK cluster, offsets may not behave as expected, duplicate events may be processed, or applications may take much longer to recover than expected.
This is why Kafka failover testing should be treated as an operational test rather than a configuration check.
Amazon MSK Replicator can replicate Kafka data between supported Amazon MSK clusters across AWS Regions. The replication layer helps maintain the secondary Kafka environment, but your application still needs a tested procedure for switching producers, consumers, and dependent services to the secondary Region.
This article focuses on testing that complete failover process.
What Kafka Failover Testing Should Prove
A useful failover test should answer several questions:
Can Kafka data reach the secondary Region?
Can applications connect to the secondary MSK cluster?
How much replication lag exists?
Can consumers continue processing?
Are duplicate events handled safely?
How long does recovery take?
Can production traffic be redirected?
Can the system return to the primary Region?
Testing only the first question is not enough.
A successful replication health check does not prove that the entire application can recover from a regional failure.
Basic Multi-Region Architecture
A typical active-passive setup looks like this:
Region A
Primary Region
|
v
Application
|
v
MSK Cluster
|
v
MSK Replicator
|
Cross-Region
|
v
MSK Cluster
|
v
Region B
DR Region
|
v
DR Application
During normal operation, Region A handles production traffic.
Region B receives replicated Kafka data and remains ready for recovery.
During a failover test, the goal is to simulate the loss of Region A and verify that Region B can take over.
Define RTO and RPO First
Do not start a failover test without defining what success means.
Two important measurements are Recovery Point Objective and Recovery Time Objective.
RPO answers:
How much recent Kafka data can the business afford to lose?
RTO answers:
How long can the application remain unavailable?
For example:
RPO = 5 minutes
RTO = 30 minutes
These are example targets.
Your test should measure actual performance against the targets rather than simply declaring the failover successful.
What to Measure During the Test
At minimum, capture:
Metric | Why It Matters |
|---|
Replication lag | Shows how current the DR cluster is |
Last replicated event | Helps identify the recovery point |
Failover start time | Establishes the recovery timeline |
Consumer recovery time | Measures application restart |
Traffic switch time | Measures routing recovery |
First successful DR event | Confirms processing resumed |
Duplicate events | Measures processing consistency |
Data loss | Validates RPO |
Total recovery time | Validates RTO |
Collect these measurements before, during, and after the test.
Create a Test Topic
Do not begin your first failover test with a critical production topic.
Create a controlled test topic such as:
dr-failover-test
Publish identifiable events to it.
For example:
event-001
event-002
event-003
event-004
event-005
The event IDs make it easier to verify whether every event reached the secondary environment.
For a more realistic test, generate events continuously.
Produce Events Continuously
A one-time test is useful, but continuous event generation provides better information.
For example:
Every 1 second:
Publish event with unique ID
The event payload might contain:
{
"eventId": "dr-test-10091",
"createdAt": "2026-11-09T10:15:00Z",
"sourceRegion": "primary",
"sequence": 10091
}
The unique sequence number makes missing or duplicated events easier to identify.
Establish a Baseline
Before testing failover, observe normal operation.
Record:
Producer rate
Consumer rate
Replication lag
Consumer latency
Processing throughput
MSK health
Application health
For example:
Metric | Baseline |
|---|
Producer rate | 100 events/sec |
Consumer rate | 100 events/sec |
Replication lag | < 2 sec |
Processing latency | 150 ms |
Failed messages | 0 |
The numbers are illustrative.
The purpose is to establish normal behavior so you can compare it with the failover state.
Step 1: Confirm Replication
Before causing any failure, verify that events produced in the primary Region appear in the secondary cluster.
Use a known event sequence.
For example:
Primary:
event-1001
event-1002
event-1003
event-1004
event-1005
Secondary:
event-1001
event-1002
event-1003
event-1004
event-1005
Do not continue until the baseline replication behavior is understood.
Step 2: Measure Replication Lag
Replication is not necessarily instantaneous.
During normal operation, measure the time between an event becoming available in the source and appearing in the destination.
A simplified calculation is:
Replication Lag =
Destination Event Time - Source Event Time
Use a consistent timestamp source when measuring this.
Also monitor the service's available replication metrics rather than relying only on application timestamps.
Step 3: Stop Primary Consumers
The first controlled failure should usually focus on consumers.
Stop the primary consumer application while leaving replication running.
This tests whether the DR application can take over without introducing two active consumers processing the same business workload.
For example:
Primary Consumer
|
X
STOP
|
v
Secondary Consumer
|
v
Secondary MSK
Verify that the DR consumer can connect and process the available events.
Step 4: Start the DR Consumer
Start the consumer application in Region B.
Confirm:
Kafka connection succeeds
Authentication succeeds
Consumer starts
Expected topic is available
Consumer group behavior is correct
New events are processed
Application dependencies are available
Do not consider the test successful merely because the consumer process started.
Verify actual business processing.
Consumer Offset Behavior
Consumer offsets are critical during failover.
A consumer needs to know where it should begin processing.
The recovery process must define whether the DR consumer:
Continues from replicated consumer state
Starts from a known offset
Reprocesses a controlled range
Uses application-level deduplication
The correct strategy depends on the replication configuration and application design.
Do not assume that source and destination consumer state behave exactly like a local Kafka failover.
Test the behavior with real events.
Test Duplicate Processing
Duplicate processing is a realistic possibility during distributed recovery.
Suppose the primary consumer processed:
event-1001
event-1002
event-1003
but the system had not fully recorded the corresponding processing state before failure.
The DR consumer may process one or more of those events again.
This is why business operations should be idempotent where possible.
For example:
Event ID:
ORDER-10091
First attempt:
Create order
Second attempt:
ORDER-10091 already processed
Result:
No duplicate order
Kafka infrastructure cannot automatically guarantee that every business operation is performed exactly once across an application-level disaster recovery event.
The application must be designed for the recovery model it uses.
Step 5: Simulate the Primary Region Failure
After validating the controlled consumer failover, perform a more realistic test.
Depending on your disaster recovery plan, simulate:
Application failure
Network failure
Consumer failure
Producer failure
Regional service unavailability
Do not immediately destroy infrastructure.
A controlled failure is safer and provides better diagnostic information.
For example, temporarily stop the application components that normally consume production events.
The goal is to reproduce the recovery condition without creating unnecessary infrastructure damage.
Step 6: Redirect Producers
If the primary Region is unavailable, producers also need a recovery path.
A common pattern is:
Application
|
v
Regional Kafka Endpoint
|
+---- Primary
|
+---- DR
During failover, the application configuration or service discovery layer needs to direct traffic toward the secondary Kafka environment.
The exact mechanism depends on your application architecture.
Do not hard-code a single Region-specific broker endpoint into every service.
Use configuration or service discovery that can be changed during recovery.
Step 7: Verify End-to-End Processing
Once the DR environment is active, generate new events.
For example:
event-2001
event-2002
event-2003
Verify the complete path:
Producer
|
v
DR MSK
|
v
DR Consumer
|
v
Business Service
|
v
Database
The final database or business system should reflect the events correctly.
This is an important distinction.
Kafka recovery is not successful if Kafka is receiving messages but the application cannot complete its business workflow.
Measure RTO
Record timestamps throughout the test.
For example:
10:00:00 Primary failure detected
10:03:00 DR procedure started
10:08:00 DR Kafka validated
10:12:00 Consumer started
10:15:00 First event processed
Recovery time can then be calculated from the defined start point to the point where the application resumes acceptable production processing.
If your target is:
RTO = 30 minutes
and recovery takes 15 minutes, the test meets the target.
If it takes 42 minutes, the architecture needs improvement.
Measure RPO
Now compare the source events with the destination events.
Suppose the last source event before failure was:
event-1050
and the last event confirmed in the DR environment was:
event-1042
Then events 1043 through 1050 require investigation.
The actual business RPO should be calculated from event timestamps and the replication state rather than simply counting event IDs.
Test Under Load
A failover test performed with ten events is useful for basic validation but does not represent production behavior.
Repeat the test under realistic load.
For example:
Normal:
100 events/sec
Failover:
100 events/sec
Peak:
1,000 events/sec
Measure:
Replication lag
Consumer throughput
Processing latency
CPU usage
Memory usage
Network throughput
Recovery time
The DR environment must be capable of handling the expected production workload.
Test Backlog Recovery
The secondary Region may have accumulated a backlog while the application was unavailable.
Measure how quickly the DR consumer can process it.
For example:
Backlog:
500,000 events
Processing rate:
5,000 events/sec
Estimated drain time:
100 seconds
This is only a simplified calculation. Real systems may have variable processing rates and downstream bottlenecks.
Monitor actual recovery behavior.
Test Application Dependencies
Kafka is rarely the only dependency.
Your DR application may also require:
Database
Cache
Secrets
Object storage
External APIs
Identity services
Monitoring
Configuration service
A Kafka failover test should therefore include these dependencies where they are part of the production recovery path.
Otherwise, you may discover during a real outage that Kafka recovered but the application did not.
Test Failback
Once Region A is healthy again, test returning production traffic.
A simplified sequence is:
DR Region Active
|
v
Validate Primary Region
|
v
Restore Replication
|
v
Synchronize State
|
v
Redirect Producers
|
v
Redirect Consumers
|
v
Validate Primary Processing
Do not immediately switch back just because the primary MSK cluster is available.
First verify:
Replication health
Topic availability
Consumer state
Application dependencies
Data consistency
Monitoring
Avoid Split-Brain Processing
One of the most dangerous failure scenarios is having both Regions actively process the same business events unintentionally.
For an active-passive architecture, enforce a clear ownership model.
For example:
Normal State:
Region A = ACTIVE
Region B = STANDBY
Failover:
Region A = STOPPED
Region B = ACTIVE
Failback:
Region B = STOPPED
Region A = ACTIVE
The exact implementation depends on the application, but the principle is important.
There should be a clear answer to:
Which Region is allowed to process production events?
Build an Automated Failover Test
Once manual testing is reliable, automate repeatable checks.
A test workflow can look like:
Start Test
|
v
Generate Events
|
v
Verify Replication
|
v
Simulate Primary Failure
|
v
Activate DR
|
v
Verify Consumers
|
v
Generate New Events
|
v
Validate Processing
|
v
Measure RPO / RTO
|
v
Restore Primary
|
v
Test Failback
|
v
Generate Report
Automation makes it possible to run recovery tests regularly instead of waiting for an annual disaster recovery exercise.
Troubleshooting Common Failover Problems
DR Consumer Cannot Connect
Check:
Topics Are Missing
Verify that the expected topics are included in the replication configuration and that replication has completed successfully.
Consumer Starts From the Wrong Position
Review the consumer group and replication behavior.
Do not assume the consumer automatically resumes from the exact application state expected by the business.
Duplicate Events Appear
Check consumer recovery timing and application idempotency.
Use unique event identifiers to detect duplicate processing.
Recovery Takes Too Long
Break the RTO into individual stages:
Detection
+
Decision
+
Traffic switch
+
Consumer startup
+
Backlog processing
+
Dependency recovery
The slowest stage usually identifies the best improvement opportunity.
Common Mistakes
Testing Only Replication
Seeing replicated events in the DR cluster does not prove application recovery.
Testing Only During Low Traffic
Recovery performance may be very different at production load.
Ignoring Consumer Offsets
Topic data and consumer state are separate concerns.
Forgetting Producer Recovery
Consumers may recover while producers continue sending events to the failed Region.
Skipping Duplicate Testing
Real failover conditions can expose duplicate-processing problems.
Never Testing Failback
A system may fail over correctly but fail during the return to the primary Region.
Using Destructive Tests First
Start with controlled failures before introducing more aggressive failure scenarios.
Best Practices
Define measurable RPO and RTO targets.
Establish normal replication and processing baselines.
Use identifiable test events.
Measure replication lag continuously.
Test both producer and consumer recovery.
Verify consumer offset behavior explicitly.
Design business operations for idempotent processing.
Test with realistic production load.
Include dependent services in recovery tests.
Measure backlog recovery time.
Prevent split-brain processing.
Test failback as well as failover.
Automate repeatable disaster recovery tests.
Document every recovery step and measured result.
Review the test results after every major infrastructure change.
Production Failover Test Report
After every test, produce a report similar to:
Metric | Target | Actual | Result |
|---|
Replication lag | < 5 min | 42 sec | Pass |
RPO | < 5 min | 1 min | Pass |
RTO | < 30 min | 17 min | Pass |
Duplicate events | 0 unhandled | 0 | Pass |
Failed processing | 0 | 0 | Pass |
Backlog recovery | < 10 min | 6 min | Pass |
The values above are examples.
The important part is to record objective results rather than simply writing “DR test successful.”
Production Checklist
Before declaring Kafka failover ready for production, verify:
[ ] Primary and DR MSK clusters are configured
[ ] MSK Replicator is operational
[ ] Required topics are replicated
[ ] Replication lag is monitored
[ ] RPO is defined
[ ] RTO is defined
[ ] Test events can be traced end to end
[ ] DR producer configuration is ready
[ ] DR consumer configuration is ready
[ ] Consumer offset behavior is understood
[ ] Duplicate processing is handled
[ ] Dependent services are available in DR
[ ] Backlog recovery has been tested
[ ] Failover has been tested under realistic load
[ ] Split-brain protection is implemented
[ ] Failback has been tested
[ ] Recovery runbook is documented
[ ] Test results are recorded
[ ] Recovery testing is repeated after major changes
Summary
MSK Replicator can keep Kafka data available across AWS Regions, but replication alone does not guarantee disaster recovery.
A real failover test must verify the complete application path. That includes replication lag, producer recovery, consumer startup, offset behavior, duplicate processing, downstream dependencies, backlog recovery, and traffic switching.
The most useful failover tests are measurable. Define RPO and RTO, generate identifiable events, record recovery timestamps, compare source and destination data, and test the workload under realistic traffic.
Once the manual recovery process is reliable, automate the test and run it regularly.
A Kafka disaster recovery architecture is ready for production only when the team can demonstrate that it can recover the workload—not merely that the secondary MSK cluster contains a copy of the data.
Join the conversation! Your thoughts help the community grow.