Imagine this scenario.
You have built an IoT fleet management platform. An operator updates a device configuration through a FastAPI endpoint.
The API returns:
HTTP/1.1 200 OK
The database contains the updated configuration. Everything appears successful.
But the dashboard still displays the old configuration. The device communication service hasn't received the update either.
What happened?
The problem might not be in your API, database query, or frontend. It could be a fundamental distributed-systems problem known as the Dual-Write Problem.
This is one of those engineering issues that often remains invisible during development but becomes painful when an application starts handling real production workloads.
Let's understand why it happens and how the Transactional Outbox Pattern can help solve it.
1. The Problem With Directly Publishing Events
In a typical event-driven backend, an API often performs two operations:
- Updates a database record.
- Publishes an event to a message broker such as RabbitMQ.
For example:
@app.put("/devices/{device_id}/configuration")
async def update_configuration(
device_id: int,
configuration: dict
):
device = await update_device_in_database(
device_id,
configuration
)
await rabbitmq.publish(
"device.config.updated",
{
"device_id": device_id,
"configuration": configuration
}
)
return {"message": "Configuration updated"}
At first glance, this implementation looks perfectly reasonable.
However, the database and RabbitMQ are two independent systems. They do not automatically participate in the same atomic transaction.
Consider what happens when the database update succeeds but RabbitMQ becomes unavailable.
- The database commits the new configuration.
- RabbitMQ fails to receive the event.
- The API encounters an exception or returns an incomplete result.
- Downstream consumers never learn about the change.
Now consider the opposite approach: publishing the event before committing the database transaction.
The broker receives the event, but the database transaction subsequently fails.
Your consumers are now processing an update that was never successfully saved.
This is the dual-write problem.
The fundamental issue: Two independent operations represent one logical business action, but there is no guarantee that both will succeed together.
2. Introducing the Transactional Outbox Pattern
The Transactional Outbox Pattern provides a more reliable way to coordinate database changes and event publication.
Instead of publishing directly to the message broker from the API, we first store the event in an outbox table.
The important part is that the business database update and the outbox record are committed within the same database transaction.
For example, consider two tables:
devices
id | configuration |
|---|
101 | {"interval": 30} |
outbox_events
event_id | event_type | status |
|---|
UUID | DEVICE_CONFIG_UPDATED | PENDING |
When a device configuration changes, both records are updated in one transaction.
If the transaction commits, the configuration change and its corresponding event record are guaranteed to exist together.
If it rolls back, neither change is persisted.
A separate background worker is responsible for reading pending events and publishing them to RabbitMQ.
This separates the responsibilities:
- API: Handles business operations.
- Database: Stores business data and pending events.
- Background worker: Publishes events.
- Message broker: Distributes events to consumers.
The API no longer depends on RabbitMQ being available at the exact moment an operator updates a device.
3. Implementing the Pattern With FastAPI and PostgreSQL
Let's implement a simplified version using FastAPI, SQLAlchemy, and PostgreSQL.
First, define an outbox model.
import uuid
from sqlalchemy import (
Column,
String,
JSON,
DateTime,
Integer,
func
)
from sqlalchemy.dialects.postgresql import UUID
from database import Base
class OutboxEvent(Base):
__tablename__ = "outbox_events"
id = Column(
UUID(as_uuid=True),
primary_key=True,
default=uuid.uuid4
)
event_type = Column(String, nullable=False)
aggregate_id = Column(String, nullable=False)
payload = Column(JSON, nullable=False)
status = Column(
String,
nullable=False,
default="PENDING"
)
attempts = Column(
Integer,
nullable=False,
default=0
)
created_at = Column(
DateTime(timezone=True),
server_default=func.now()
)
published_at = Column(
DateTime(timezone=True),
nullable=True
)
The outbox table stores events that still need to be published.
Now let's update the API endpoint.
@app.put("/devices/{device_id}/configuration")
def update_configuration(
device_id: int,
configuration: dict,
session: Session = Depends(get_db)
):
with session.begin():
device = session.get(Device, device_id)
if not device:
raise HTTPException(
status_code=404,
detail="Device not found"
)
device.configuration = configuration
event = OutboxEvent(
event_type="DEVICE_CONFIG_UPDATED",
aggregate_id=str(device_id),
payload={
"device_id": device_id,
"configuration": configuration
}
)
session.add(event)
return {
"message": "Configuration updated",
"device_id": device_id
}
Notice something important.
There is no direct RabbitMQ publication inside this endpoint.
The database transaction handles both operations:
- Updating the device configuration.
- Inserting the corresponding outbox event.
If either operation fails, the transaction rolls back.
This gives us a durable record of the event that needs to be published.
For simplicity, this example uses synchronous SQLAlchemy. The same architectural approach works with asynchronous SQLAlchemy sessions.
4. Building the Background Event Publisher
Now we need a worker that continuously checks the outbox table and publishes pending events.
A simplified implementation looks like this:
def publish_pending_events(session, rabbitmq):
events = (
session.query(OutboxEvent)
.filter(
OutboxEvent.status == "PENDING"
)
.order_by(
OutboxEvent.created_at
)
.limit(100)
.all()
)
for event in events:
try:
rabbitmq.publish(
routing_key=event.event_type,
message={
"event_id": str(event.id),
"event_type": event.event_type,
"payload": event.payload
}
)
event.status = "PUBLISHED"
event.published_at = datetime.now(
timezone.utc
)
except Exception:
event.attempts += 1
session.commit()
This illustrates the basic publishing workflow, but it is not yet production-safe.
A production implementation needs additional protections.
First: Concurrent workers.
If multiple workers process the same pending events, duplicate publications can occur.
PostgreSQL provides FOR UPDATE SKIP LOCKED, which can help workers claim different events without waiting for one another.
Second: Broker acknowledgements.
Do not mark an event as published simply because a publish method returned successfully.
The publisher should use the broker's appropriate publisher-confirmation mechanism and mark the event as published only after receiving confirmation.
Third: Crash recovery.
A worker could publish an event successfully and crash before updating its database status.
When it restarts, it might publish the same event again.
This is an expected failure scenario, not an exceptional edge case.
Fourth: Retry management.
Temporary broker failures should trigger retries with controlled backoff rather than unlimited rapid retries.
A production system should also monitor failed events and provide a dead-letter or manual recovery mechanism.
5. The Part Developers Often Miss: Duplicate Event Delivery
One common misconception is that the Transactional Outbox Pattern guarantees exactly-once event delivery.
It does not.
Consider this sequence:
- Worker reads an event from PostgreSQL.
- Worker publishes the event to RabbitMQ.
- RabbitMQ confirms successful publication.
- Worker crashes before updating the outbox record.
- Worker restarts and publishes the same event again.
The consumer receives a duplicate.
This is why event-driven systems should be designed around at-least-once delivery and idempotent processing.
A practical solution is to assign every event a unique identifier.
For example:
{
"event_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
"event_type": "DEVICE_CONFIG_UPDATED",
"payload": {
"device_id": 101,
"configuration": {
"interval": 30
}
}
}
The consumer can maintain a table of processed event identifiers.
Before processing an event, it checks whether the identifier has already been handled.
Conceptually:
def process_event(event, session):
with session.begin():
if is_event_processed(
session,
event["event_id"]
):
return
apply_business_operation(
session,
event["payload"]
)
mark_event_as_processed(
session,
event["event_id"]
)
The processed-event record and business operation should be committed in the same consumer-side database transaction.
This prevents duplicate deliveries from repeating the same database-side business operation.
However, external side effects such as HTTP calls or physical device commands need their own idempotency strategy.
The important lesson is that reliable event processing requires more than a reliable publisher.
It requires both sides of the communication to handle failures.
6. When Should You Use the Transactional Outbox Pattern?
The Transactional Outbox Pattern is especially useful when a backend must maintain consistency between its database and asynchronous consumers.
Some practical scenarios include:
- IoT device configuration updates.
- Order processing and inventory synchronization.
- Payment status notifications.
- User activity and audit events.
- Microservice communication.
- Real-time dashboard updates.
- Background job scheduling.
However, it is not necessary for every application.
For a simple CRUD application where no downstream event needs to be published, introducing an outbox table and background worker may add unnecessary complexity.
There are also alternatives worth considering.
Direct message publishing: Simpler, but vulnerable to dual-write failures.
Transactional Outbox: Keeps business changes and event records atomic within one database transaction. A separate publisher handles delivery.
Change Data Capture (CDC): Tools such as Debezium can capture committed database changes from the transaction log and route outbox events to messaging infrastructure.
CDC can be useful at larger scales, but it introduces additional infrastructure and operational complexity.
The right choice depends on reliability requirements, throughput, infrastructure, and operational maturity.
Conclusion
A successful HTTP response does not necessarily mean every part of a distributed system has successfully completed the requested operation.
Your database might be correct while your message broker, dashboard, or downstream service remains unaware of the change.
The Transactional Outbox Pattern addresses this by making event creation part of the same database transaction as the business operation.
It does not eliminate every distributed-systems problem. Duplicate delivery, retries, ordering, monitoring, and idempotency still need careful engineering.
But it provides a much stronger foundation for building reliable event-driven applications.
One engineering principle worth remembering:
A system should not depend on every external service being available at the exact moment a business operation happens. It should be designed to recover when those services eventually become available again.
That is the difference between an application that works under ideal conditions and a backend designed for production.
Technologies covered: Python, FastAPI, SQLAlchemy, PostgreSQL, RabbitMQ, Event-Driven Architecture, Transactional Outbox Pattern, Distributed Systems.
Suggested tags: Backend Development, Python, Distributed Systems, Software Architecture, Microservices, PostgreSQL, RabbitMQ, FastAPI, System Design.
Join the conversation! Your thoughts help the community grow.