Distributed workflows rarely fail in one simple way.
A Lambda function may successfully call one service and then fail while calling the next. A downstream API may become unavailable for several minutes. A database transaction may time out. A worker may be interrupted after completing part of a business operation. An event may be delivered again after a timeout.
For a simple function, retrying the entire invocation may be enough. For a multi-step business process, it can become expensive and unreliable because previously completed work may be repeated.
This is where durable workflows become useful.
A durable workflow preserves enough execution state to resume work after failures instead of restarting the entire process from the beginning. For .NET teams building serverless applications, the interesting engineering question is not simply whether durable workflows recover from failures. It is how much recovery time, compute, storage, orchestration overhead, and operational complexity they introduce compared with conventional Lambda retries.
That question requires benchmarking.
This article presents a practical way to compare conventional Lambda retry-based recovery with durable workflow execution for .NET workloads. The focus is on measurable behavior: recovery latency, repeated work, throughput, failure handling, storage overhead, concurrency, and cost.
Why Durable Workflow Benchmarking Matters
Consider a five-step business process:
Create Order
|
v
Reserve Inventory
|
v
Process Payment
|
v
Create Shipment
|
v
Send Notification
Suppose the payment step fails.
With a conventional Lambda implementation, the entire function may be retried:
Create Order
Reserve Inventory
Process Payment -> Failure
Retry
Create Order
Reserve Inventory
Process Payment
Create Shipment
Send Notification
Some operations may execute twice.
A durable workflow can instead preserve the completed steps:
Create Order -> Completed
Reserve Inventory -> Completed
Process Payment -> Failed
|
v
Resume here
|
v
Create Shipment
Send Notification
This can reduce duplicated work, but it introduces orchestration and state-management overhead.
The benchmark needs to measure both sides.
Define the Two Architectures
Before measuring anything, create two implementations of the same business workflow.
Conventional Lambda Retry
The first implementation can use a normal Lambda handler.
public async Task Handler(
OrderRequest request,
CancellationToken cancellationToken)
{
await CreateOrderAsync(request, cancellationToken);
await ReserveInventoryAsync(request, cancellationToken);
await ProcessPaymentAsync(request, cancellationToken);
await CreateShipmentAsync(request, cancellationToken);
await SendNotificationAsync(request, cancellationToken);
}
If the function fails, the surrounding event or invocation mechanism may retry the Lambda.
The entire handler can therefore execute again.
Durable Workflow
The second implementation separates the workflow into recoverable steps.
Conceptually:
Workflow
|
+--> CreateOrder
|
+--> ReserveInventory
|
+--> ProcessPayment
|
+--> CreateShipment
|
+--> SendNotification
The orchestration layer maintains enough state to know which activities have completed.
The exact AWS implementation should be selected based on the application's requirements and the durable workflow capability being evaluated. The benchmark methodology remains the same: compare equivalent business workflows under identical failure conditions.
Define the Benchmark Questions
A useful benchmark should answer specific questions.
For example:
How long does a successful workflow take?
How long does recovery take after a transient failure?
How much work is repeated?
How many downstream requests are generated?
How much Lambda compute is consumed?
How much workflow state is stored?
How does performance change with concurrency?
What happens when multiple steps fail?
What happens when a failure lasts several minutes?
How does the system behave when a workflow is cancelled?
What is the operational cost of recovery?
These questions are more valuable than simply comparing average execution time.
Build a Representative Workflow
Avoid a benchmark containing five empty Lambda functions.
Each activity should perform realistic work.
For example:
Step 1: Insert order
Step 2: Reserve inventory
Step 3: Call payment service
Step 4: Create shipment record
Step 5: Publish notification
Each step should have measurable duration.
For example:
| Step | Typical Work |
|---|
| Create Order | PostgreSQL write |
| Reserve Inventory | Database transaction |
| Process Payment | External HTTP request |
| Create Shipment | Database write |
| Send Notification | Messaging/API call |
This creates a meaningful failure-recovery scenario.
Keep Business Semantics Identical
The conventional and durable implementations should perform the same logical operations.
If the traditional implementation performs:
Database
HTTP API
Database
Queue
the durable implementation should not replace those operations with mocks that execute instantly.
Otherwise, the benchmark measures different applications.
The workflow engine should be the primary variable.
Establish a Success Baseline
Before introducing failures, measure the successful path.
Run the workflow at:
1 concurrent workflow
10 concurrent workflows
50 concurrent workflows
100 concurrent workflows
500 concurrent workflows
The exact levels depend on the expected production load.
Measure:
This creates the baseline against which failure recovery can be compared.
Measure Recovery Latency Separately
Recovery latency is different from total workflow latency.
Suppose:
Workflow starts at 10:00:00
Payment fails at 10:00:03
Payment becomes available at 10:00:10
Workflow completes at 10:00:11
The recovery latency is approximately:
10:00:11 - 10:00:10 = 1 second
while the overall workflow duration is approximately 11 seconds.
These metrics answer different questions.
A durable workflow might have slightly higher successful-path overhead but significantly lower recovery cost after a late-stage failure.
Inject Controlled Failures
Failure injection is essential.
Do not wait for random production failures to determine whether the architecture recovers correctly.
Add controlled failure modes such as:
Fail step 1
Fail step 2
Fail step 3
Fail step 4
Fail step 5
Then repeat the same test multiple times.
For example:
Test A: Payment fails once
Test B: Payment fails twice
Test C: Payment unavailable for 30 seconds
Test D: Payment unavailable for 5 minutes
This exposes how the architecture behaves under different failure durations.
Compare Early and Late Failures
Failure position is an important benchmark dimension.
Consider:
Workflow A
Failure at step 1
Workflow B
Failure at step 3
Workflow C
Failure at step 5
A traditional retry becomes increasingly expensive as the failure moves later in the workflow because more work has already been completed.
If five steps take:
1s + 2s + 3s + 2s + 1s
and step 5 fails, restarting the workflow can repeat approximately nine seconds of previous work.
A durable workflow can potentially resume from the failed step.
This is one of the primary hypotheses the benchmark should test.
Measure Repeated Work
Create a metric for repeated business operations.
For example:
Expected database writes
Actual database writes
Expected HTTP requests
Actual HTTP requests
Suppose one workflow should produce:
5 database/API operations
but a failure causes:
8 operations
The additional three operations represent duplicated work.
A useful metric is:
Replay overhead =
(actual operations - expected operations)
/
expected operations
This gives a workload-specific measure of recovery efficiency.
Make Activities Idempotent
Durable workflows do not remove the need for idempotency.
An activity may be retried or replayed depending on the workflow implementation and failure mode.
For example:
public async Task ProcessPaymentAsync(
string orderId,
CancellationToken cancellationToken)
{
var existing = await paymentRepository
.FindByOrderIdAsync(orderId, cancellationToken);
if (existing is not null)
{
return;
}
await paymentGateway.ChargeAsync(
orderId,
cancellationToken);
}
The exact implementation should depend on the payment system.
The important principle is that repeating the same logical activity should not accidentally create duplicate business effects.
Separate Orchestration From Activity Work
A durable workflow should coordinate work rather than perform large amounts of business logic inside the orchestration layer.
A useful conceptual structure is:
Orchestrator
|
+--> Activity
|
+--> Activity
|
+--> Activity
The orchestrator determines what should happen.
Activities perform external work.
This separation makes benchmarking easier because orchestration overhead can be measured separately from actual business operations.
Benchmark State Size
Durable execution generally requires persistence of workflow state.
That state has a cost.
Test workflows with:
Small state
Medium state
Large state
For example:
| Workflow State | Example |
|---|
| Small | IDs and status |
| Medium | Order metadata |
| Large | Detailed processing context |
| Very large | Large serialized payload |
Avoid storing large response bodies or unnecessary objects in workflow state.
Instead, store references:
{
"orderId": "12345",
"paymentId": "pay-98765",
"status": "pending"
}
rather than embedding an entire external API response.
Smaller state generally makes workflow persistence and recovery easier to reason about.
Measure Orchestration Overhead
A successful durable workflow may take longer than a direct Lambda implementation because additional coordination is required.
That is not necessarily a problem.
Measure the difference.
For example:
| Metric | Direct Lambda | Durable Workflow |
|---|
| Success latency | Measure | Measure |
| Recovery latency | Measure | Measure |
| Compute duration | Measure | Measure |
| Downstream calls | Measure | Measure |
| State operations | N/A | Measure |
| Repeated work | Measure | Measure |
The important question is whether the additional orchestration overhead is justified by improved recovery behavior.
Benchmark Long Waits
Durable workflows become especially interesting when a workflow needs to wait.
For example:
Order created
|
v
Wait for approval
|
v
Process payment
|
v
Ship order
A traditional Lambda should generally not remain active while waiting for a human or external event.
A durable workflow can model the wait as part of the workflow rather than consuming Lambda execution time continuously.
Benchmark:
Wait: 1 minute
Wait: 10 minutes
Wait: 1 hour
Wait: several hours
Measure the cost and operational behavior.
The key metric is whether the workflow can wait without unnecessarily holding compute resources.
Test High Concurrency
A workflow architecture that works for 10 concurrent executions may behave differently at 10,000.
Gradually increase concurrency.
For example:
10
100
500
1,000
5,000
10,000
At each level, measure:
Do not increase concurrency indefinitely just to obtain a large number.
The benchmark should focus on the application's expected scale and a reasonable stress margin.
Test Dependency Failures Under Load
A particularly important scenario is:
1,000 workflows running
|
v
Payment API starts returning 503
Now compare the two architectures.
The conventional retry implementation may produce a large number of repeated payment requests.
A durable workflow can preserve completed state while waiting for recovery.
The benchmark should measure the resulting downstream traffic.
This is where retry amplification becomes important.
Compare Recovery After a Late Failure
Suppose the workflow contains:
A -> B -> C -> D -> E
and each step takes approximately two seconds.
A failure at E occurs after roughly eight seconds of successful work.
A traditional retry may repeat:
A -> B -> C -> D -> E
A durable approach can potentially resume from:
E
The difference becomes significant when:
Steps are expensive.
Steps make external API calls.
Steps perform database writes.
Steps have side effects.
Workflows contain many stages.
Failures occur late.
This should be one of the central benchmark scenarios.
Measure Failure Recovery Cost
Create a simple recovery-cost model:
Recovery cost =
additional compute
+ repeated downstream work
+ additional database operations
+ orchestration/state operations
+ recovery latency
You do not need to convert every component into money immediately.
First measure the physical resources.
Then apply the current AWS pricing applicable to your account, region, architecture, and services.
This keeps the benchmark independent of changing pricing assumptions.
Test Cancellation
Production workflows are sometimes cancelled.
For example:
Customer cancels order
Workflow no longer required
The benchmark should verify that cancellation stops unnecessary future work.
Test:
Cancel before step 1
Cancel during step 2
Cancel while waiting
Cancel before final step
Then verify that downstream operations do not continue unexpectedly.
Test Duplicate Events
Serverless systems often need to tolerate duplicate delivery.
Send the same logical workflow request multiple times:
Event: order-1001
Event: order-1001
Event: order-1001
The system should have a clear idempotency strategy.
Measure:
Logical workflows
Actual workflows
Database side effects
External API side effects
A durable workflow architecture should not be considered production-ready if duplicate events can produce duplicate business operations.
Test Partial Infrastructure Failure
Do not test only application exceptions.
Also simulate:
Database unavailable
External API unavailable
Network interruption
Lambda execution interruption
Temporary throttling
Workflow activity timeout
The purpose is to understand where recovery happens.
A useful failure matrix looks like this:
| Failure | Direct Lambda Retry | Durable Workflow |
|---|
| Step 1 failure | Restart | Resume/retry according to workflow |
| Step 3 failure | Repeat earlier work | Preserve completed state |
| Long dependency outage | Repeated retries | Durable waiting/recovery |
| Duplicate event | Application-dependent | Application-dependent |
| Lambda interruption | Invocation retry | Workflow recovery |
| Long human wait | Poor fit | Stronger fit |
The exact behavior depends on the implementation, but the matrix provides a consistent testing framework.
Common Benchmark Mistakes
Comparing Different Business Logic
If the durable implementation removes database operations that exist in the normal implementation, the benchmark is invalid.
Ignoring Idempotency
Durability does not automatically make side effects safe.
Measuring Only Successful Workflows
The primary value of durability appears during failure and recovery.
Ignoring Late Failures
Failing the first step does not demonstrate the benefit of preserving completed work.
Using Huge Workflow State
Large serialized state can increase storage and orchestration overhead.
Treating Orchestration Overhead as a Failure
A durable system may have some additional successful-path overhead. The benchmark should determine whether the recovery benefits justify it.
Running Only Low Concurrency
Serverless architectures should be evaluated under realistic concurrency.
Practical Benchmark Matrix
A comprehensive first-pass benchmark can use:
| Dimension | Test Values |
|---|
| Workflow steps | 3 / 5 / 10 |
| Concurrency | 1 / 10 / 100 / 1,000 |
| Failure position | First / middle / last |
| Failure duration | Immediate / 30s / 5m |
| Retry count | 0 / 1 / 3 |
| State size | Small / Medium / Large |
| Wait time | None / 1m / 10m / 1h |
| Dependency | Healthy / Slow / Unavailable |
| Event delivery | Single / Duplicate |
| Workload | Read / Write / Mixed |
For each scenario, record:
Completion rate
End-to-end latency
Recovery latency
Lambda duration
Repeated operations
Downstream requests
Database operations
State size
Concurrency
Throttling
Errors
How to Interpret the Results
Do not expect one architecture to win every category.
A direct Lambda implementation may have lower latency and less orchestration overhead for short, reliable workflows.
A durable workflow may become more attractive as workflows become longer, failures become more expensive, or waiting periods become longer.
For example:
Short workflow + low failure rate
|
v
Direct Lambda may be sufficient
Long workflow + expensive steps
|
v
Durability becomes more valuable
Long waits + external dependencies
|
v
Durable workflow becomes increasingly attractive
The benchmark should identify this boundary for your application.
Frequently Asked Questions
What is the main advantage of a durable workflow?
The primary advantage is preserving workflow progress so that recovery does not necessarily require repeating all previously completed work.
Are durable workflows always cheaper?
No. They introduce orchestration and state-management overhead. Their economic advantage depends on how expensive failures and repeated work are in the workload.
Do durable workflows eliminate retries?
No. Individual activities can still require retries. Durability and retry policies solve related but different problems.
Do activities need to be idempotent?
Yes. Any activity with external side effects should be designed carefully for retries, duplicate delivery, and recovery.
Should I benchmark only Lambda execution time?
No. Measure end-to-end workflow latency, repeated work, downstream requests, state operations, database activity, and recovery behavior.
When are durable workflows most useful?
They are particularly valuable for long-running, multi-step processes where restarting from the beginning would be expensive or unsafe, or where the workflow needs to wait for external events.
Should workflow state contain complete API responses?
Usually not. Store the minimum information required to resume the workflow and keep large payloads in appropriate external storage when necessary.
Conclusion
Durable workflows should not be evaluated simply by asking whether they are faster than ordinary Lambda execution. Their value appears when a workflow fails after completing meaningful work, when a dependency remains unavailable for an extended period, or when a process must wait without continuously consuming compute resources.
A fair .NET benchmark should implement the same business workflow using both conventional Lambda retry behavior and a durable orchestration approach. Run successful workflows first, then inject failures at different stages, measure recovery latency, track repeated operations, test duplicate events, evaluate long waits, and increase concurrency until the expected production boundary is reached.
The most useful result will not necessarily be a single performance number. It will show where the additional orchestration overhead of durability is justified by lower recovery cost, less duplicated work, and more predictable behavior during failures.
For short and highly reliable operations, conventional Lambda execution may remain the simpler choice. For long-running workflows with expensive steps, external dependencies, or meaningful failure-recovery requirements, durable execution can provide a stronger foundation. The benchmark should determine exactly where that trade-off makes sense for the application rather than assuming that one architecture is universally better.