Benchmarking AWS Lambda Durable Workflows for .NET Failure Recovery

Distributed workflows rarely fail in one simple way.

A Lambda function may successfully call one service and then fail while calling the next. A downstream API may become unavailable for several minutes. A database transaction may time out. A worker may be interrupted after completing part of a business operation. An event may be delivered again after a timeout.

For a simple function, retrying the entire invocation may be enough. For a multi-step business process, it can become expensive and unreliable because previously completed work may be repeated.

This is where durable workflows become useful.

A durable workflow preserves enough execution state to resume work after failures instead of restarting the entire process from the beginning. For .NET teams building serverless applications, the interesting engineering question is not simply whether durable workflows recover from failures. It is how much recovery time, compute, storage, orchestration overhead, and operational complexity they introduce compared with conventional Lambda retries.

That question requires benchmarking.

This article presents a practical way to compare conventional Lambda retry-based recovery with durable workflow execution for .NET workloads. The focus is on measurable behavior: recovery latency, repeated work, throughput, failure handling, storage overhead, concurrency, and cost.

Why Durable Workflow Benchmarking Matters

Consider a five-step business process:

Create Order
    |
    v
Reserve Inventory
    |
    v
Process Payment
    |
    v
Create Shipment
    |
    v
Send Notification

Suppose the payment step fails.

With a conventional Lambda implementation, the entire function may be retried:

Create Order
Reserve Inventory
Process Payment -> Failure

Retry
Create Order
Reserve Inventory
Process Payment
Create Shipment
Send Notification

Some operations may execute twice.

A durable workflow can instead preserve the completed steps:

Create Order       -> Completed
Reserve Inventory  -> Completed
Process Payment    -> Failed
                         |
                         v
                    Resume here
                         |
                         v
Create Shipment
Send Notification

This can reduce duplicated work, but it introduces orchestration and state-management overhead.

The benchmark needs to measure both sides.

Define the Two Architectures

Before measuring anything, create two implementations of the same business workflow.

Conventional Lambda Retry

The first implementation can use a normal Lambda handler.

public async Task Handler(
    OrderRequest request,
    CancellationToken cancellationToken)
{
    await CreateOrderAsync(request, cancellationToken);
    await ReserveInventoryAsync(request, cancellationToken);
    await ProcessPaymentAsync(request, cancellationToken);
    await CreateShipmentAsync(request, cancellationToken);
    await SendNotificationAsync(request, cancellationToken);
}

If the function fails, the surrounding event or invocation mechanism may retry the Lambda.

The entire handler can therefore execute again.

Durable Workflow

The second implementation separates the workflow into recoverable steps.

Conceptually:

Workflow
   |
   +--> CreateOrder
   |
   +--> ReserveInventory
   |
   +--> ProcessPayment
   |
   +--> CreateShipment
   |
   +--> SendNotification

The orchestration layer maintains enough state to know which activities have completed.

The exact AWS implementation should be selected based on the application's requirements and the durable workflow capability being evaluated. The benchmark methodology remains the same: compare equivalent business workflows under identical failure conditions.

Define the Benchmark Questions

A useful benchmark should answer specific questions.

For example:

How long does a successful workflow take?
How long does recovery take after a transient failure?
How much work is repeated?
How many downstream requests are generated?
How much Lambda compute is consumed?
How much workflow state is stored?
How does performance change with concurrency?
What happens when multiple steps fail?
What happens when a failure lasts several minutes?
How does the system behave when a workflow is cancelled?
What is the operational cost of recovery?

These questions are more valuable than simply comparing average execution time.

Build a Representative Workflow

Avoid a benchmark containing five empty Lambda functions.

Each activity should perform realistic work.

For example:

Step 1: Insert order
Step 2: Reserve inventory
Step 3: Call payment service
Step 4: Create shipment record
Step 5: Publish notification

Each step should have measurable duration.

For example:

Step	Typical Work
Create Order	PostgreSQL write
Reserve Inventory	Database transaction
Process Payment	External HTTP request
Create Shipment	Database write
Send Notification	Messaging/API call

This creates a meaningful failure-recovery scenario.

Keep Business Semantics Identical

The conventional and durable implementations should perform the same logical operations.

If the traditional implementation performs:

Database
HTTP API
Database
Queue

the durable implementation should not replace those operations with mocks that execute instantly.

Otherwise, the benchmark measures different applications.

The workflow engine should be the primary variable.

Establish a Success Baseline

Before introducing failures, measure the successful path.

Run the workflow at:

1 concurrent workflow
10 concurrent workflows
50 concurrent workflows
100 concurrent workflows
500 concurrent workflows

The exact levels depend on the expected production load.

Measure:

End-to-end latency.
Lambda execution duration.
Activity duration.
Workflow completion rate.
Requests per second.
Database operations.
External API requests.
Memory usage.
Error rate.

This creates the baseline against which failure recovery can be compared.

Measure Recovery Latency Separately

Recovery latency is different from total workflow latency.

Suppose:

Workflow starts at 10:00:00
Payment fails at 10:00:03
Payment becomes available at 10:00:10
Workflow completes at 10:00:11

The recovery latency is approximately:

10:00:11 - 10:00:10 = 1 second

while the overall workflow duration is approximately 11 seconds.

These metrics answer different questions.

A durable workflow might have slightly higher successful-path overhead but significantly lower recovery cost after a late-stage failure.

Inject Controlled Failures

Failure injection is essential.

Do not wait for random production failures to determine whether the architecture recovers correctly.

Add controlled failure modes such as:

Fail step 1
Fail step 2
Fail step 3
Fail step 4
Fail step 5

Then repeat the same test multiple times.

For example:

Test A: Payment fails once
Test B: Payment fails twice
Test C: Payment unavailable for 30 seconds
Test D: Payment unavailable for 5 minutes

This exposes how the architecture behaves under different failure durations.

Compare Early and Late Failures

Failure position is an important benchmark dimension.

Consider:

Workflow A
Failure at step 1

Workflow B
Failure at step 3

Workflow C
Failure at step 5

A traditional retry becomes increasingly expensive as the failure moves later in the workflow because more work has already been completed.

If five steps take:

1s + 2s + 3s + 2s + 1s

and step 5 fails, restarting the workflow can repeat approximately nine seconds of previous work.

A durable workflow can potentially resume from the failed step.

This is one of the primary hypotheses the benchmark should test.

Measure Repeated Work

Create a metric for repeated business operations.

For example:

Expected database writes
Actual database writes
Expected HTTP requests
Actual HTTP requests

Suppose one workflow should produce:

5 database/API operations

but a failure causes:

8 operations

The additional three operations represent duplicated work.

A useful metric is:

Replay overhead =
(actual operations - expected operations)
/
expected operations

This gives a workload-specific measure of recovery efficiency.

Make Activities Idempotent

Durable workflows do not remove the need for idempotency.

An activity may be retried or replayed depending on the workflow implementation and failure mode.

For example:

public async Task ProcessPaymentAsync(
    string orderId,
    CancellationToken cancellationToken)
{
    var existing = await paymentRepository
        .FindByOrderIdAsync(orderId, cancellationToken);

    if (existing is not null)
    {
        return;
    }

    await paymentGateway.ChargeAsync(
        orderId,
        cancellationToken);
}

The exact implementation should depend on the payment system.

The important principle is that repeating the same logical activity should not accidentally create duplicate business effects.

Separate Orchestration From Activity Work

A durable workflow should coordinate work rather than perform large amounts of business logic inside the orchestration layer.

A useful conceptual structure is:

Orchestrator
    |
    +--> Activity
    |
    +--> Activity
    |
    +--> Activity

The orchestrator determines what should happen.

Activities perform external work.

This separation makes benchmarking easier because orchestration overhead can be measured separately from actual business operations.

Benchmark State Size

Durable execution generally requires persistence of workflow state.

That state has a cost.

Test workflows with:

Small state
Medium state
Large state

For example:

Workflow State	Example
Small	IDs and status
Medium	Order metadata
Large	Detailed processing context
Very large	Large serialized payload

Avoid storing large response bodies or unnecessary objects in workflow state.

Instead, store references:

{
  "orderId": "12345",
  "paymentId": "pay-98765",
  "status": "pending"
}

rather than embedding an entire external API response.

Smaller state generally makes workflow persistence and recovery easier to reason about.

Measure Orchestration Overhead

A successful durable workflow may take longer than a direct Lambda implementation because additional coordination is required.

That is not necessarily a problem.

Measure the difference.

For example:

Metric	Direct Lambda	Durable Workflow
Success latency	Measure	Measure
Recovery latency	Measure	Measure
Compute duration	Measure	Measure
Downstream calls	Measure	Measure
State operations	N/A	Measure
Repeated work	Measure	Measure

The important question is whether the additional orchestration overhead is justified by improved recovery behavior.

Benchmark Long Waits

Durable workflows become especially interesting when a workflow needs to wait.

For example:

Order created
    |
    v
Wait for approval
    |
    v
Process payment
    |
    v
Ship order

A traditional Lambda should generally not remain active while waiting for a human or external event.

A durable workflow can model the wait as part of the workflow rather than consuming Lambda execution time continuously.

Benchmark:

Wait: 1 minute
Wait: 10 minutes
Wait: 1 hour
Wait: several hours

Measure the cost and operational behavior.

The key metric is whether the workflow can wait without unnecessarily holding compute resources.

Test High Concurrency

A workflow architecture that works for 10 concurrent executions may behave differently at 10,000.

Gradually increase concurrency.

For example:

At each level, measure:

Completion rate.
End-to-end latency.
Activity latency.
Queue depth where applicable.
Database connections.
External API requests.
Throttling.
Lambda concurrency.
Workflow failures.

Do not increase concurrency indefinitely just to obtain a large number.

The benchmark should focus on the application's expected scale and a reasonable stress margin.

Test Dependency Failures Under Load

A particularly important scenario is:

1,000 workflows running
        |
        v
Payment API starts returning 503

Now compare the two architectures.

The conventional retry implementation may produce a large number of repeated payment requests.

A durable workflow can preserve completed state while waiting for recovery.

The benchmark should measure the resulting downstream traffic.

This is where retry amplification becomes important.

Compare Recovery After a Late Failure

Suppose the workflow contains:

A -> B -> C -> D -> E

and each step takes approximately two seconds.

A failure at E occurs after roughly eight seconds of successful work.

A traditional retry may repeat:

A -> B -> C -> D -> E

A durable approach can potentially resume from:

The difference becomes significant when:

Steps are expensive.
Steps make external API calls.
Steps perform database writes.
Steps have side effects.
Workflows contain many stages.
Failures occur late.

This should be one of the central benchmark scenarios.

Measure Failure Recovery Cost

Create a simple recovery-cost model:

Recovery cost =
additional compute
+ repeated downstream work
+ additional database operations
+ orchestration/state operations
+ recovery latency

You do not need to convert every component into money immediately.

First measure the physical resources.

Then apply the current AWS pricing applicable to your account, region, architecture, and services.

This keeps the benchmark independent of changing pricing assumptions.

Test Cancellation

Production workflows are sometimes cancelled.

For example:

Customer cancels order
Workflow no longer required

The benchmark should verify that cancellation stops unnecessary future work.

Test:

Cancel before step 1
Cancel during step 2
Cancel while waiting
Cancel before final step

Then verify that downstream operations do not continue unexpectedly.

Test Duplicate Events

Serverless systems often need to tolerate duplicate delivery.

Send the same logical workflow request multiple times:

Event: order-1001
Event: order-1001
Event: order-1001

The system should have a clear idempotency strategy.

Measure:

Logical workflows
Actual workflows
Database side effects
External API side effects

A durable workflow architecture should not be considered production-ready if duplicate events can produce duplicate business operations.

Test Partial Infrastructure Failure

Do not test only application exceptions.

Also simulate:

Database unavailable
External API unavailable
Network interruption
Lambda execution interruption
Temporary throttling
Workflow activity timeout

The purpose is to understand where recovery happens.

A useful failure matrix looks like this:

Failure	Direct Lambda Retry	Durable Workflow
Step 1 failure	Restart	Resume/retry according to workflow
Step 3 failure	Repeat earlier work	Preserve completed state
Long dependency outage	Repeated retries	Durable waiting/recovery
Duplicate event	Application-dependent	Application-dependent
Lambda interruption	Invocation retry	Workflow recovery
Long human wait	Poor fit	Stronger fit

The exact behavior depends on the implementation, but the matrix provides a consistent testing framework.

Common Benchmark Mistakes

Comparing Different Business Logic

If the durable implementation removes database operations that exist in the normal implementation, the benchmark is invalid.

Ignoring Idempotency

Durability does not automatically make side effects safe.

Measuring Only Successful Workflows

The primary value of durability appears during failure and recovery.

Ignoring Late Failures

Failing the first step does not demonstrate the benefit of preserving completed work.

Using Huge Workflow State

Large serialized state can increase storage and orchestration overhead.

Treating Orchestration Overhead as a Failure

A durable system may have some additional successful-path overhead. The benchmark should determine whether the recovery benefits justify it.

Running Only Low Concurrency

Serverless architectures should be evaluated under realistic concurrency.

Practical Benchmark Matrix

A comprehensive first-pass benchmark can use:

Dimension	Test Values
Workflow steps	3 / 5 / 10
Concurrency	1 / 10 / 100 / 1,000
Failure position	First / middle / last
Failure duration	Immediate / 30s / 5m
Retry count	0 / 1 / 3
State size	Small / Medium / Large
Wait time	None / 1m / 10m / 1h
Dependency	Healthy / Slow / Unavailable
Event delivery	Single / Duplicate
Workload	Read / Write / Mixed

For each scenario, record:

Completion rate
End-to-end latency
Recovery latency
Lambda duration
Repeated operations
Downstream requests
Database operations
State size
Concurrency
Throttling
Errors

How to Interpret the Results

Do not expect one architecture to win every category.

A direct Lambda implementation may have lower latency and less orchestration overhead for short, reliable workflows.

A durable workflow may become more attractive as workflows become longer, failures become more expensive, or waiting periods become longer.

For example:

Short workflow + low failure rate
        |
        v
Direct Lambda may be sufficient

Long workflow + expensive steps
        |
        v
Durability becomes more valuable

Long waits + external dependencies
        |
        v
Durable workflow becomes increasingly attractive

The benchmark should identify this boundary for your application.

Frequently Asked Questions

What is the main advantage of a durable workflow?

The primary advantage is preserving workflow progress so that recovery does not necessarily require repeating all previously completed work.

Are durable workflows always cheaper?

No. They introduce orchestration and state-management overhead. Their economic advantage depends on how expensive failures and repeated work are in the workload.

Do durable workflows eliminate retries?

No. Individual activities can still require retries. Durability and retry policies solve related but different problems.

Do activities need to be idempotent?

Yes. Any activity with external side effects should be designed carefully for retries, duplicate delivery, and recovery.

Should I benchmark only Lambda execution time?

No. Measure end-to-end workflow latency, repeated work, downstream requests, state operations, database activity, and recovery behavior.

When are durable workflows most useful?

They are particularly valuable for long-running, multi-step processes where restarting from the beginning would be expensive or unsafe, or where the workflow needs to wait for external events.

Should workflow state contain complete API responses?

Usually not. Store the minimum information required to resume the workflow and keep large payloads in appropriate external storage when necessary.

Conclusion

Durable workflows should not be evaluated simply by asking whether they are faster than ordinary Lambda execution. Their value appears when a workflow fails after completing meaningful work, when a dependency remains unavailable for an extended period, or when a process must wait without continuously consuming compute resources.

A fair .NET benchmark should implement the same business workflow using both conventional Lambda retry behavior and a durable orchestration approach. Run successful workflows first, then inject failures at different stages, measure recovery latency, track repeated operations, test duplicate events, evaluate long waits, and increase concurrency until the expected production boundary is reached.

The most useful result will not necessarily be a single performance number. It will show where the additional orchestration overhead of durability is justified by lower recovery cost, less duplicated work, and more predictable behavior during failures.

For short and highly reliable operations, conventional Lambda execution may remain the simpler choice. For long-running workflows with expensive steps, external dependencies, or meaningful failure-recovery requirements, durable execution can provide a stronger foundation. The benchmark should determine exactly where that trade-off makes sense for the application rather than assuming that one architecture is universally better.