Saving Long-Running .NET Workflows When an Application Restarts

Ananya Desai

Introduction

Imagine an application that processes customer orders. After an order is placed, the application needs to verify payment, generate an invoice, update inventory, and send a confirmation email. Each operation may depend on the previous one, and the entire process can take several minutes.

Now consider what happens if the application restarts while processing the order.

If the workflow exists only in application memory, its progress can be lost. The customer may have already paid, but the application might restart before generating the invoice. When the application starts again, it may not know which operations were completed and which still need to run.

This problem is common in applications that handle payment processing, document generation, report creation, data imports, and background jobs.

A simple BackgroundService can execute long-running operations, but it does not automatically preserve their progress when the application stops. To handle restarts safely, the application needs to store workflow state outside its process memory.

In this article, we will build a basic persistent workflow in .NET using a database and a background worker. We will save each step's status, recover unfinished workflows after an application restart, and prevent completed operations from running again unnecessarily.

Why Long-Running Workflows Need Persistence

A normal background task runs inside the application process. It can use memory, dependency injection, and other application services while it is running.

However, application memory does not survive a process restart.

Consider the following implementation:

public class OrderProcessor
{
    public async Task ProcessAsync(int orderId)
    {
        await VerifyPayment(orderId);

        await GenerateInvoice(orderId);

        await SendConfirmation(orderId);
    }
}

This code works when everything completes successfully. But suppose the application stops after payment verification and before invoice generation.

When the application starts again, there is no built-in mechanism to know that payment verification was already completed.

Running the entire method again may create duplicate operations. Skipping the order may leave it incomplete.

The solution is to save the workflow's progress in persistent storage.

Instead of storing only the order ID, the application should maintain information such as:

Current workflow status.
Completed steps.
Pending steps.
Failed operations.
Last execution time.
Retry information.
Error details.

With this information, the application can identify unfinished work after restarting.

Understanding Persistent Workflow Execution

A persistent workflow separates business progress from the lifetime of the application process.

The basic execution flow looks like this:

New Order
    |
    v
Create Workflow Record
    |
    v
Verify Payment
    |
    v
Save Step Status
    |
    v
Generate Invoice
    |
    v
Save Step Status
    |
    v
Send Confirmation
    |
    v
Mark Workflow Completed

If the application stops during any step, the next application instance can load the workflow record and continue from the appropriate point.

For this example, we will use Entity Framework Core with a relational database.

The database will store workflow information, while a .NET BackgroundService will execute pending workflows.

Creating the Workflow Data Model

Start by creating an entity that represents a workflow.

public enum WorkflowStatus
{
    Pending,
    Running,
    Completed,
    Failed
}

public class OrderWorkflow
{
    public Guid Id { get; set; }

    public int OrderId { get; set; }

    public WorkflowStatus Status { get; set; }

    public bool PaymentVerified { get; set; }

    public bool InvoiceGenerated { get; set; }

    public bool ConfirmationSent { get; set; }

    public DateTime CreatedAt { get; set; }

    public DateTime UpdatedAt { get; set; }

    public string? LastError { get; set; }
}

Each Boolean property records whether an individual operation has completed.

For a small workflow, this structure is easy to understand. However, a larger application may have dozens of workflow steps. In that situation, storing every step as a separate property becomes difficult to maintain.

A separate workflow-step table is usually a better choice for complex processes.

For this example, we will keep the model simple so that the recovery process is easier to follow.

Next, create the database context:

using Microsoft.EntityFrameworkCore;

public class ApplicationDbContext : DbContext
{
    public ApplicationDbContext(
        DbContextOptions<ApplicationDbContext> options)
        : base(options)
    {
    }

    public DbSet<OrderWorkflow> OrderWorkflows =>
        Set<OrderWorkflow>();
}

builder.Services.AddDbContext<ApplicationDbContext>(
    options =>
        options.UseSqlServer(
            builder.Configuration.GetConnectionString("DefaultConnection")
        )
);

The database connection string should be stored in configuration or a secure secrets provider rather than hardcoded in the application.

Create and apply an EF Core migration to generate the workflow table.

dotnet ef migrations add CreateOrderWorkflow
dotnet ef database update

Now the application has a place to store workflow progress independently of its running process.

Creating a Workflow When an Order Is Received

When an order is created, save its workflow record before starting background processing.

public class OrderWorkflowService
{
    private readonly ApplicationDbContext _dbContext;

    public OrderWorkflowService(
        ApplicationDbContext dbContext)
    {
        _dbContext = dbContext;
    }

    public async Task<Guid> CreateWorkflowAsync(
        int orderId,
        CancellationToken cancellationToken)
    {
        var workflow = new OrderWorkflow
        {
            Id = Guid.NewGuid(),
            OrderId = orderId,
            Status = WorkflowStatus.Pending,
            CreatedAt = DateTime.UtcNow,
            UpdatedAt = DateTime.UtcNow
        };

        _dbContext.OrderWorkflows.Add(workflow);

        await _dbContext.SaveChangesAsync(
            cancellationToken);

        return workflow.Id;
    }
}

The method saves the workflow before returning its identifier.

This is important because the application should not depend on an in-memory task to remember that an order needs processing.

If the process stops immediately after the record is saved, the background worker can still find the pending workflow after the next startup.

Processing Workflows With BackgroundService

A BackgroundService provides a convenient way to execute background operations inside a .NET application.

Create a worker:

using Microsoft.EntityFrameworkCore;

public class OrderWorkflowWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<OrderWorkflowWorker> _logger;

    public OrderWorkflowWorker(
        IServiceScopeFactory scopeFactory,
        ILogger<OrderWorkflowWorker> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                using var scope =
                    _scopeFactory.CreateScope();

                var dbContext = scope.ServiceProvider
                    .GetRequiredService<ApplicationDbContext>();

                var workflow = await dbContext.OrderWorkflows
                    .Where(x =>
                        x.Status == WorkflowStatus.Pending)
                    .OrderBy(x => x.CreatedAt)
                    .FirstOrDefaultAsync(stoppingToken);

                if (workflow is null)
                {
                    await Task.Delay(
                        TimeSpan.FromSeconds(5),
                        stoppingToken);

                    continue;
                }

                await ProcessWorkflowAsync(
                    workflow.Id,
                    stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(
                    ex,
                    "An error occurred while processing workflows.");

                await Task.Delay(
                    TimeSpan.FromSeconds(5),
                    stoppingToken);
            }
        }
    }

    private async Task ProcessWorkflowAsync(
        Guid workflowId,
        CancellationToken cancellationToken)
    {
        using var scope = _scopeFactory.CreateScope();

        var processor = scope.ServiceProvider
            .GetRequiredService<OrderWorkflowProcessor>();

        await processor.ProcessAsync(
            workflowId,
            cancellationToken);
    }
}

The worker periodically checks the database for pending workflows.

Notice that the worker creates a dependency injection scope before resolving the database context. This is necessary because BackgroundService is registered as a singleton, while DbContext is normally scoped.

The worker also uses the stopping token so that the application can request cancellation during shutdown.

builder.Services.AddScoped<OrderWorkflowProcessor>();

builder.Services.AddHostedService<OrderWorkflowWorker>();

This implementation demonstrates basic background processing. It is suitable for understanding the workflow structure, but a production application needs additional coordination when multiple worker instances can process the same database.

Saving Progress After Every Step

The processor should save progress after each successful operation.

public class OrderWorkflowProcessor
{
    private readonly ApplicationDbContext _dbContext;
    private readonly IOrderService _orderService;

    public OrderWorkflowProcessor(
        ApplicationDbContext dbContext,
        IOrderService orderService)
    {
        _dbContext = dbContext;
        _orderService = orderService;
    }

    public async Task ProcessAsync(
        Guid workflowId,
        CancellationToken cancellationToken)
    {
        var workflow = await _dbContext.OrderWorkflows
            .FirstOrDefaultAsync(
                x => x.Id == workflowId,
                cancellationToken);

        if (workflow is null)
            return;

        try
        {
            workflow.Status = WorkflowStatus.Running;

            if (!workflow.PaymentVerified)
            {
                await _orderService.VerifyPaymentAsync(
                    workflow.OrderId,
                    cancellationToken);

                workflow.PaymentVerified = true;

                await SaveProgressAsync(
                    workflow,
                    cancellationToken);
            }

            if (!workflow.InvoiceGenerated)
            {
                await _orderService.GenerateInvoiceAsync(
                    workflow.OrderId,
                    cancellationToken);

                workflow.InvoiceGenerated = true;

                await SaveProgressAsync(
                    workflow,
                    cancellationToken);
            }

            if (!workflow.ConfirmationSent)
            {
                await _orderService.SendConfirmationAsync(
                    workflow.OrderId,
                    cancellationToken);

                workflow.ConfirmationSent = true;

                await SaveProgressAsync(
                    workflow,
                    cancellationToken);
            }

            workflow.Status = WorkflowStatus.Completed;
            workflow.UpdatedAt = DateTime.UtcNow;

            await _dbContext.SaveChangesAsync(
                cancellationToken);
        }
        catch (Exception ex)
        {
            workflow.Status = WorkflowStatus.Failed;
            workflow.LastError = ex.Message;
            workflow.UpdatedAt = DateTime.UtcNow;

            await _dbContext.SaveChangesAsync(
                cancellationToken);

            throw;
        }
    }

    private async Task SaveProgressAsync(
        OrderWorkflow workflow,
        CancellationToken cancellationToken)
    {
        workflow.UpdatedAt = DateTime.UtcNow;

        await _dbContext.SaveChangesAsync(
            cancellationToken);
    }
}

The important part is that the application saves the workflow state after each completed operation.

For example, if payment verification and invoice generation finish successfully but the application stops before sending the confirmation, the database still contains both completed flags.

However, this code is an introductory implementation, not a complete production-ready workflow engine. In particular, marking an operation as complete after an external call leaves a small failure window. If the external operation succeeds but the application stops before saving the flag, that operation may run again after recovery.

This is why external operations must also support idempotency.

Preventing Duplicate Operations During Recovery

Saving progress does not guarantee that an operation will execute only once.

Consider payment verification. The payment provider may successfully process a request, but the application may restart before updating the workflow record.

When processing resumes, the application cannot know from its database alone whether the external operation succeeded.

One solution is to use an idempotency key.

For example:

public async Task VerifyPaymentAsync(
    int orderId,
    CancellationToken cancellationToken)
{
    var idempotencyKey = $"payment-order-{orderId}";

    await _paymentClient.VerifyAsync(
        orderId,
        idempotencyKey,
        cancellationToken);
}

The payment integration should use this key according to the provider's idempotency support.

Repeated requests with the same key should not create duplicate payment operations.

The same principle applies to invoice generation and confirmation delivery. Use unique business identifiers, database constraints, or provider-supported idempotency keys where appropriate.

For operations that involve multiple database changes, use database transactions to keep related updates consistent.

Handling Multiple Workers and Application Restarts

The previous worker selects a pending workflow and starts processing it. If two application instances run simultaneously, both may select the same workflow.

This can result in duplicate execution.

A production implementation should claim work atomically before processing it. One common approach is to use a status such as Running, together with a lease expiration time and a unique worker identifier.

The claim operation should be performed through a database transaction or an atomic update so that only one worker can successfully acquire the workflow.

If a worker crashes, its lease eventually expires. Another worker can then recover the workflow.

The workflow should also record retry counts and the next eligible execution time. This prevents an external service failure from causing the worker to retry the same operation continuously.

BackgroundService vs Durable Workflow Engines

A database-backed worker is not always the best solution. The right approach depends on how complex the workflow is.

Feature	BackgroundService with database	Durable workflow engine
Basic background processing	Suitable	Supported
Manual state management	Required	Usually handled by the framework
Restart recovery	Must be implemented	Built into durable execution
Retry management	Developer-managed	Often provided by the engine
Long-running orchestration	Requires additional code	Designed for this purpose
Workflow visibility	Must be implemented	Often includes workflow tracking
Complex branching	Can become difficult	Better suited to orchestration

For simple background tasks, a hosted service with persistent job records may be sufficient.

For workflows involving multiple services, long delays, approvals, retries, and complex execution paths, consider a durable orchestration solution such as Durable Task or a compatible workflow platform.

Best Practices

Keep workflow state in persistent storage rather than relying on application memory. Save progress after each meaningful operation, and make external operations idempotent.

Use cancellation tokens for database operations and external calls. During application shutdown, allow active work to stop safely rather than assuming every operation will finish.

Avoid holding database transactions open while calling external APIs. These calls may take a long time, and keeping transactions open increases locking and resource usage.

Use atomic workflow claiming when multiple application instances are running. Track retries, execution timestamps, and failure details so that unfinished workflows can be investigated.

Finally, monitor workflows that remain in the running or failed state for longer than expected. Persistent storage helps recovery, but monitoring is what helps developers identify problems before they affect more customers.

Common Mistakes and Troubleshooting

Problem	Possible reason	Recommended solution
Workflow starts again from the beginning	Progress was not saved	Persist state after each completed step
Duplicate payment or email	External operation is not idempotent	Use stable idempotency keys
Two workers process the same workflow	No atomic claim mechanism	Implement database-based claiming
Workflow remains in Running state	Worker stopped unexpectedly	Use leases and recovery rules
Database context error	Scoped service resolved incorrectly	Create a dependency injection scope
Repeated failures	No retry limit or delay	Add retry count and backoff
Workflow is marked completed too early	Final state saved before all operations finish	Update completion only after all required steps succeed

Summary

Long-running workflows should not depend on application memory if their progress needs to survive application restarts.

A database-backed workflow allows a .NET application to store execution status, track completed operations, and recover unfinished work after restarting. Combining Entity Framework Core with BackgroundService provides a straightforward way to implement this pattern for simpler workflows.

However, saving progress alone does not guarantee exactly-once execution. External operations can succeed before the application records their completion, so idempotency, atomic work claiming, retry handling, and proper failure recovery are essential for production applications.

For more complex processes involving multiple services and long-running orchestration, a durable workflow engine may be a better choice.

The important design decision is to make workflow progress independent of the application process. Once that is done, an application restart becomes a recoverable event rather than a reason to lose the entire operation.

.NETpersistent workflowsBackgroundServiceEntity Framework Coreworkflow recoverySQL Serverbackground jobs

Comments

Join the conversation! Your thoughts help the community grow.