Recovering AI Agent Tasks After an External API Failure

AI agents often depend on external APIs to complete business operations. An agent might check an order, update a customer record, create a support ticket, or trigger a payment workflow.

These operations do not always finish successfully. An external API may become unavailable, return a timeout, reject a request, or respond with an unexpected error.

The problem becomes more complicated when an agent performs several steps in sequence. If the fourth step fails after the first three have succeeded, restarting the entire task may repeat operations that have already been completed.

A reliable application needs a recovery mechanism that can identify completed steps, retry failed operations safely, and continue the task without creating duplicate business records.

This article explains how to build a recoverable AI agent task workflow using ASP.NET Core, C#, Entity Framework Core, and background processing.

Why AI Agent Tasks Need Recovery

Consider an AI-powered customer support application.

A user asks:

"Check order 4521, confirm its delivery status, and create a support ticket if the order is delayed."

The agent performs these operations:

Retrieve the order from the order service.
Check the latest delivery status.
Create a support ticket.
Return the ticket reference to the user.

Now imagine that the ticket service becomes unavailable during the third operation.

The order lookup and delivery check have already succeeded. If the application starts again from the beginning, it may repeat the first two operations and potentially create multiple tickets when the service becomes available.

There are several reasons an agent task can stop unexpectedly:

External API timeout.
Network interruption.
Rate limiting.
Temporary service unavailability.
Application restart.
Container termination.
Database connection failure.
Invalid response from an external service.

A recoverable workflow should preserve the task's progress so the application can continue from the last completed operation.

Design a Recoverable Task Workflow

Instead of treating the agent's entire request as one large operation, divide it into smaller steps.

Each step should have its own execution status.

For the order support example, the workflow could look like this:

Task Created
     |
     v
Retrieve Order
     |
     v
Check Delivery Status
     |
     v
Create Support Ticket
     |
     v
Return Result

If ticket creation fails, the workflow should preserve the successful order lookup and delivery check.

When the ticket service becomes available again, the application should retry only the ticket creation step.

A useful task state model is:

public enum AgentTaskStatus
{
    Pending,
    Running,
    WaitingForRetry,
    Completed,
    Failed,
    Cancelled
}

Each individual step also needs a status:

public enum AgentStepStatus
{
    Pending,
    Running,
    Completed,
    Failed,
    Skipped
}

Keeping task status separate from step status makes it easier to understand whether a workflow is still running, waiting for an external service, or permanently failed.

Create the Task and Step Entities

Use Entity Framework Core to persist workflow progress.

Start with the main task entity:

public class AgentTask
{
    public Guid Id { get; set; }

    public string ConversationId { get; set; }
        = string.Empty;

    public string UserId { get; set; }
        = string.Empty;

    public AgentTaskStatus Status { get; set; }

    public DateTime CreatedAtUtc { get; set; }

    public DateTime UpdatedAtUtc { get; set; }

    public DateTime? CompletedAtUtc { get; set; }

    public int RetryCount { get; set; }

    public ICollection<AgentTaskStep> Steps { get; set; }
        = new List<AgentTaskStep>();
}

Next, define the individual workflow steps:

public class AgentTaskStep
{
    public Guid Id { get; set; }

    public Guid AgentTaskId { get; set; }

    public string StepName { get; set; }
        = string.Empty;

    public int SequenceNumber { get; set; }

    public AgentStepStatus Status { get; set; }

    public int RetryCount { get; set; }

    public string? ErrorType { get; set; }

    public DateTime? StartedAtUtc { get; set; }

    public DateTime? CompletedAtUtc { get; set; }

    public DateTime? NextRetryAtUtc { get; set; }

    public AgentTask AgentTask { get; set; }
        = null!;
}

The main task represents the user's request, while each step represents one operation.

For example, a task may have three steps:

Step	Operation	Initial status
1	RetrieveOrder	Pending
2	CheckDeliveryStatus	Pending
3	CreateSupportTicket	Pending

After the first two steps succeed, their statuses remain completed even if the third step fails.

Configure the relationship:

public class AppDbContext : DbContext
{
    public AppDbContext(
        DbContextOptions<AppDbContext> options)
        : base(options)
    {
    }

    public DbSet<AgentTask> AgentTasks => Set<AgentTask>();

    public DbSet<AgentTaskStep> AgentTaskSteps =>
        Set<AgentTaskStep>();

    protected override void OnModelCreating(
        ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<AgentTask>()
            .HasMany(x => x.Steps)
            .WithOne(x => x.AgentTask)
            .HasForeignKey(x => x.AgentTaskId);

        modelBuilder.Entity<AgentTaskStep>()
            .HasIndex(x => new
            {
                x.AgentTaskId,
                x.SequenceNumber
            })
            .IsUnique();
    }
}

The unique index prevents duplicate sequence numbers within the same task.

Execute Each Step Independently

The next requirement is a consistent way to execute workflow steps.

Create a common interface:

public interface IAgentTaskStepHandler
{
    string StepName { get; }

    Task ExecuteAsync(
        Guid taskId,
        CancellationToken cancellationToken);
}

Now implement the order lookup operation:

public class RetrieveOrderHandler
    : IAgentTaskStepHandler
{
    private readonly IOrderService _orderService;

    public string StepName => "RetrieveOrder";

    public RetrieveOrderHandler(
        IOrderService orderService)
    {
        _orderService = orderService;
    }

    public async Task ExecuteAsync(
        Guid taskId,
        CancellationToken cancellationToken)
    {
        await _orderService.GetOrderAsync(
            taskId,
            cancellationToken);
    }
}

The example uses an application-level service interface. Its implementation can call an HTTP API or another internal service.

In a real workflow, pass the actual order identifier through a validated task input or persisted step data. The task ID identifies the workflow and should not be treated as the business order ID.

A separate handler can check delivery status:

public class CheckDeliveryHandler
    : IAgentTaskStepHandler
{
    private readonly IDeliveryService _deliveryService;

    public string StepName => "CheckDeliveryStatus";

    public CheckDeliveryHandler(
        IDeliveryService deliveryService)
    {
        _deliveryService = deliveryService;
    }

    public async Task ExecuteAsync(
        Guid taskId,
        CancellationToken cancellationToken)
    {
        await _deliveryService.CheckStatusAsync(
            taskId,
            cancellationToken);
    }
}

Each handler should perform one well-defined operation. This makes it easier to retry a failed step without repeating unrelated work.

Save Progress After Every Successful Step

The workflow executor should update the database immediately after a step finishes successfully.

A simplified implementation looks like this:

public class AgentWorkflowExecutor
{
    private readonly AppDbContext _context;
    private readonly IEnumerable<IAgentTaskStepHandler>
        _handlers;

    public AgentWorkflowExecutor(
        AppDbContext context,
        IEnumerable<IAgentTaskStepHandler> handlers)
    {
        _context = context;
        _handlers = handlers;
    }

    public async Task ExecuteAsync(
        Guid taskId,
        CancellationToken cancellationToken)
    {
        var task = await _context.AgentTasks
            .Include(x => x.Steps)
            .FirstOrDefaultAsync(
                x => x.Id == taskId,
                cancellationToken);

        if (task == null)
        {
            throw new InvalidOperationException(
                "Agent task was not found.");
        }

        task.Status = AgentTaskStatus.Running;
        task.UpdatedAtUtc = DateTime.UtcNow;

        await _context.SaveChangesAsync(
            cancellationToken);

        foreach (var step in task.Steps
            .OrderBy(x => x.SequenceNumber))
        {
            if (step.Status == AgentStepStatus.Completed ||
                step.Status == AgentStepStatus.Skipped)
            {
                continue;
            }

            var handler = _handlers.FirstOrDefault(
                x => x.StepName == step.StepName);

            if (handler == null)
            {
                throw new InvalidOperationException(
                    $"Handler not found: {step.StepName}");
            }

            step.Status = AgentStepStatus.Running;
            step.StartedAtUtc = DateTime.UtcNow;

            await _context.SaveChangesAsync(
                cancellationToken);

            await handler.ExecuteAsync(
                taskId,
                cancellationToken);

            step.Status = AgentStepStatus.Completed;
            step.CompletedAtUtc = DateTime.UtcNow;

            task.UpdatedAtUtc = DateTime.UtcNow;

            await _context.SaveChangesAsync(
                cancellationToken);
        }

        task.Status = AgentTaskStatus.Completed;
        task.CompletedAtUtc = DateTime.UtcNow;
        task.UpdatedAtUtc = DateTime.UtcNow;

        await _context.SaveChangesAsync(
            cancellationToken);
    }
}

This example demonstrates the basic recovery principle: completed steps are skipped when the workflow runs again.

However, this implementation is a starting point, not a complete production scheduler. It assumes one worker executes a task at a time and does not yet handle concurrent workers, leases, retry scheduling, or uncertain external API outcomes.

Those concerns must be addressed before using the workflow for important business operations.

Retry Failed API Calls With Backoff

Not every API failure requires the same response.

A timeout or HTTP 503 response may be temporary. An HTTP 400 response caused by invalid input is usually not fixed by retrying.

A retry policy should classify errors before scheduling another attempt.

A typical policy looks like this:

Error	Recommended action
HTTP 408	Retry with backoff
HTTP 429	Respect Retry-After
HTTP 500	Retry when appropriate
HTTP 502	Retry with backoff
HTTP 503	Retry with backoff
HTTP 400	Do not retry automatically
HTTP 401	Check authentication
HTTP 403	Check authorization
HTTP 404	Review resource availability

Use exponential backoff to avoid repeatedly calling an unavailable service.

For example:

public static TimeSpan GetRetryDelay(int retryCount)
{
    var delaySeconds = Math.Min(
        Math.Pow(2, retryCount),
        300);

    return TimeSpan.FromSeconds(delaySeconds);
}

The delay increases with each failed attempt and is capped at five minutes.

In production, add random jitter to prevent multiple workers from retrying at exactly the same time. Also respect provider-supplied retry instructions, such as the HTTP Retry-After header.

Set a maximum retry count. Once that limit is reached, move the task into a failed or manual-review state instead of retrying forever.

Prevent Duplicate Operations With Idempotency

Retrying an API call introduces an important problem.

Imagine the ticket service successfully creates a support ticket, but the response never reaches your application because the network connection drops.

The application sees a timeout and assumes the operation failed.

If it retries the request without protection, the service may create a second ticket.

This is why idempotency is essential.

Generate a stable idempotency key for each logical operation:

var idempotencyKey =
    $"{taskId}:CreateSupportTicket";

Pass this key to the external service when it supports idempotency:

public async Task CreateTicketAsync(
    Guid taskId,
    CancellationToken cancellationToken)
{
    var idempotencyKey =
        $"{taskId}:CreateSupportTicket";

    using var request = new HttpRequestMessage(
        HttpMethod.Post,
        "/api/tickets");

    request.Headers.Add(
        "Idempotency-Key",
        idempotencyKey);

    request.Content = JsonContent.Create(
        new
        {
            OrderId = "4521",
            Reason = "Delivery delayed"
        });

    var response = await _httpClient.SendAsync(
        request,
        cancellationToken);

    response.EnsureSuccessStatusCode();
}

The order ID and reason are illustrative values. In a real application, retrieve them from validated task data.

The external service must actually support and enforce idempotency for this header to provide protection.

If the provider does not support idempotency, use a local operation record, a unique business constraint, or a reconciliation process to determine whether the operation already succeeded.

Do not assume that a timeout means the external operation did not happen.

Recover Tasks After an Application Restart

Saving step progress makes recovery possible, but the application still needs a process to find unfinished tasks.

A background service can periodically identify tasks waiting for retry.

For example:

public class AgentTaskRecoveryService : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<AgentTaskRecoveryService> _logger;

    public AgentTaskRecoveryService(
        IServiceScopeFactory scopeFactory,
        ILogger<AgentTaskRecoveryService> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                using var scope = _scopeFactory
                    .CreateScope();

                var processor = scope.ServiceProvider
                    .GetRequiredService<IAgentTaskProcessor>();

                await processor.ProcessReadyTasksAsync(
                    stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(
                    ex,
                    "Agent task recovery failed.");
            }

            await Task.Delay(
                TimeSpan.FromSeconds(15),
                stoppingToken);
        }
    }
}

The processor should select only tasks that are eligible for execution, such as pending tasks or tasks whose retry time has arrived.

It must also prevent two application instances from processing the same task simultaneously.

For a single-instance demonstration, this may be handled with a simple database check. For a distributed production environment, use database leases, atomic row updates, or a durable queue with appropriate visibility timeouts.

A worker should claim a task with a lease expiration time. If the worker crashes, another worker can reclaim the task after the lease expires.

Handle Tasks That Stop in the Running State

An application may terminate after marking a step as running but before saving its result.

When the application restarts, that step may remain marked as running even though no worker is executing it.

A recovery process should identify stale running steps using their start time and worker lease.

However, do not automatically assume that a stale running step failed.

The external API may have completed the operation before the application stopped. Retrying it without checking its result could create duplicate business operations.

For operations with uncertain outcomes:

Check whether the external system supports querying the operation by an idempotency key or reference.
Reconcile the external result with the local workflow record.
Mark the step completed if the operation already succeeded.
Retry only when it is safe to do so.
Send unresolved operations for manual review when necessary.

This reconciliation step is especially important for payments, order creation, deployments, and other operations with significant business impact.

Best Practices for Production Workflows

A reliable agent workflow should follow a few important principles.

Persist task inputs and outputs carefully

Store the minimum information needed to resume a task. Avoid storing unrestricted prompts, API credentials, or sensitive customer data.

Keep agent reasoning separate from execution state

Do not depend on conversation history to determine which operations have completed. The workflow database should be the authoritative source of execution state.

Use a durable queue for larger workloads

A background service that polls a database can work for smaller systems. For higher-volume workloads, a durable messaging system or workflow engine may provide better scheduling, retries, and operational visibility.

Make every step independently recoverable

Each step should have a clear input, output, status, and retry policy. Avoid combining multiple external operations into one handler when they need different recovery behavior.

Record execution history

Track each attempt separately, including its start time, completion time, error category, and idempotency key. This makes troubleshooting much easier.

Add timeouts and circuit breakers

Timeouts prevent workers from waiting indefinitely. Circuit breakers can temporarily stop calls to an unhealthy service and allow it to recover.

Define manual recovery procedures

Some operations cannot be safely retried automatically. Provide an administrative process to inspect, reconcile, and resume those tasks.

Summary

External API failures should not force an AI agent to restart an entire workflow.

By dividing agent tasks into independent steps, saving progress after every successful operation, and applying controlled retry policies, an ASP.NET Core application can recover interrupted tasks without repeating completed work.

Idempotency is equally important because a timeout does not always mean an external operation failed.

For production systems, combine persistent workflow state, durable task scheduling, worker leases, retry backoff, and reconciliation. This gives AI agents a reliable execution model even when external services or application instances become unavailable.