Testing Webhook Retry Logic When Your Server Is Unavailable

Webhooks allow external services to notify an application when an event occurs. For example, a payment provider can send a webhook when a transaction succeeds, an e-commerce platform can notify an order service about a new order, or a delivery provider can report a shipment update.

The problem starts when the receiving server is unavailable.

A temporary network failure, application restart, database outage, or deployment can prevent the application from receiving a webhook. The sending service may retry the request, but the receiving application must be prepared to handle repeated deliveries.

Testing webhook retry logic is important because a system that works during normal operation may still lose events or process the same business operation multiple times during an outage.

This article explains how to build and test reliable webhook handling in ASP.NET Core using C#, background processing, and integration tests.

Understand How Webhook Retries Work

Consider a payment application that receives a payment confirmation from an external provider.

The provider sends a request:

POST /api/webhooks/payment
Content-Type: application/json

The request contains an event indicating that a payment has completed.

If the application is unavailable, the provider may receive a connection error or timeout. Depending on its retry policy, it may send the same event again.

A typical delivery sequence looks like this:

Payment Provider
       |
       v
Webhook Request
       |
       v
ASP.NET Core API
       |
       v
Application Unavailable
       |
       v
Provider Waits
       |
       v
Retry Webhook
       |
       v
Application Available
       |
       v
Process Event

The exact retry behavior depends on the provider. Some providers use exponential backoff, while others use fixed intervals or a configurable delivery schedule.

Your application should not assume that a webhook will arrive exactly once.

Design the receiver to handle duplicate deliveries safely.

Build a Basic Webhook Endpoint

Start by creating a model for the incoming payment event.

public class PaymentWebhook
{
    public string EventId { get; set; }
        = string.Empty;

    public string PaymentId { get; set; }
        = string.Empty;

    public string Status { get; set; }
        = string.Empty;

    public decimal Amount { get; set; }
}

The EventId identifies the webhook event. It is different from the payment ID because one payment can generate multiple events during its lifecycle.

Next, create an ASP.NET Core endpoint:

[ApiController]
[Route("api/webhooks")]
public class WebhookController : ControllerBase
{
    private readonly IWebhookService _webhookService;

    public WebhookController(
        IWebhookService webhookService)
    {
        _webhookService = webhookService;
    }

    [HttpPost("payment")]
    public async Task<IActionResult> ReceivePaymentAsync(
        [FromBody] PaymentWebhook webhook,
        CancellationToken cancellationToken)
    {
        await _webhookService.ProcessAsync(
            webhook,
            cancellationToken);

        return Ok();
    }
}

This is a basic endpoint. In a production application, verify the provider's webhook signature before accepting the event.

Signature verification should use the original request body and the provider's documented signing algorithm. Do not trust a payment status or amount simply because it appears in a JSON request.

Also, avoid performing lengthy business operations directly inside the HTTP request. A webhook endpoint should generally validate and durably record the event, then return a successful response once the application has accepted responsibility for processing it.

Store Webhook Events Before Processing

A reliable webhook receiver should persist incoming events.

If the application acknowledges a webhook before storing it, a process crash can result in permanent event loss.

Create a database entity:

public class WebhookEvent
{
    public long Id { get; set; }

    public string EventId { get; set; }
        = string.Empty;

    public string EventType { get; set; }
        = string.Empty;

    public string Payload { get; set; }
        = string.Empty;

    public string Status { get; set; }
        = "Pending";

    public int RetryCount { get; set; }

    public DateTime ReceivedAtUtc { get; set; }

    public DateTime? ProcessedAtUtc { get; set; }

    public DateTime? NextRetryAtUtc { get; set; }

    public string? LastError { get; set; }
}

Add a unique index to prevent the same event from being inserted multiple times:

modelBuilder.Entity<WebhookEvent>()
    .HasIndex(x => x.EventId)
    .IsUnique();

The event ID should be provided by the webhook provider and validated before storage.

Now the application has a durable record of incoming events.

If the application restarts, a background worker can continue processing pending records.

Handle Duplicate Webhook Deliveries

Webhook providers commonly retry requests when they do not receive a successful response.

This means the same event may reach your endpoint more than once.

Without duplicate protection, the application might process a payment confirmation repeatedly.

For example, create an event storage service:

public class WebhookService : IWebhookService
{
    private readonly AppDbContext _context;

    public WebhookService(AppDbContext context)
    {
        _context = context;
    }

    public async Task ProcessAsync(
        PaymentWebhook webhook,
        CancellationToken cancellationToken)
    {
        var existing = await _context.WebhookEvents
            .AnyAsync(
                x => x.EventId == webhook.EventId,
                cancellationToken);

        if (existing)
        {
            return;
        }

        var eventRecord = new WebhookEvent
        {
            EventId = webhook.EventId,
            EventType = "PaymentUpdated",
            Payload = JsonSerializer.Serialize(webhook),
            Status = "Pending",
            ReceivedAtUtc = DateTime.UtcNow
        };

        _context.WebhookEvents.Add(eventRecord);

        await _context.SaveChangesAsync(
            cancellationToken);
    }
}

The unique database index is essential.

A separate existence check alone is not sufficient because two identical requests can arrive simultaneously. Both requests may check before either inserts the record.

The database constraint prevents both records from being stored.

Production code should catch the specific unique-constraint violation and treat it as an already-accepted duplicate. Do not treat every database exception as a duplicate.

The endpoint should return a successful response for an event that has already been accepted, provided the existing record is valid and the application has safely retained responsibility for it.

Build a Background Webhook Processor

Once an event is stored, process it independently of the HTTP request.

Create a background service:

public class WebhookProcessor : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<WebhookProcessor> _logger;

    public WebhookProcessor(
        IServiceScopeFactory scopeFactory,
        ILogger<WebhookProcessor> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                using var scope =
                    _scopeFactory.CreateScope();

                var processor = scope.ServiceProvider
                    .GetRequiredService<IWebhookEventProcessor>();

                await processor.ProcessPendingAsync(
                    stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(
                    ex,
                    "Webhook processing cycle failed.");
            }

            await Task.Delay(
                TimeSpan.FromSeconds(5),
                stoppingToken);
        }
    }
}

builder.Services.AddHostedService<WebhookProcessor>();

The processor should retrieve pending events, claim them safely, execute the business operation, and update their status.

For multiple application instances, do not let every worker select the same pending record. Use atomic database claiming, leases, or a durable message queue to coordinate processing.

Implement Retry Logic With Exponential Backoff

External APIs and databases can fail temporarily.

Instead of retrying immediately, increase the delay after each failed attempt.

public static TimeSpan GetRetryDelay(int retryCount)
{
    var delaySeconds = Math.Min(
        Math.Pow(2, retryCount),
        300);

    return TimeSpan.FromSeconds(delaySeconds);
}

This produces delays of approximately:

Retry	Delay
1	2 seconds
2	4 seconds
3	8 seconds
4	16 seconds
5	32 seconds
6	64 seconds

Add random jitter in production to avoid multiple workers retrying at the same time.

The processor should classify failures before scheduling a retry.

For example:

Network timeout: Retry.
HTTP 429: Respect the Retry-After header.
HTTP 503: Retry with backoff.
Invalid webhook payload: Do not retry automatically.
Authentication failure: Investigate credentials.
Business validation failure: Move to manual review when appropriate.

After a configured retry limit, mark the event as failed or move it to a dead-letter workflow.

Do not retry permanent failures indefinitely.

Test What Happens When the Server Is Unavailable

Now comes the important part: testing the retry behavior.

A normal integration test verifies that a valid webhook reaches the application and gets stored.

An outage test verifies what happens when the receiving application cannot accept requests.

There are several scenarios worth testing.

Test 1: Server returns an error

Configure a test endpoint or mock service to return HTTP 503.

The webhook sender should treat the delivery as unsuccessful and retry according to its configured policy.

For a controlled test, use a fake webhook sender rather than depending on an actual payment provider.

Test 2: Server does not respond

Simulate a timeout by delaying the response beyond the sender's timeout.

This helps verify that the sender retries when it cannot determine whether the request was accepted.

The receiver must handle the possibility that it stored the event even though the sender did not receive the response.

Test 3: Server restarts during processing

Store the webhook event, then stop the application before the background worker completes processing.

Restart the application and verify that the pending event is processed.

Test 4: Duplicate webhook delivery

Send the same event multiple times.

Verify that only one event record is created and the business operation is not repeated.

Test 5: External dependency fails

Allow the webhook endpoint to accept the event, but make the downstream payment or order service unavailable.

Verify that the event remains available for retry without requiring the original provider to send it again.

Write an Integration Test for Duplicate Events

ASP.NET Core integration tests can use WebApplicationFactory to host the application in a test environment.

The following example shows the structure of a duplicate-delivery test:

[Fact]
public async Task DuplicateWebhook_ShouldCreateOneEvent()
{
    using var factory = new WebApplicationFactory<Program>();

    using var client = factory.CreateClient();

    var webhook = new PaymentWebhook
    {
        EventId = "evt-1001",
        PaymentId = "pay-4521",
        Status = "Completed",
        Amount = 1500
    };

    var firstResponse = await client.PostAsJsonAsync(
        "/api/webhooks/payment",
        webhook);

    var secondResponse = await client.PostAsJsonAsync(
        "/api/webhooks/payment",
        webhook);

    Assert.True(firstResponse.IsSuccessStatusCode);
    Assert.True(secondResponse.IsSuccessStatusCode);
}

This test verifies that both deliveries are accepted.

It does not yet prove that only one database record exists. Add a database assertion using a test database and inspect the number of stored events.

Use an isolated database for each test run so that previous test data does not affect the result.

Test Recovery After Application Restart

A useful recovery test should verify that a pending event survives application termination.

The test can follow this sequence:

Send a webhook to the application.
Verify that the event is stored as pending.
Stop the test host before processing completes.
Start the application again using the same test database.
Allow the background processor to run.
Verify that the event reaches the completed state.
Confirm that the business operation happened only once.

This test checks persistence and recovery together.

For greater control, separate the background processor from the test host and trigger its processing method directly. This avoids relying on arbitrary delays or timing assumptions in automated tests.

Monitor Webhook Delivery and Processing

Webhook reliability should be visible in production monitoring.

Track metrics such as:

Metric	Purpose
Received events	Measures incoming webhook volume
Duplicate events	Identifies repeated deliveries
Processing failures	Shows unsuccessful operations
Retry count	Measures recovery activity
Processing duration	Identifies slow handlers
Pending event count	Shows backlog size
Oldest pending event	Identifies delayed processing

A growing pending event count may indicate that the application cannot process events as quickly as they arrive.

Also monitor the time between receiving an event and completing its business operation. A webhook endpoint may return successfully while its background processing remains delayed.

Avoid logging complete webhook payloads by default. They may contain customer information, payment details, or other sensitive data.

Common Mistakes to Avoid

One common mistake is returning HTTP 200 before the event has been safely stored. If the application then crashes, the sender may believe the event was accepted even though it has been lost.

Another mistake is assuming that webhook providers deliver events only once. Duplicate delivery is a normal part of reliable webhook systems.

Developers also sometimes retry every error without classification. Invalid data and authorization failures generally require investigation rather than repeated requests.

Finally, avoid testing only successful HTTP responses. A reliable webhook implementation must also be tested during timeouts, application restarts, duplicate deliveries, and downstream service failures.

Summary

Webhook retry logic must account for server outages, network failures, duplicate requests, and interruptions during business processing.

A reliable ASP.NET Core implementation should verify webhook signatures, store incoming events durably, prevent duplicate processing, and use background workers for business operations.

Use exponential backoff for temporary failures, classify permanent errors, and test recovery after application restarts.

The main objective is to ensure that an external event is not lost when your server becomes unavailable and that receiving the same event multiple times does not repeat the underlying business operation.