How to Test Webhook Delivery When the Receiver Is Temporarily Offline

Webhooks allow applications to notify other systems when an event occurs. A payment provider can notify an application when a payment succeeds, an e-commerce platform can report an order update, and a deployment service can send an event when a build finishes.

This approach avoids requiring one application to repeatedly poll another system for updates. Instead, the sender delivers an HTTP request to a configured endpoint when an event occurs.

However, webhook delivery depends on the receiver being available. The receiver might be temporarily offline because of a deployment, network interruption, database outage, or unexpected traffic spike. If the sender cannot deliver the event, the system must decide whether to retry, delay delivery, or eventually mark the event as failed.

These situations can create problems when webhook handling is not designed carefully. An event might be delivered multiple times, processed out of order, or lost after a temporary failure.

Testing webhook delivery requires more than checking whether an endpoint returns HTTP 200. Developers must verify retry behavior, duplicate handling, delivery limits, and recovery after the receiver becomes available again.

In this article, we will explore how to test webhook delivery during outages, simulate temporary receiver failures, and build reliable webhook processing in a .NET application.

Understand How Webhook Delivery Works

A typical webhook workflow involves two applications:

The sender generates an event and sends an HTTP request to the receiver.
The receiver validates the request and processes the event.
The receiver returns an HTTP response indicating whether it accepted the request.
If delivery fails, the sender follows its retry policy.

Consider an order management system that sends a webhook when an order is shipped.

The sender creates an event containing the order identifier and shipment details. It sends the event to the receiver, which updates its internal records.

If the receiver is unavailable, the sender may receive a connection error or an HTTP 503 response. Depending on its implementation, the sender might retry the delivery after a delay.

The receiver could also process the event successfully but fail to return its response before the sender's timeout expires. In that case, the sender may retry an event that has already been processed.

This leads to an important distinction: a delivery attempt is not the same as a successfully completed business operation.

A reliable webhook design must account for both temporary delivery failures and repeated delivery attempts.

Define the Expected Delivery Behavior

Before writing tests, document how the sender and receiver should behave when delivery fails.

Different webhook providers implement different retry schedules, response handling rules, and delivery limits. Do not assume that every provider retries the same HTTP status codes or uses the same backoff strategy.

For a webhook system you control, define a clear policy.

For example:

Situation	Expected behavior
Receiver returns HTTP 200	Mark the delivery as acknowledged
Receiver returns HTTP 503	Retry according to the configured policy
Connection cannot be established	Retry if the failure is considered transient
Request times out	Treat the delivery outcome as uncertain
Receiver returns HTTP 400	Usually stop retrying unless the system defines another policy
Receiver processes an event but the response is lost	Allow redelivery without repeating the business operation
Retry limit is reached	Record the failure and make the event available for investigation
Receiver recovers after an outage	Allow pending events to be delivered according to the retry policy

These are example policies, not universal webhook rules. Your implementation should follow the sender's documented contract.

For example, some providers retry certain client errors, while others do not. Some offer manual redelivery or a delivery history. Your tests must reflect the actual behavior you need to support.

Build a Webhook Receiver That Can Handle Retries

A webhook receiver should validate incoming requests, identify the event, and process it safely.

A simple ASP.NET Core endpoint might look like this:

using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("webhooks/orders")]
public sealed class OrderWebhookController : ControllerBase
{
    private readonly IOrderWebhookProcessor _processor;

    public OrderWebhookController(
        IOrderWebhookProcessor processor)
    {
        _processor = processor;
    }

    [HttpPost]
    public async Task<IActionResult> Receive(
        [FromBody] OrderWebhookEvent webhookEvent,
        CancellationToken cancellationToken)
    {
        if (string.IsNullOrWhiteSpace(webhookEvent.EventId))
        {
            return BadRequest("Event ID is required.");
        }

        await _processor.ProcessAsync(
            webhookEvent,
            cancellationToken);

        return Ok();
    }
}

public sealed record OrderWebhookEvent(
    string EventId,
    string OrderId,
    string Status);

public interface IOrderWebhookProcessor
{
    Task ProcessAsync(
        OrderWebhookEvent webhookEvent,
        CancellationToken cancellationToken);
}

This example demonstrates the endpoint structure, but it does not yet implement duplicate detection or signature validation.

In production, the receiver should authenticate the sender according to the provider's documented security mechanism. For signed webhooks, validate the signature against the original request body before trusting the event.

The event identifier should also be used to prevent duplicate business processing. Checking that an event ID exists is not the same as enforcing uniqueness.

Test the Receiver When It Is Temporarily Unavailable

One way to test sender retry behavior is to simulate a receiver that returns a temporary failure before becoming available.

Suppose you control both the webhook sender and its retry mechanism. You can use a fake HTTP handler to return HTTP 503 on the first attempt and HTTP 200 on the next attempt.

The following example uses xUnit and .NET's standard HTTP abstractions.

using System.Net;
using Xunit;

public sealed class WebhookDeliveryTests
{
    [Fact]
    public async Task Delivery_RetriesAfterTemporaryFailure()
    {
        var attempts = 0;

        var handler = new FakeHttpMessageHandler(
            (request, cancellationToken) =>
            {
                attempts++;

                var statusCode = attempts == 1
                    ? HttpStatusCode.ServiceUnavailable
                    : HttpStatusCode.OK;

                return Task.FromResult(
                    new HttpResponseMessage(statusCode));
            });

        using var httpClient = new HttpClient(handler);

        using var firstResponse = await httpClient.PostAsync(
            "https://receiver.example.test/webhooks/orders",
            new StringContent(
                """{"eventId":"evt-100","orderId":"ORD-100","status":"Shipped"}"""));

        Assert.Equal(
            HttpStatusCode.ServiceUnavailable,
            firstResponse.StatusCode);

        using var secondResponse = await httpClient.PostAsync(
            "https://receiver.example.test/webhooks/orders",
            new StringContent(
                """{"eventId":"evt-100","orderId":"ORD-100","status":"Shipped"}"""));

        Assert.Equal(
            HttpStatusCode.OK,
            secondResponse.StatusCode);

        Assert.Equal(2, attempts);
    }

    private sealed class FakeHttpMessageHandler
        : HttpMessageHandler
    {
        private readonly Func<
            HttpRequestMessage,
            CancellationToken,
            Task<HttpResponseMessage>> _handler;

        public FakeHttpMessageHandler(
            Func<
                HttpRequestMessage,
                CancellationToken,
                Task<HttpResponseMessage>> handler)
        {
            _handler = handler;
        }

        protected override Task<HttpResponseMessage> SendAsync(
            HttpRequestMessage request,
            CancellationToken cancellationToken)
        {
            return _handler(request, cancellationToken);
        }
    }
}

This test verifies that the fake receiver produces the expected sequence of responses. It does not test an automatic retry policy because the test sends two requests explicitly.

For a complete retry test, place the retry logic in a separate component and invoke that component once. Then assert that it retries after the first failure, stops after success, and respects the configured attempt limit.

The example also uses a fixed event ID in both requests. In production, the sender should preserve the same event identity when retrying the same logical event.

Test the Retry Policy Itself

A retry policy should have predictable limits and delays.

For example, a sender might retry a transient failure up to three times, using increasing delays between attempts. The actual delay values should be configurable and appropriate for the service.

Avoid writing tests that wait for long periods in real time. Instead, inject a delay abstraction or use a retry library's supported testing facilities.

An injectable delay can make retry behavior easier to verify.

public interface IRetryDelay
{
    Task WaitAsync(
        TimeSpan delay,
        CancellationToken cancellationToken);
}

A production implementation can use Task.Delay. A test implementation can record requested delays and return immediately.

This allows tests to verify that:

A temporary failure triggers another attempt.
A successful response stops further attempts.
The configured maximum attempt count is respected.
Cancellation prevents unnecessary future attempts.
The delay increases according to the chosen backoff policy.

For production workloads, exponential backoff with jitter can reduce pressure on a receiver that is recovering from an outage.

Be careful when defining the maximum number of attempts. Some systems count the initial request as the first attempt, while others describe retries separately. Make the convention explicit in the implementation and tests.

Prevent Duplicate Event Processing

Webhook senders often use at-least-once delivery. This means an event may be delivered more than once, especially when the sender cannot determine whether an earlier attempt succeeded.

Receivers should therefore make event processing idempotent.

Suppose a shipping event updates an order's status. Processing the same event twice should not create two shipment records or trigger the same downstream action twice.

A common approach is to store processed event IDs in a database with a unique constraint.

The processing workflow should atomically record the event and apply its business changes, where the architecture permits.

Conceptually, the receiver follows this process:

Validate the webhook and its sender.
Begin a database transaction.
Attempt to record the event ID in a table with a uniqueness constraint.
If the event has already been processed, return an appropriate acknowledgement.
Otherwise, apply the business changes.
Commit the transaction.
Return a successful response.

The event record and business changes should be coordinated carefully. If the application marks an event as processed before completing its business operation, a crash could leave the event recorded without the required change.

Likewise, if the business operation completes but the event record is not committed, a redelivery could repeat the operation.

The exact implementation depends on the database and transaction boundaries. External side effects, such as sending email or publishing another message, may require an outbox pattern or another reliable delivery mechanism.

Test Duplicate Delivery

Create a test that sends the same event more than once and verifies that the business operation occurs only once.

For example, if an event creates a shipment record, send the same event ID twice and assert that only one shipment record exists.

Do not rely solely on an in-memory set for production deduplication. An in-memory collection is lost when the process restarts and cannot reliably coordinate multiple application instances.

Use persistent storage with appropriate uniqueness guarantees when duplicate processing would cause a business problem.

Test the Most Difficult Failure: Processing Succeeds but the Response Is Lost

One of the most important webhook scenarios occurs when the receiver processes an event successfully but the sender never receives the acknowledgement.

Consider this sequence:

The sender delivers an order event.
The receiver records the event and updates the order.
The receiver sends HTTP 200.
The connection breaks before the sender receives the response.
The sender retries the event.

From the sender's perspective, the delivery may have failed. From the receiver's perspective, the event was already processed.

Your tests should reproduce this situation by allowing the first request to complete its business operation while simulating a lost response from the sender's point of view.

On redelivery, the receiver should recognize the event ID and avoid repeating the business operation.

The test should verify both outcomes:

The sender is allowed to retry the event.
The receiver applies the business operation only once.

This scenario is more valuable than testing only a straightforward HTTP 503 because it verifies that the system can handle uncertainty without losing data or duplicating business effects.

Test Recovery After a Longer Outage

A receiver may remain unavailable for several minutes because of a deployment, infrastructure failure, or database outage.

Testing recovery requires more than simulating one failed request.

Verify that the sender retains pending events, follows its retry policy, and eventually resumes delivery when the receiver becomes available.

For a sender you control, useful tests include:

A receiver returns temporary failures for several attempts.
The retry limit is reached and the event is recorded for further handling.
The receiver becomes available and a pending event is redelivered.
A sender process restarts while deliveries are pending.
Multiple events remain ordered where ordering is required.
A large backlog does not overwhelm the recovering receiver.

Not every webhook system guarantees event ordering. If your business process requires a particular order, define how ordering is maintained and what happens when an earlier event remains undelivered.

For example, an order cancellation and a shipment notification may arrive in an unexpected order. The receiver may need event versions, timestamps, or state-transition validation to prevent an older event from overwriting newer state.

Test with Real HTTP Endpoints When Necessary

Fake HTTP handlers are useful for unit tests, but they do not reproduce every networking behavior.

They cannot fully simulate DNS failures, connection resets, load balancer timeouts, TLS problems, reverse proxy behavior, or a real receiver process becoming unavailable.

Use integration tests with a local test server or containerized receiver when you need to verify the interaction between the sender and a real HTTP endpoint.

For more realistic outage tests, temporarily make the test receiver unavailable or configure it to delay responses and return controlled status codes.

Keep these tests isolated from production endpoints and credentials. Avoid relying on external public services because their availability and behavior are outside your control.

For third-party webhook providers, use their documented test-event features, sandbox environments, or delivery replay mechanisms where available.

Common Mistakes to Avoid

Assuming HTTP 200 Means the Entire Workflow Is Complete

An HTTP response should acknowledge only the work the receiver has actually accepted or completed.

If the receiver queues the event for asynchronous processing, its acknowledgement semantics should be explicit. Returning success before safely storing the event can cause data loss if the process crashes immediately afterward.

Retrying Every Failure Indefinitely

Unlimited retries can overwhelm an unavailable receiver and create a large backlog.

Use a bounded retry policy, appropriate delays, and a clear process for investigating events that cannot be delivered.

Ignoring Duplicate Events

A sender may deliver an event more than once even when both systems are functioning correctly.

Use persistent event identifiers and idempotent processing rather than assuming each HTTP request represents a new business event.

Testing Only the Sender

The sender's retry policy and the receiver's duplicate-handling logic are separate responsibilities. Test both.

Assuming a Local Fake Proves Production Reliability

A fake handler verifies the behavior you programmed into the test double. It does not prove that your production network, database, or third-party provider behaves the same way.

Combine unit tests with integration tests and provider-specific validation.

Best Practices for Reliable Webhook Testing

Give every logical event a stable, unique identifier.
Document which HTTP responses trigger retries.
Use bounded retries with backoff and jitter.
Preserve pending events when delivery cannot be completed.
Make receiver-side processing idempotent.
Use database constraints to prevent duplicate event records.
Test response loss after successful processing.
Test recovery after the receiver becomes available.
Define event ordering requirements explicitly.
Record delivery attempts, timestamps, response codes, and correlation identifiers.
Avoid logging authentication secrets or sensitive payload data.
Monitor failed deliveries and provide a controlled process for replaying them.
Use integration tests to validate actual HTTP behavior.

Advantages and Limitations

Testing webhook delivery during outages helps reveal problems with retry handling, duplicate processing, recovery, and event persistence before they affect real workflows.

Fake handlers make unit tests fast and repeatable. Integration tests provide additional confidence that the sender and receiver work together over HTTP.

However, neither approach can guarantee that every infrastructure failure will be reproduced. Third-party providers also differ in retry schedules, event ordering, acknowledgement requirements, and replay capabilities.

The most reliable approach combines deterministic tests, persistent idempotency controls, realistic integration testing, and monitoring of actual delivery behavior.

Summary

Webhook delivery can fail even when the sender and receiver are implemented correctly. Receivers may be temporarily offline, network connections may break, and successful processing may occur without the sender receiving an acknowledgement.

To handle these situations safely, define an explicit retry policy, preserve stable event identifiers, and make receiver-side processing idempotent.

Test ordinary temporary failures, exhausted retries, duplicate delivery, and the more difficult case where processing succeeds but the response is lost. Use fake HTTP handlers for deterministic unit tests and real endpoints or provider sandbox environments for integration testing.

The goal is to ensure that temporary outages delay event processing without silently losing important events or repeating business operations.