A worker sends an article to a publishing API. The API accepts it, but the response times out. The queue retries the job. If the worker simply calls Publish again, readers may see two articles.

This is a hypothetical failure, but the state ambiguity is real: the caller cannot infer from a timeout whether the destination accepted the request.

The practical answer has three parts:

  1. Assign a stable key to the publishing intent.

  2. Enforce uniqueness for that intent in durable storage.

  3. Make the remote call idempotent or provide a way to reconcile uncertain results.

A unique local record alone prevents duplicate records, not necessarily duplicate external articles.

Define the Operation Before Choosing a Key

Suppose a publishing request has a tenant, source article, revision, and destination. Those fields define the business operation:

tenant-17 / article-84 / revision-3 / destination-A

A retry of that exact operation must reuse its key.

A new article revision needs a new key. Publishing to another destination also needs a new key.

Do not generate a fresh Guid for every queue delivery and call it an idempotency key. That identifies the attempt, not the intended publication.

Record a hash of the rendered payload as well. If the same business key arrives with different content, the worker should raise a conflict for investigation. It should not quietly return an earlier result or overwrite the destination.

Reserve One Local Job With a Database Constraint

The following EF Core model is illustrative. It assumes non-null string fields and PostgreSQL through Npgsql.

Put it in PublicationJob.cs. Apply a migration so the named unique index exists in the actual database before processing jobs.

using Microsoft.EntityFrameworkCore;

public sealed class PublicationJob
{
    public Guid Id { get; set; } = Guid.NewGuid();

    public string TenantId { get; set; } = "";

    public string ArticleId { get; set; } = "";

    public string RevisionId { get; set; } = "";

    public string Destination { get; set; } = "";

    public string PayloadHash { get; set; } = "";

    public string State { get; set; } = "Pending";

    public string? RemoteId { get; set; }
}

public sealed class PublishingContext : DbContext
{
    public PublishingContext(DbContextOptions<PublishingContext> options)
        : base(options)
    {
    }

    public DbSet<PublicationJob> Jobs => Set<PublicationJob>();

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<PublicationJob>()
            .HasIndex(x => new
            {
                x.TenantId,
                x.ArticleId,
                x.RevisionId,
                x.Destination
            })
            .IsUnique()
            .HasDatabaseName("UX_PublicationJob_Identity");
    }
}

The unique index protects the key even when two workers both check for an existing job and find nothing.

A preliminary existence check can improve efficiency, but it is not a concurrency guarantee. The database constraint decides which insert wins.

EF Core supports configuring unique indexes through the model configuration.

Reserve or Load the Job

The following PostgreSQL-specific example handles a unique-constraint violation on the named index by re-reading the existing record.

Put this in JobReservations.cs in the same project as the model.

using System.Security.Cryptography;
using System.Text;
using Microsoft.EntityFrameworkCore;
using Npgsql;

public static class JobReservations
{
    public static async Task<PublicationJob> ReserveAsync(
        PublishingContext db,
        string tenant,
        string article,
        string revision,
        string destination,
        string payload,
        CancellationToken ct)
    {
        var hash = Convert.ToHexString(
            SHA256.HashData(Encoding.UTF8.GetBytes(payload)));

        var matching = db.Jobs.Where(x =>
            x.TenantId == tenant &&
            x.ArticleId == article &&
            x.RevisionId == revision &&
            x.Destination == destination);

        var current = await matching.SingleOrDefaultAsync(ct);

        if (current is not null)
            return Validate(current, hash);

        var candidate = new PublicationJob
        {
            TenantId = tenant,
            ArticleId = article,
            RevisionId = revision,
            Destination = destination,
            PayloadHash = hash
        };

        db.Jobs.Add(candidate);

        try
        {
            await db.SaveChangesAsync(ct);
            return candidate;
        }
        catch (DbUpdateException ex) when (
            ex.InnerException is PostgresException pg &&
            pg.SqlState == PostgresErrorCodes.UniqueViolation &&
            pg.ConstraintName == "UX_PublicationJob_Identity")
        {
            db.Entry(candidate).State = EntityState.Detached;

            var winner = await matching.SingleOrDefaultAsync(ct);

            if (winner is null)
                throw;

            return Validate(winner, hash);
        }
    }

    private static PublicationJob Validate(
        PublicationJob job,
        string hash)
    {
        if (job.PayloadHash != hash)
        {
            throw new InvalidOperationException(
                "Same publication key was used for different content.");
        }

        return job;
    }
}

Important Boundary

This code reserves a durable local job. It does not implement:

The code should be tested with the pinned Npgsql provider, and the migration should be verified to ensure the named unique index was created.

A different database provider requires its own error classification. Do not catch and ignore unrelated database failures.

What Happens After Reservation?

Before making the external call, claim the job so two workers do not publish it concurrently.

If the destination supports an idempotency key, send the same business key on every retry and record the returned remote ID.

If the destination does not support idempotency, investigate whether it provides another way to query a publication by a stable external reference.

A timeout after the request was sent should move the job into an uncertain or reconciliation state. An automatic blind retry can create a duplicate.

For example, two workers may find the same saved row while the first worker is still sending the request.

Reservation tells them they share the same operation. It does not, by itself, give either worker exclusive permission to perform the remote call.

A durable claim needs its own concurrency rule, such as:

After a timeout, record enough information to query the remote destination safely.

If the remote system cannot confirm whether it accepted the article and provides no idempotency guarantee, the operator may need to reconcile the uncertain job manually.

No retry setting can resolve that uncertainty on its own.

Idempotency and the Outbox Pattern

Message-based systems commonly combine retryable processing with idempotent operations and durable state.

An outbox pattern can also help when the application needs to reliably connect a local database transaction with a message or background job.

A hosted service can run the queue worker, but a process-local lock is not durable across application restarts or multiple application instances.

The worker therefore needs durable coordination when multiple consumers can process the same operation.

How Should You Test the Design?

Use a disposable PostgreSQL test database with the pinned provider and a controllable fake publisher.

Test Concurrent Reservations

Run two reservations with the same key and payload concurrently.

Verify that:

Test Conflicting Payloads

Use the same business key with a different payload.

Verify that the second attempt fails without changing the existing job.

This prevents a retry or caller bug from silently changing the meaning of an existing publication operation.

Test a Timeout After Remote Acceptance

Simulate the destination accepting a request while the client sees a timeout.

Retry using the same remote idempotency key.

Verify that only one remote record exists when the fake destination supports that contract.

Test a Destination Without Idempotency

Repeat the timeout scenario with a destination that does not support idempotency.

The worker should flag the operation for reconciliation instead of blindly sending the request again.

Test Legitimate New Operations

Create a new article revision and publish to another destination.

Each should create its own legitimate publication operation.

For example:

Article 84 / Revision 3 / Destination A
        |
        +---- Same operation -> Same key

Article 84 / Revision 4 / Destination A
        |
        +---- New revision -> New key

Article 84 / Revision 3 / Destination B
        |
        +---- New destination -> New key

The Two Guarantees Are Different

The most important distinction in this design is between local idempotency and remote idempotency.

A unique database constraint can guarantee that the application has one durable record for a particular publishing intent.

It cannot guarantee that the external publishing API created only one article.

The remote system needs its own idempotency mechanism or a reliable reconciliation mechanism.

The flow therefore looks like this:

Queue Delivery
      |
      v
Stable Business Key
      |
      v
Unique Local Job
      |
      v
Durable Claim
      |
      v
Remote Publish
      |
      +---- Success ----> Record Remote ID
      |
      +---- Timeout ----> Reconcile
                              |
                              v
                    Query Remote State
                              |
                 +------------+------------+
                 |                         |
              Accepted                  Not Found
                 |                         |
                 v                         v
          Record Result             Safe Retry

This makes the recovery path explicit instead of relying on a retry setting to solve an inherently ambiguous outcome.

Conclusion

A timeout does not tell a worker whether the remote system accepted a request. That uncertainty is the core problem behind duplicate publications.

A reliable design starts by defining the business operation and assigning it a stable identity. A durable database constraint then prevents multiple local records from representing the same operation.

That is only the first half of the solution.

The remote publishing call must also be idempotent or provide a reliable way to reconcile an uncertain result. Otherwise, a retry after a timeout can still create a duplicate external article.

The key distinction is simple:

One local job does not prove one remote article.

When those two guarantees are designed and tested separately, timeout recovery becomes an explicit engineering decision rather than a hopeful retry strategy.