Use Fencing Tokens to Prevent Stale Distributed Locks from Writing Data

Distributed locks are commonly used when multiple application instances need to work on the same resource without interfering with each other. For example, two background workers might attempt to process the same invoice, update the same inventory record, or generate the same report.

A distributed lock helps coordinate these workers. However, there is an important problem that a lock alone cannot always prevent: a worker may continue running after its lock has expired and another worker has acquired the same lock.

This situation can cause stale data, duplicated operations, and unexpected changes to shared resources.

Fencing tokens provide a way to protect the resource itself. Instead of trusting that a worker still owns the lock, the storage system checks whether the worker's token is older than the most recent token it has accepted.

In this article, we will look at how stale locks happen, how fencing tokens work, and how to implement the approach in a .NET application.

Why Distributed Locks Can Fail

Imagine an application that generates a daily sales report. Three instances of the application are running, but only one should generate the report at a time.

Worker A acquires a distributed lock and starts processing. The lock service grants a lease that remains valid for 30 seconds.

Worker A then encounters a long-running database query or a temporary network delay. It takes longer than expected to finish its work. Meanwhile, its lock lease expires.

Worker B acquires the same lock and begins generating the report. Worker B finishes first and saves the latest report.

A short time later, Worker A resumes execution and saves its older result. Although Worker A no longer owns the lock, the database may accept its write because nothing checks whether the worker is still authorized to update the resource.

The sequence looks like this:

Time	Worker A	Worker B
T1	Acquires the lock	Waiting
T2	Starts processing	Waiting
T3	Lock expires while A is delayed	Waiting
T4	Still processing with stale ownership	Acquires the lock
T5	Still delayed	Saves the new report
T6	Resumes and saves an older report	Finished

The lock service may be working correctly. It expired the lease and allowed another worker to proceed. The problem is that Worker A can still send a write after losing its ownership.

This is why checking a lock only when acquiring it is not sufficient for protecting long-running operations.

What Is a Fencing Token?

A fencing token is a monotonically increasing number assigned whenever a worker acquires a lock. Each new lock acquisition receives a token greater than the previous token for that resource.

For example:

Worker A acquires the lock and receives token 41.
Worker A pauses, and its lease expires.
Worker B acquires the lock and receives token 42.
Worker B writes data using token 42.
Worker A resumes and attempts to write using token 41.

The storage system remembers that token 42 has already been accepted. When Worker A submits token 41, the storage system rejects the write.

<box border radius="lg" padding={3} gap={3}> <title size="md">How fencing tokens protect a resource</title> <box background="surface-secondary" radius="md" padding={3} align="center" gap={1}> Worker A

<text color="secondary" size="sm">Lock acquisition returns token 41</text>

</box> <row justify="center"> <icon name="arrow-down" size="lg" /> </row> <box background="surface" border radius="md" padding={3} align="center" gap={1}> Lease expires

<text color="secondary" size="sm">Worker A pauses; Worker B acquires token 42</text>

</box> <row justify="center"> <icon name="arrow-down" size="lg" /> </row> <box background="surface" border radius="md" padding={3} align="center" gap={1}> Storage receives token 42

<badge color="success">Accepted</badge>
<text color="secondary" size="sm">The latest fencing token is recorded</text>

</box> <row justify="center"> <icon name="arrow-down" size="lg" /> </row> <box background="surface" border radius="md" padding={3} align="center" gap={1}> Worker A submits token 41

<badge color="danger">Rejected</badge>
<text color="secondary" size="sm">The token is older than the recorded token</text>

</box> </box>

The key difference is that the storage system does not need to know whether Worker A's lease has expired. It only needs to compare the token attached to the write with the highest token it has accepted.

A fencing token must be generated by a trusted lock or coordination mechanism. A worker should not generate its own token, and a random lock identifier is not a substitute for an ordered fencing token.

Implementing Fencing Tokens in .NET

The implementation has two parts: acquiring a lock that returns a fencing token and enforcing that token when writing to the protected resource.

The lock implementation depends on the coordination technology you use. Your distributed lock provider must issue a strictly increasing token for each new acquisition of the same protected resource. If the provider cannot do this, you need another trusted mechanism to generate and maintain the sequence.

Start with an abstraction that makes the token explicit:

public interface IDistributedLockProvider
{
    Task<ILockLease> AcquireAsync(
        string resource,
        CancellationToken cancellationToken);
}

public interface ILockLease : IAsyncDisposable
{
    long FencingToken { get; }
}

The lease represents the current lock acquisition. Its FencingToken identifies that acquisition in an ordered sequence.

A worker can then use the token when updating a resource:

public sealed class ReportWorker
{
    private readonly IDistributedLockProvider _lockProvider;
    private readonly IReportRepository _reportRepository;

    public ReportWorker(
        IDistributedLockProvider lockProvider,
        IReportRepository reportRepository)
    {
        _lockProvider = lockProvider;
        _reportRepository = reportRepository;
    }

    public async Task GenerateReportAsync(
        CancellationToken cancellationToken)
    {
        await using var lease =
            await _lockProvider.AcquireAsync(
                "daily-sales-report",
                cancellationToken);

        long fencingToken = lease.FencingToken;

        var report = await GenerateReportDataAsync(
            cancellationToken);

        await _reportRepository.SaveAsync(
            report,
            fencingToken,
            cancellationToken);
    }

    private static Task<SalesReport> GenerateReportDataAsync(
        CancellationToken cancellationToken)
    {
        // Replace with the actual report-generation logic.
        return Task.FromResult(new SalesReport(
            DateOnly.FromDateTime(DateTime.UtcNow),
            TotalSales: 12500m));
    }
}

public sealed record SalesReport(
    DateOnly ReportDate,
    decimal TotalSales);

public interface IReportRepository
{
    Task SaveAsync(
        SalesReport report,
        long fencingToken,
        CancellationToken cancellationToken);
}

The lock provider and repository interfaces are application-defined abstractions. They do not provide distributed locking or fencing automatically. The critical requirement is that SaveAsync passes the token to storage and that storage validates it atomically before accepting the write.

Also, releasing a lease when the await using scope ends does not guarantee that every operation has stopped. A paused or partitioned worker may resume later. That is precisely the case fencing tokens are designed to handle.

Enforce the Token at the Database

A fencing token only protects a resource if the resource checks it. Passing a token through application code without validating it at the database does not prevent stale writes.

One practical approach is to store the highest accepted token alongside the resource. Every write must compare its token with that value inside the same transaction that updates the business data.

For example, a SQL Server table could include a fencing-token column:

CREATE TABLE dbo.SalesReports
(
    ReportDate DATE NOT NULL PRIMARY KEY,
    TotalSales DECIMAL(18, 2) NOT NULL,
    LastFencingToken BIGINT NOT NULL
);

The application should use a stored procedure or equivalent transactional operation to enforce the token check. The following example locks the resource row before comparing and updating it.

CREATE OR ALTER PROCEDURE dbo.SaveSalesReport
    @ReportDate DATE,
    @TotalSales DECIMAL(18, 2),
    @FencingToken BIGINT
AS
BEGIN
    SET NOCOUNT ON;
    SET XACT_ABORT ON;

    BEGIN TRY
        BEGIN TRANSACTION;

        DECLARE @LastToken BIGINT;

        SELECT @LastToken = LastFencingToken
        FROM dbo.SalesReports WITH (UPDLOCK, HOLDLOCK)
        WHERE ReportDate = @ReportDate;

        IF @LastToken IS NOT NULL
           AND @FencingToken < @LastToken
        BEGIN
            THROW 50001, 'Stale fencing token rejected.', 1;
        END;

        IF @LastToken IS NULL
        BEGIN
            INSERT INTO dbo.SalesReports
                (ReportDate, TotalSales, LastFencingToken)
            VALUES
                (@ReportDate, @TotalSales, @FencingToken);
        END
        ELSE
        BEGIN
            UPDATE dbo.SalesReports
            SET TotalSales = @TotalSales,
                LastFencingToken =
                    CASE
                        WHEN @FencingToken > LastFencingToken
                            THEN @FencingToken
                        ELSE LastFencingToken
                    END
            WHERE ReportDate = @ReportDate;
        END;

        COMMIT TRANSACTION;
    END TRY
    BEGIN CATCH
        IF @@TRANCOUNT > 0
            ROLLBACK TRANSACTION;

        THROW;
    END CATCH;
END;

This example allows multiple writes using the same token because a single lock holder may legitimately perform more than one update. A token lower than the recorded token is rejected.

The row lock and transaction are important. Without them, two concurrent requests could both read the same old token and then race to update the record. The application must also handle the case where the row does not exist yet. In a production implementation, use appropriate transaction isolation and locking for that initial insert path, such as a pre-created resource row or a separate resource-token table with a unique key.

If you use a different database, implement the equivalent atomic comparison and update with that database's transaction and concurrency mechanisms.

Fencing Tokens vs. Lock Expiration and Retries

These mechanisms solve different problems. They are often used together.

Mechanism	Main purpose	Important limitation
Distributed lock	Coordinates access to a shared resource	A worker can continue after its lease expires
Lease expiration	Allows another worker to acquire an abandoned lock	Does not stop the previous worker from running
Cancellation token	Requests that application work stop	Cannot guarantee that blocked or remote operations stop immediately
Retry logic	Recovers from temporary failures	May repeat an operation that already succeeded
Fencing token	Lets storage reject writes from older lock acquisitions	Requires ordered tokens and enforcement at the resource
Idempotency key	Prevents duplicate logical operations	Does not establish which lock holder is newer

For many applications, a distributed lock reduces unnecessary contention, while a fencing token provides the final protection against stale writers. Idempotency can provide an additional layer when a repeated operation would otherwise create duplicate effects.

Common Mistakes to Avoid

Generating tokens locally. Two application instances can independently generate values that do not reflect acquisition order. Use a trusted coordination mechanism or another atomic, durable sequence.

Checking the token only in application memory. The application instance that accepted the latest token may restart or another instance may handle the next request. The resource must retain the accepted token in shared storage.

Updating the token separately from the data. If the token check and business write happen in separate transactions, another worker may intervene between them. Perform the validation and protected write atomically.

Using a lock without protecting every write path. A background job, administrative endpoint, or maintenance script that bypasses the token check can still overwrite current data. Apply the same rule to all writers that can modify the protected resource.

Assuming the token cancels old work. A fencing token does not stop an old worker from consuming CPU or calling other services. It prevents a compliant resource from accepting a stale write. Other side effects may need their own fencing checks or idempotency controls.

Reusing tokens across unrelated resources without a defined ordering model. Tokens need to be ordered for the resource they protect. If multiple resources have independent lock sequences, compare tokens within the correct resource scope.

Best Practices for Production Systems

Make the token mandatory. Avoid optional token parameters that allow a caller to bypass fencing.
Use a durable, monotonic sequence. A newly granted lock must receive a token greater than earlier acquisitions for the same resource, including after service restarts and failover.
Validate at the write boundary. The database or protected service should enforce the token rather than relying only on the worker.
Keep validation and writes atomic. Use transactions, row-level locking, or a suitable conditional-write operation.
Handle stale-write errors explicitly. Record enough information to diagnose rejected operations, but do not automatically retry with a new token unless the operation is safely re-evaluated under the new ownership.
Test expiration and concurrency. Simulate a worker pausing beyond its lease, a second worker acquiring the lock, and the original worker attempting to write afterward.
Protect external side effects too. If a worker calls another service, that service must support an equivalent protection mechanism or an appropriate idempotency strategy.

Advantages and Limitations

Fencing tokens are useful because they protect the resource even when a worker's lease expires unexpectedly. They can reduce the risk of stale writes during network interruptions, long-running tasks, process pauses, and lock-service failover.

However, they add implementation complexity. The lock provider must issue ordered tokens, storage must validate them, and every relevant write path must follow the same rule. They also do not provide a complete solution for exactly-once processing or for side effects in external systems that do not check the token.

Summary

Distributed locks coordinate workers, but they cannot guarantee that an expired lock holder has stopped executing. A delayed worker can resume and overwrite newer data unless the protected resource rejects stale writes.

Fencing tokens solve this by assigning an increasing number to each lock acquisition. The storage system records the highest accepted token and rejects any later write carrying a lower value.

For a reliable .NET implementation, acquire a token from a trusted lock provider, pass it with every protected write, and validate it atomically at the database or resource boundary. Combined with idempotent processing and concurrency testing, fencing tokens make distributed workloads more resilient to the failures that occur in real systems.