A reliable POS API integration layer in ASP.NET Core rests on three rules. Every write request carries an idempotency key, so a retried sale never charges twice. Every incoming webhook is verified with an HMAC signature, saved to an inbox table and processed later, so duplicate or late events do no harm. Every outbound call goes through a retry policy with exponential backoff and jitter, and POST calls are retried only when they carry an idempotency key.
Key takeaways
Networks fail in the middle of requests, so the client often cannot tell whether a sale went through. Idempotency keys turn "did it work?" into a safe repeat.
POS platforms and payment providers deliver webhooks at least once, which means you will receive some events twice. Store the event ID with a unique index and treat repeats as a no-op.
Retries without idempotency create duplicate orders. Retries with idempotency are safe.
Return 200 OK to a webhook quickly and do the real work in a background worker.
Contents
Why do POS integrations break in production?
What does the integration layer look like?
How do you add idempotency keys to an ASP.NET Core endpoint?
How do you retry calls to a POS or payment API safely?
How do you receive and verify POS webhooks?
How do you process webhook events without duplicates or stale data?
How do you test the layer before go-live?
Security checklist
FAQ
Why do POS integrations break in production?
A point of sale talks to many systems at once: the payment gateway, inventory, accounting, loyalty, online ordering and sometimes a kitchen display. In a demo, every call succeeds on the first try. In a busy store on a Saturday, the picture changes.
Three failure patterns cause most of the damage:
The lost response. The terminal sends "create payment", the gateway charges the card, and the response is lost to a timeout. The cashier taps again. The customer is charged twice.
The repeated webhook. The POS platform sends "inventory.updated", your server is slow to answer, and the platform sends it again. Stock is reduced twice.
The late webhook. An older stock update arrives after a newer one and overwrites the correct number.
None of these are rare bugs. They are normal behaviour of distributed systems, and the integration layer has to expect them. If you want a broader view of the systems a POS connects to, the types of APIs involved and how rollout is usually phased, this guide to POS API integration covers the business side well. This article stays on the code.
What does the integration layer look like?
The layer sits between your POS front end (terminal, tablet or web checkout) and the outside services. In this walkthrough it has four parts:
Part | Job | Failure it handles |
|---|
Idempotency filter | Stores each Idempotency-Key with the request hash and the response | Lost responses, double taps, client retries |
Resilient HTTP client | Calls the gateway or POS platform with retries, timeouts and a circuit breaker | Short network drops, 5xx errors, rate limits |
Webhook endpoint | Verifies the signature, saves the raw event, answers fast | Forged events, replayed events, slow handlers |
Inbox processor | Works through saved events in order, with backoff on failure | Duplicate delivery, out-of-order events, downstream outages |
The samples target .NET 8 with Minimal APIs and EF Core on SQL Server. The same ideas work in controllers.
How do you add idempotency keys to an ASP.NET Core endpoint?
An idempotency key is a unique value, usually a UUID, that the client creates once per business action and sends with every attempt of that action. The server saves the first result under that key. When the same key comes back, the server returns the saved result instead of running the action again.
Three rules make it safe:
Same key and same body: return the stored response.
Same key and a different body: reject with 422, because the client reused a key by mistake.
Same key while the first request is still running: return 409 so the client waits and tries again.
Start with a table to hold the keys:
csharp
public sealed class IdempotencyRecord
{
public string Key { get; set; } = default!;
public string RequestHash { get; set; } = default!;
public int? StatusCode { get; set; } // null = still in progress
public string? ResponseBody { get; set; }
public DateTimeOffset CreatedAt { get; set; }
}
public sealed class PosDbContext(DbContextOptions<PosDbContext> options) : DbContext(options)
{
public DbSet<IdempotencyRecord> IdempotencyRecords => Set<IdempotencyRecord>();
public DbSet<InboxEvent> InboxEvents => Set<InboxEvent>();
public DbSet<Sale> Sales => Set<Sale>();
protected override void OnModelCreating(ModelBuilder b)
{
b.Entity<IdempotencyRecord>().HasKey(r => r.Key);
b.Entity<IdempotencyRecord>().Property(r => r.Key).HasMaxLength(64);
b.Entity<InboxEvent>().HasIndex(e => e.EventId).IsUnique();
}
}
The primary key on Key does the heavy lifting. Two requests with the same key cannot both insert a row, so the database settles any race for you.
Next, an endpoint filter that claims the key before the handler runs and saves the result after:
csharp
public sealed class IdempotencyFilter<TRequest>(PosDbContext db) : IEndpointFilter
{
private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web);
public async ValueTask<object?> InvokeAsync(
EndpointFilterInvocationContext ctx, EndpointFilterDelegate next)
{
var key = ctx.HttpContext.Request.Headers["Idempotency-Key"].ToString();
if (string.IsNullOrWhiteSpace(key) || key.Length > 64)
return Results.Problem("A valid Idempotency-Key header is required.", statusCode: 400);
var body = ctx.Arguments.OfType<TRequest>().First();
var hash = Convert.ToHexString(
SHA256.HashData(JsonSerializer.SerializeToUtf8Bytes(body, Json)));
var record = new IdempotencyRecord
{
Key = key, RequestHash = hash, CreatedAt = DateTimeOffset.UtcNow
};
db.IdempotencyRecords.Add(record);
try
{
await db.SaveChangesAsync(); // claim the key
}
catch (DbUpdateException)
{
db.Entry(record).State = EntityState.Detached;
var existing = await db.IdempotencyRecords.AsNoTracking()
.SingleAsync(r => r.Key == key);
if (existing.RequestHash != hash)
return Results.Problem("This key was already used with a different request.", statusCode: 422);
if (existing.StatusCode is null)
return Results.Problem("A request with this key is still in progress.", statusCode: 409);
return Results.Content(existing.ResponseBody, "application/json",
statusCode: existing.StatusCode);
}
try
{
var result = await next(ctx);
if (result is IStatusCodeHttpResult { StatusCode: int code } and IValueHttpResult value
&& code < 500)
{
record.StatusCode = code;
record.ResponseBody = JsonSerializer.Serialize(value.Value, Json);
}
else
{
db.IdempotencyRecords.Remove(record); // let the client retry
}
await db.SaveChangesAsync();
return result;
}
catch
{
db.IdempotencyRecords.Remove(record);
await db.SaveChangesAsync();
throw;
}
}
}
A few choices here are deliberate. The filter hashes the bound request object rather than the raw body, so whitespace differences do not trigger a false 422. Server errors (5xx) and exceptions release the key, because the action did not finish and the client should be allowed to try again. Client errors such as a declined card are stored, because repeating the same request should give the same answer.
Attach the filter to the sale endpoint and pass the same key on to the payment gateway:
csharp
app.MapPost("/api/sales", async (
CreateSaleRequest req, HttpContext http, PosDbContext db,
PaymentGatewayClient gateway, CancellationToken ct) =>
{
var key = http.Request.Headers["Idempotency-Key"].ToString();
var payment = await gateway.CreatePaymentAsync(
new GatewayPaymentRequest(req.AmountMinor, req.Currency, req.CardToken),
idempotencyKey: key, ct);
var sale = new Sale
{
Id = Guid.NewGuid(), StoreId = req.StoreId, AmountMinor = req.AmountMinor,
Currency = req.Currency, PaymentId = payment.Id, CreatedAt = DateTimeOffset.UtcNow
};
db.Sales.Add(sale);
await db.SaveChangesAsync(ct);
return TypedResults.Created($"/api/sales/{sale.Id}",
new SaleResponse(sale.Id, payment.Id, payment.Status));
})
.AddEndpointFilter<IdempotencyFilter<CreateSaleRequest>>();
Forwarding the key matters. If your own server crashes after the gateway charged the card but before the sale row was saved, the client retries, your filter lets the request through again, and the gateway recognises the key and returns the original payment instead of charging twice. Many payment and POS APIs accept an idempotency key, though the field name and where it goes (header or body) differ by provider, so check the docs for yours.
Keep amounts in minor units (cents, paise) as integers. Floating-point money leads to reconciliation errors that are painful to trace.
Finally, clean up old keys. A scheduled job that deletes records older than 24 to 72 hours is usually enough, depending on how long your clients may retry.
How do you retry calls to a POS or payment API safely?
Retries fix short failures: a dropped connection, a 503 during a deploy, a 429 when you hit a rate limit. They also cause harm when they hit a non-idempotent call. The rule is simple: retry GET calls freely, retry POST calls only when they carry an idempotency key.
In .NET 8, the Microsoft.Extensions.Http.Resilience package gives you retries, timeouts and a circuit breaker in one call:
csharp
builder.Services.AddHttpClient<PaymentGatewayClient>(c =>
{
c.BaseAddress = new Uri(builder.Configuration["Gateway:BaseUrl"]!);
})
.AddStandardResilienceHandler(o =>
{
o.Retry.MaxRetryAttempts = 4;
o.Retry.BackoffType = DelayBackoffType.Exponential;
o.Retry.UseJitter = true;
o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);
o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(30);
});
The standard handler retries on transient results such as 5xx, 408 and 429, and it respects the Retry-After header by default. Jitter spreads retries out so that fifty terminals do not all hit the gateway at the same second after an outage.
The typed client adds the key to every attempt. Because the key lives on the request message, each retry sends the same key:
csharp
public sealed class PaymentGatewayClient(HttpClient http)
{
public async Task<GatewayPayment> CreatePaymentAsync(
GatewayPaymentRequest body, string idempotencyKey, CancellationToken ct)
{
using var msg = new HttpRequestMessage(HttpMethod.Post, "v1/payments")
{
Content = JsonContent.Create(body)
};
msg.Headers.Add("Idempotency-Key", idempotencyKey);
using var res = await http.SendAsync(msg, ct);
res.EnsureSuccessStatusCode();
return (await res.Content.ReadFromJsonAsync<GatewayPayment>(cancellationToken: ct))!;
}
}
If you call an endpoint that does not support idempotency keys, turn retries off for that client, or retry only on errors where you know the request never reached the server (for example, a DNS failure or a refused connection).
How do you receive and verify POS webhooks?
Webhooks tell you what changed on the POS platform: a payment settled, a refund was issued, stock moved, a menu item was updated. Your endpoint has three jobs, in this order:
Prove the event came from the platform.
Save it.
Answer quickly.
Most providers sign each webhook with HMAC-SHA256 using a shared secret. The exact signed string and header names vary. Some sign only the body, some sign a timestamp plus the body, and some include the notification URL. The helper below uses a common timestamp-plus-body scheme; adjust the payload string to match your provider's docs.
csharp
public static class WebhookSignature
{
public static bool IsValid(string rawBody, string timestamp, string signatureHex,
string secret, TimeSpan tolerance)
{
if (!long.TryParse(timestamp, out var unix) || unix <= 0 || unix > 253402300799)
return false;
var age = DateTimeOffset.UtcNow - DateTimeOffset.FromUnixTimeSeconds(unix);
if (age.Duration() > tolerance)
return false; // stops replayed requests
var expected = HMACSHA256.HashData(
Encoding.UTF8.GetBytes(secret),
Encoding.UTF8.GetBytes($"{timestamp}.{rawBody}"));
byte[] provided;
try { provided = Convert.FromHexString(signatureHex); }
catch (FormatException) { return false; }
return CryptographicOperations.FixedTimeEquals(expected, provided);
}
}
Two details are easy to get wrong. First, verify against the raw body exactly as received. If you deserialize and serialize it again, spacing or field order can change and the signature will fail. Second, use CryptographicOperations.FixedTimeEquals instead of == or SequenceEqual, so the comparison time does not leak how many bytes matched.
The endpoint itself stays small:
csharp
app.MapPost("/webhooks/pos", async (
HttpRequest request, PosDbContext db, IOptions<WebhookOptions> opts, CancellationToken ct) =>
{
using var reader = new StreamReader(request.Body, Encoding.UTF8);
var rawBody = await reader.ReadToEndAsync(ct);
var ok = WebhookSignature.IsValid(
rawBody,
request.Headers["X-Pos-Timestamp"].ToString(),
request.Headers["X-Pos-Signature"].ToString(),
opts.Value.SigningSecret,
TimeSpan.FromMinutes(5));
if (!ok) return Results.Unauthorized();
var envelope = JsonSerializer.Deserialize<WebhookEnvelope>(
rawBody, new JsonSerializerOptions(JsonSerializerDefaults.Web));
if (envelope is null || string.IsNullOrEmpty(envelope.Id))
return Results.BadRequest();
if (await db.InboxEvents.AnyAsync(e => e.EventId == envelope.Id, ct))
return Results.Ok(); // duplicate delivery
db.InboxEvents.Add(new InboxEvent
{
EventId = envelope.Id, Type = envelope.Type, Payload = rawBody,
OccurredAt = envelope.OccurredAt, ReceivedAt = DateTimeOffset.UtcNow
});
try { await db.SaveChangesAsync(ct); }
catch (DbUpdateException) { /* lost a race with a parallel duplicate: safe to ignore */ }
return Results.Ok();
});
The AnyAsync check handles the common duplicate. The unique index on EventId handles the rare case where two copies arrive at the same moment. In production, check the database error code in that catch so that real failures are not hidden.
Answer within a few seconds. Many platforms treat a slow answer as a failure and send the event again, which adds load right when your system is already struggling.
How do you process webhook events without duplicates or stale data?
Saving events to an inbox table and processing them in the background is called the inbox pattern. It splits "we received it" from "we acted on it", so a slow accounting API or a database lock never causes the platform to resend.
csharp
public sealed class InboxEvent
{
public long Id { get; set; }
public string EventId { get; set; } = default!;
public string Type { get; set; } = default!;
public string Payload { get; set; } = default!;
public DateTimeOffset OccurredAt { get; set; }
public DateTimeOffset ReceivedAt { get; set; }
public DateTimeOffset? ProcessedAt { get; set; }
public DateTimeOffset? NextAttemptAt { get; set; }
public int Attempts { get; set; }
public string? LastError { get; set; }
}
public sealed class InboxProcessor(IServiceScopeFactory scopes, ILogger<InboxProcessor> log)
: BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(2));
while (await timer.WaitForNextTickAsync(stoppingToken))
{
using var scope = scopes.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<PosDbContext>();
var now = DateTimeOffset.UtcNow;
var batch = await db.InboxEvents
.Where(e => e.ProcessedAt == null && e.Attempts < 10
&& (e.NextAttemptAt == null || e.NextAttemptAt <= now))
.OrderBy(e => e.OccurredAt)
.Take(50)
.ToListAsync(stoppingToken);
foreach (var evt in batch)
{
try
{
await EventHandlers.HandleAsync(evt, db, stoppingToken);
evt.ProcessedAt = DateTimeOffset.UtcNow;
}
catch (Exception ex)
{
evt.Attempts++;
evt.NextAttemptAt = DateTimeOffset.UtcNow.AddSeconds(Math.Pow(2, evt.Attempts));
evt.LastError = ex.Message;
log.LogWarning(ex, "Inbox event {EventId} failed (attempt {Attempts})",
evt.EventId, evt.Attempts);
}
}
await db.SaveChangesAsync(stoppingToken);
}
}
}
Register it with builder.Services.AddHostedService<InboxProcessor>();. Events that fail ten times stay in the table with their last error, which gives your support team a clear dead-letter list to review.
Ordering by OccurredAt helps, but it does not guarantee order, because events can arrive late. Guard each update with the event time so an old event cannot overwrite newer data:
csharp
// inside the "inventory.updated" handler
var item = await db.StockItems.SingleAsync(s => s.Sku == payload.Sku, ct);
if (item.LastEventAt is { } last && last >= evt.OccurredAt)
return; // older than what we already have
item.Quantity = payload.Quantity;
item.LastEventAt = evt.OccurredAt;
If you run more than one instance of the service, two workers can pick up the same rows. Either run the processor on a single instance, or claim rows with a lock hint (on SQL Server, UPDLOCK, READPAST in a raw query) before handling them.
Webhooks can also be lost entirely, for example when a platform gives up after its retry window. A nightly reconciliation job that pulls the day's payments and stock levels through the platform's list endpoints, and compares them with your tables, catches anything that slipped through.
How do you test the layer before go-live?
Use WebApplicationFactory<Program> and an in-memory or containerised database, and cover these cases at minimum:
Test | Expected result |
|---|
Same Idempotency-Key, same body, sent twice | Second call returns the stored body and status; one payment at the gateway |
Same key, different body | 422
|
Two parallel calls with the same key | One 201, one 409 |
Missing key | 400
|
Gateway returns 503 twice, then 200 | One successful sale; gateway received the same key three times |
Webhook with a wrong signature | 401
|
Webhook with a timestamp older than five minutes | 401
|
Same webhook event delivered twice | One inbox row; handler runs once |
Older stock event processed after a newer one | Quantity keeps the newer value |
For the gateway, a fake HttpMessageHandler that fails a set number of times and records the headers it receives is enough to prove the retry and key-forwarding behaviour.
Security checklist
Keep webhook secrets and API keys in a secret store (Azure Key Vault, AWS Secrets Manager or user secrets in development), never in appsettings.json in source control.
Support two active webhook secrets during rotation so you can switch without dropping events.
Give each integration its own API credentials with the smallest permissions it needs. A stock sync does not need refund access.
Send card data only as tokens from the payment provider. Raw card numbers in your API widen your PCI DSS scope.
Log event IDs, keys and status codes, but never full card data, secrets or signatures.
Enforce HTTPS on the webhook endpoint and apply request size limits so a large payload cannot exhaust memory.
FAQ
What is an idempotency key in a POS API integration? An idempotency key is a unique value the client creates once for a business action, such as a sale, and sends with every retry of that action. The server stores the first result under the key and returns it for repeats, so the action runs only once.
Is it safe to retry a POST request to a payment API? A POST retry is safe only when the request carries an idempotency key that the payment API honours. Without a key, a retry after a lost response can create a second charge.
How do I verify POS webhook signatures in ASP.NET Core? Read the raw request body, rebuild the signed string your provider documents, compute an HMAC-SHA256 with your shared secret, and compare it with the signature header using CryptographicOperations.FixedTimeEquals. Also reject requests whose timestamp is outside a short window.
Why do I receive the same webhook more than once? Most platforms deliver webhooks at least once and resend when your endpoint is slow or returns an error. Store each event ID with a unique index and ignore repeats.
Should webhook processing happen inside the HTTP request? No. Save the event, return 200 OK and process it in a background worker. This keeps responses fast and stops the platform from resending events while your downstream systems are busy.
How long should I keep idempotency keys? Keep them at least as long as your clients may retry a request. For most POS flows, 24 to 72 hours is enough, after which a cleanup job can delete them.
Join the conversation! Your thoughts help the community grow.