The payment gateway starts timing out. The HTTP client retries, the queue retries, and workers pile up waiting on calls that won't come back. Soon pages that have nothing to do with payments are slow as well, and one sick dependency has dragged down the whole app.
This drill protects an online marketplace's checkout and login with timeouts, circuit breakers, bulkheads and fallbacks, so a flapping payment or SMS provider stays a contained problem.
Core concepts
When a dependency such as a payment gateway, an SMS provider or even Redis starts failing or slowing down, blind retries make things worse. Requests queue up, workers stay busy waiting, and the healthy parts of the app go down with the sick one.
The first defence is a timeout, a cap on how long a single call may wait.
A circuit breaker works like the trip switch in a fuse box. It watches recent failures, and past a threshold it opens: for a cool-down period every call fails immediately without touching the dependency. After the cool-down it goes half-open and lets one probe request through. If the probe succeeds, the breaker closes and normal traffic resumes.
Figure 1: The three breaker states and what moves you between them.
Bulkheads give each dependency its own worker pool or queue, so SMS trouble can't starve checkout. Fallbacks return a reduced answer instead of an error, such as a cached value, "pay later", or a different OTP channel.
These pieces depend on each other. Timeouts without a breaker still stampede a sick dependency, and a breaker without timeouts never sees the calls that hang. Retries into an open circuit are wasted work unless you check the circuit before retrying or enqueuing.
Pause and think: You've set a 3 second timeout on the payment call but no circuit breaker. The provider is down. What does your system still do wrong?
Show answer
Every request still calls the dead provider and waits the full timeout before failing. Workers and PHP processes stay tied up, and the provider gets hammered while it's trying to recover. A breaker would fail fast without making the call at all.
The design problem
The marketplace (Laravel, 3 app servers, Redis, queues) calls an eSewa/Khalti-style payment API when an order is placed, and an SMS OTP provider for login and high-value checkout. Lately both flap: about 30% of calls time out for 2 to 5 minutes, then recover. HTTP client retries and queue job retries pile up, checkout p95 jumps from about 400 ms to several seconds, and workers clog on SMS jobs.
Decide where the timeouts live (HTTP client, job or gateway). Keep one circuit per dependency with its state in Redis so all 3 servers agree, and choose the thresholds, cool-down and half-open probe. Then define what checkout and login return while each circuit is open, and how all of this interacts with the Day 7 rate limits, Day 8 locks, Day 9 idempotency keys and queue retries.
Try the problem before reading the solution.
Questions to think about
Why share the circuit state in Redis instead of keeping a breaker on each server?
The payment circuit is open. Should checkout fail now, queue the payment for later, or mark the order payment_pending?
The SMS circuit is open. What should login do, and what should an NPR 50k checkout do?
Model solution
Timeouts and circuits
Every outbound HTTP call gets an explicit timeout; in Laravel's HTTP client that's a single ->timeout() call. Without it, the client waits up to its default of 30 seconds, which is far too long for a checkout request. Keep one circuit per dependency (payment and SMS) with its state in Redis, so all 3 servers trip and recover together. While a circuit is open, calls fail fast without reaching the provider, and after the cool-down one half-open probe decides whether it closes.
Figure 2: Shared circuit state in Redis, plus separate queues per dependency (bulkheads).
When the SMS circuit is open
Fall back to another channel, such as an in-app OTP if the app has one. If SMS is the only channel, show "SMS delayed, try again in a minute", which is a product decision. For high-value checkout I'd fail closed or require another factor, since protecting money matters more than conversion. Day 7 rate limits still apply to whichever channel you fall back to.
Retries
Allow 0 to 1 HTTP retries on payment calls unless you're passing the provider an idempotency key. Queue jobs should use exponential backoff and release themselves while the circuit is open, so workers don't spend their retry budget on guaranteed failures. How many attempts a job gets depends on your queue config, the job's $tries property or the worker's --tries option; with the default of one attempt, a released job is never run again, so set it (or retryUntil()) deliberately. The Day 8 lock still wraps only the stock change, on the success path after payment confirms.
If Redis (the circuit store) is down
Fall back to a local in-memory breaker on each server, which is still better than calling blindly. For payment, lean toward failing closed; for non-critical SMS such as promotions, fail open. It's the same reasoning as the Day 7 Redis-down case.
Observability
Log every state change in both directions, graph it, and alert when a circuit stays open too long. Make half-open probes visible as well, so you never have to guess why one request suddenly succeeded.
Pause and think: Why should a queued payment job release itself back onto the queue while the circuit is open, instead of just throwing an exception?
Show answer
Throwing counts as a failure, so the job burns through its retries on failures you already know are coming and ends up in failed_jobs. Releasing it with a delay puts it back to run once the circuit has had time to close. In Laravel a release still counts as an attempt, so limit the job with maxExceptions or a time-based retryUntil() instead of a low $tries.
Answers to the questions
With a breaker per server, each server has to discover the outage on its own and keeps hammering the provider until it does. A shared circuit trips once for everyone, and one probe speaks for all 3 servers.
Failing now is simple and honest, but you lose some sales. Queuing for later keeps the sale, but the user waits with no answer. payment_pending is the nicest experience, but you need reconciliation and stock holds that expire. For a wallet API that flaps for a few minutes, a clear "try again shortly" with the cart kept is usually the right call.
For login, switch channels or show a short delay, since locking everyone out is worse. For an NPR 50k checkout, fail closed or require another factor.
Interview one-liner
"Timeouts on every call, one Redis-backed circuit per dependency, bulkheaded queues that release jobs while a circuit is open, and a deliberate fail-open or fail-closed choice per feature, with money always failing closed."
Conclusion
A timeout limits how long one call can hurt you, and a shared circuit stops every server from hammering a provider that's already down; you need both. Whatever else you decide, anything that moves money should fail closed.
Try it yourself
Point a Laravel HTTP call at a local endpoint that sleeps longer than your timeout. Build a small breaker from three Redis keys: a failure counter with a short TTL, an open_until timestamp, and a probe lock.
Make the call fail fast while open_until is in the future, and let exactly one request through after it passes. Watch the logs while you start and stop the slow endpoint.
Join the conversation! Your thoughts help the community grow.