Timeouts, Retries and Circuit Breakers

A checkout request can wait on inventory, payment and shipping. If one dependency stalls, waiting forever ties up resources. Retrying without limits can multiply work on the already struggling dependency.

Design a complete deadline and failure policy. A timeout means the caller stopped waiting; it does not prove the remote operation never happened.

Suppose an operation must finish within 1,000 ms. Connection setup, remote processing, retries and later steps all share that budget. Giving every dependency an independent 1,000 ms timeout can violate the overall promise. Pass the remaining deadline and cancel work where supported.

Cancellation is not proof of rollback. A server might have committed before it received cancellation. Track the operation identity and expose a way to inspect its outcome when success is uncertain.

In this simplified worst case, every selected layer makes the same number of attempts, and every attempt reaches the next layer. Three layers making three attempts each can produce 27 calls at the deepest dependency.

Change layer count and attempts per layer. The model does not simulate success, deadlines or timing. It shows why independent retries at every layer can amplify an outage; real policies should retry in a deliberate place within a bounded budget.

Retry only failures the operation contract considers transient and only within its total budget. Reuse a stable idempotency key for effects such as payment; the server must atomically coordinate that key and effect. A new key on every attempt defeats deduplication.

Exponential backoff increases the wait between attempts. Jitter varies that wait to avoid synchronized retry bursts. Cap attempts and elapsed time, consider Retry-After where applicable, and avoid unbounded retry of permanent validation or permission failures.

A circuit breaker can stop calling an unhealthy dependency after a configured failure threshold, then allow limited probes. It does not repair the dependency or determine whether an earlier payment committed. Define safe fallback behaviour; showing a stale price may be tolerable while approving a payment from stale state is not.

A bulkhead isolates resource pools so one failing dependency cannot occupy every worker or connection. Bound queues and concurrency too. Resilience4j and Envoy provide practical mechanisms; integration and defaults still need testing.

Read AWS’s guidance on timeouts, retries, backoff and jitter. Observe attempted calls as well as useful completions so amplification is visible.

Continue with Retry with backoff and jitter, Circuit Breaker and Bulkhead to compare repeated attempts, health-based gating and resource isolation. None determines whether an earlier write committed.

Bound the full request, retry in a deliberate place and preserve operation identity. Isolation and breakers limit damage; they do not resolve uncertain writes.