Resilience Patterns: Circuit Breakers, Retries, and Timeout Budgets
Configured poorly, automated retries multiply outages tenfold; configured correctly with circuit breakers and deadline propagation, gateways contain faults gracefully.
Why Unbounded Timeouts Cause Cascading Outages
Stalled downstream calls without timeouts hold proxy sockets and worker threads open until memory and file descriptors exhaust across the cluster. Explicit connect, read, and total timeouts convert slow hangs into fast failures.
Set hierarchical budgets—a short connect timeout (250–500 ms) inside a bounded route timeout—and propagate remaining deadlines downstream so backends abort orphaned queries.
- Set tight connect timeouts (250–500 ms) to detect dead pods rapidly
- Keep route timeouts strictly below client/load-balancer idle timeouts
- Return HTTP 504 Gateway Timeout cleanly when budgets expire
Safe Retries and Retry Amplification
Naive retries across three service tiers multiply a single failure into 27× load hitting an overloaded database. Restrict gateway retries strictly to idempotent `GET`/`HEAD` calls or `Idempotency-Key` requests on connect/503 errors.
Cap total retries with a 10–20% retry budget and jittered backoff, ensuring `per_try_timeout × (1 + retries)` fits inside the overall route timeout.
- Never retry non-idempotent `POST` timeouts blindly
- Cap retry volume via a <= 20% cluster retry budget
- Verify per-try retries fit comfortably inside total route timeout
Circuit Breakers and Passive Outlier Detection
Circuit breakers halt traffic with HTTP 503 when upstream errors breach thresholds, giving overloaded dependencies breathing room before half-open recovery probes.
Passive outlier detection complements circuit breakers by ejecting only the single failing pod from load balancing while keeping healthy pods serving.
- Pair active `/healthz` checks with passive 5xx outlier ejection
- Cap maximum ejected pool share (e.g., 50%) so clusters stay reachable
- Use half-open limits to ramp recovery traffic gradually
Adaptive Concurrency and Queue Caps
Pair circuit breakers with max-pending-request caps so excess traffic fails fast at the edge with `Retry-After` guidance instead of queueing behind degraded backends.
Resilience Mechanism Scope & Configuration Guide
| Mechanism | Failure Mode Addressed | Scope | Key Guardrail | Anti-Pattern to Avoid |
|---|---|---|---|---|
| Connect Timeout | Unreachable / dead backend IP | Per TCP/TLS dial | 250–500 ms max | Waiting 30 s OS default TCP SYN timeout |
| Per-Try + Route Timeout | Hung database / slow upstream | Per request lifecycle | Route timeout > retry sum | Infinite proxy read timeouts |
| Idempotent Retries | Transient pod restart / reset | `GET`/`HEAD` & keyed ops | Max 1–2 retries + 20% budget | Retrying non-idempotent `POST` timeouts |
| Outlier Detection | Single degraded backend pod | Per upstream endpoint | Eject max 50% of pool | Letting one poisoned pod fail 10% of all calls |
| Circuit Breaker | Total downstream outage | Per upstream cluster | Fail fast + half-open probe | Hammering a recovering service with full traffic |
Test retry and circuit-breaker thresholds via controlled fault-injection experiments in staging.
Worked Illustration: Calculating a Safe Retry and Timeout Budget
This worked illustration sizes timeout and retry bounds for a critical `GET /v1/inventory/{sku}` route backed by 6 pods.
- Step 1 (Overall Budget): Establish a 1,500 ms total route timeout SLA at the gateway.
- Step 2 (Per-Try Sizing): Set `connect_timeout: 250 ms`, `per_try_timeout: 600 ms`, and `max_retries: 1` (1,200 ms worst case, leaving 300 ms headroom).
- Step 3 (Retry Filter): Retry solely on `connect-failure`, `reset`, or HTTP `503` within a 20% cluster retry budget—never on HTTP 4xx or 500.
- Step 4 (Outlier Ejection): Eject any pod returning 5 consecutive 5xx/connect errors for 30 s (max 50% pool ejection).
Engineering Illustration Summary: Single-pod blips recover transparently within 850 ms while systemic outages fail fast without retry storms.
Gateway Resilience & Fault-Isolation Checklist
- Define explicit connect, per-try, and total route timeouts on every upstream route.
- Restrict automatic retries strictly to safe, idempotent methods or keyed operations.
- Enforce a retry budget (<= 20% active traffic) to stop retry amplification.
- Enable passive outlier detection to eject degraded pods faster than active polls.
- Cap maximum ejected host percentage so blips never empty the entire pool.
- Validate circuit-breaker tripping and half-open recovery with fault injection.
Frequently Asked Questions
Why avoid retrying HTTP 500 Internal Server Error responses automatically?
HTTP 500 often reflects a deterministic bug or overloaded database query; retrying it multiplies load on failing dependencies.
What is retry amplification?
When multiple hops (client, gateway, internal services) each retry independently, failures multiply exponentially down the call chain.
How do passive outlier detection and active health checks differ?
Active checks poll `/healthz` every few seconds, while passive detection watches live traffic responses and ejects bad pods immediately.
What HTTP status should an open circuit breaker return?
Return HTTP `503 Service Unavailable` with a `Retry-After` header signaling temporary unavailability.
Primary Standards & Specifications
- RFC 9110 — HTTP Semantics (Idempotent Methods & 503/504 Status Codes, IETF)
- Site Reliability Engineering — Handling Overload & Cascading Failures Principles
- Kubernetes Gateway API — HTTPTimeouts & Backend Traffic Policy Specifications