A slow dependency will take down your service faster than a broken one.
Circuit breakers borrow their name from electrical engineering: when a downstream dependency shows signs of failure, the breaker trips open and further calls fail immediately rather than piling up. This protects your service from thread exhaustion, memory blowup, and the cascading collapse that occurs when everything waits on a slow dependency. Together with sensible timeouts and jittered retries, circuit breakers are the minimum viable resilience toolkit for any service that calls another service.
Closed — normal operation, requests pass through, failures are counted in a rolling window. Open — after error rate crosses a threshold (e.g., 50% of the last 20 requests failed), further calls fail immediately with a 'circuit open' error rather than hitting the dependency. Half-open — after a cool-down period (typically 30-60 seconds), a small number of probe requests are allowed through; if they succeed, the breaker closes and normal traffic resumes; if they fail, back to open. The point is not perfect availability — it is preventing your service from crashing while the dependency recovers.
Every network call needs an explicit timeout. Default HTTP client timeouts in most languages are absurdly long (60 seconds, or unbounded). Set: connection timeout 1-3 seconds, read timeout 2-10 seconds depending on the call, total request timeout no more than 30 seconds. Cascade timeouts — a service being called by a user with a 30s budget should not be waiting 30s on its own downstream call; the deeper the chain, the shorter the timeouts. Enforce timeouts at every hop; passing a deadline header through the call chain is the mature version of this pattern.
Naive retries (fixed delay, unlimited) turn one failed request into 10 and take out the dependency instead of protecting it. Correct pattern: (1) Retry only idempotent operations (GET, PUT, DELETE — never POST unless you have an idempotency key). (2) Cap retries at 2-3 attempts total. (3) Exponential backoff with jitter (delay = base * 2^attempt + random 0-100ms) — jitter prevents thundering herd. (4) Do NOT retry through a tripped circuit breaker; let it fail fast. (5) Distinguish retriable errors (network, 5xx, 429) from non-retriable (4xx). (6) Give up gracefully — return a partial result or degraded response, not an infinite spinner.
The bulkhead pattern: isolate resources for each downstream so one bad dependency cannot exhaust the entire pool. Practically: one connection pool per external service (not one shared pool), separate thread pools or async concurrency limits per client, per-tenant quotas on shared resources. Load shedding: when the service is overloaded, return 503 with Retry-After for the lowest-priority requests rather than degrading service quality for everyone. Combined, bulkheads and shedding preserve the ability to serve the requests you can, instead of failing all requests together.
Metrics per breaker: state (open/closed/half-open), success rate, failure rate, request count, average latency. Alert on 'breaker is open' as a signal that a downstream is unhealthy AND that your service is currently rejecting traffic to that downstream. Libraries: Resilience4j (Java/Kotlin), Polly (.NET), Hystrix (deprecated), gobreaker (Go), tenacity + circuitbreaker (Python), opossum (Node). Service mesh (Istio, Linkerd) can add breakers without library changes for gRPC/HTTP. Roll out on the highest-risk dependency first (external API, slow database, third-party payment provider), verify metrics, expand from there.
Investor directory · Fundraising library · Articles A–Z · Company funding database