Availability & Fault Tolerance
Intermediate · 19 min read · High-Level Design, Reliability, Security & Operations
SLAs and nines, redundancy, timeouts, retries with backoff, circuit breakers and idempotency.
At scale, failure isn't an exception — it's a constant. Disks die, processes crash, deployments go wrong, networks drop packets, and dependencies slow down. With thousands of machines, something is always broken.
This lesson covers how availability is measured, how redundancy provides it, and the patterns — timeouts, retries, circuit breakers, bulkheads, idempotency, graceful degradation — that stop one failure from cascading into an outage.
Measuring availability
Availability = the fraction of time (or of requests) the system works correctly.
Redundancy: eliminating single points of failure
A single point of failure (SPOF) is any component whose failure takes the whole system down. Walk through every box in your design and ask: what if this dies?
Patterns that stop failures from spreading
The most dangerous failures aren't crashes — they're slowness. A crashed dependency fails fast; a slow one ties up threads, connections and memory in every caller until they all fall over. This is a cascading failure.
Detecting failures
Building confidence
Interview questions
- Your service calls five dependencies, each 99.9% available. What's your maximum availability, and how would you improve it?
- What is a retry storm and how do you prevent one?
- Explain how a circuit breaker works and why it helps.
- What's the difference between RPO and RTO? Give an example.
Practice
Write your answer before opening the suggested one. Anything you mark "Partly" or "Missed it" comes back in your review deck.
Further reading
Key takeaways
- Availability is measured in nines; each extra nine is ~10× harder. Use SLOs and error budgets to manage reliability explicitly.
- Dependencies in series multiply availability down; redundancy in parallel raises it — but only if failures are independent.
- Eliminate single points of failure and spread redundancy across failure domains (zones, regions).
- Slowness causes cascading failures: use timeouts, bounded retries with backoff and jitter, circuit breakers and bulkheads.
- Design graceful degradation, shed load under pressure, make operations idempotent, and test failure paths regularly.