Observability

Intermediate · 15 min read · High-Level Design, Reliability, Security & Operations

Metrics, logs, traces, SLO-based alerting and debugging distributed systems.

A distributed system you can't see into is a system you can't operate. When checkout latency doubles at 2 AM, you need to know which of 40 services is responsible, for which users, and why — in minutes. Observability is the ability to answer such questions from the telemetry a system emits.

The three pillars

They work best linked together: an alert fires on a metric → a dashboard narrows it to one service and endpoint → an exemplar trace shows the slow span → logs for that trace ID show the error.

Metrics

The four golden signals (Google SRE) for every service:

Logs

Pipeline: services write to stdout → an agent (Fluent Bit, Vector) ships logs → a buffer (Kafka) → storage and search (Elasticsearch/OpenSearch, Loki, ClickHouse, or a managed service).

Distributed tracing

Alerting on SLOs

Alerting on raw symptoms ("CPU above 80%") produces noisy pages that often don't affect users. Better: alert when users are being hurt, as measured against your SLO.

Observability in a design interview

When you finish a design, a short observability paragraph goes a long way:

Interview questions

  • Why monitor p99 latency rather than average latency?
  • How does distributed tracing work across services and message queues?
  • What is a burn-rate alert and why is it better than a threshold alert?

Practice

Write your answer before opening the suggested one. If you mark it "Partly" or "Missed it", it comes back in your review deck.

Further reading

Key takeaways

  • Metrics detect, traces locate, logs explain — link them with trace IDs.
  • Monitor the four golden signals plus business metrics; use percentiles from histograms.
  • Keep metric labels low-cardinality; put per-request detail in logs and traces.
  • Propagate trace context through every hop, including queues; sample sensibly.
  • Alert on SLO burn rate with actionable, runbook-backed pages.

Previous: Availability & Fault Tolerance · Next: Multi-Region Architecture