Overview

Saturday, 09:59. A stadium show goes on sale at 10:00 on BoxOffice: 2 million fans for 60 thousand seats. At 10:04 the support channel fills with "checkout spins forever". The team opens the dashboards: every pod is green, CPU sits at 45%, the error rate is 0.3%, the health checks pass. And yet the median fan waits nine seconds to pay. Is it the gateway? The seat holds in inventory? The external payment provider? The database? Four teams join a call and each one shows a dashboard proving that their service is fine.

This is the moment observability exists for. The system is not down — it is slow in a way nobody predicted, and the only way out is to ask new questions of the running system and get answers in minutes, not days. This class is about designing a system so that those questions can be answered: which signals to emit, how to carry context across services, when to wake a human, and how to run the incident once someone is awake.

Roadmap

1. This overview — monitoring vs observability, the observability palette, the investigation loop, health checks, and what telemetry costs.
2. Signals — metrics, logs, traces and profiles; RED, USE and golden signals; cardinality; dashboards.
3. SLIs, SLOs and error budgets — reliability from the user's point of view: what to measure, which target to promise, and how to spend the budget.
4. OpenTelemetry — the vendor-neutral standard: context propagation, instrumentation, the Collector, sampling and backends.
5. Alerting — symptom-based alerts, SLO burn rate, multiwindow rules, routing and runbooks.
6. Incident Response — on-call, severities, incident roles, mitigation first, postmortems.
7. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Monitoring vs observability

The two words are often used as synonyms, but they describe different capabilities.

Monitoring watches a set of pre-defined indicators and tells you when one crosses a threshold: "error rate above 1%", "disk above 90%". It answers questions you knew to ask when you built the dashboard — the known unknowns.

Observability is a property of the system: how well you can infer its internal state from its external outputs¹. The term comes from control theory (Kalman, 1960); in software it means that, without shipping new code, you can ask a question you never thought of — "why are only checkouts with more than four tickets, paid with one card brand, slow since 10:02?" — and answer it from the telemetry the system already emits². Those are the unknown unknowns, and distributed systems are full of them.

Monitoring is not obsolete — it is a subset. You still need dashboards and alerts on known symptoms; observability is what lets you go from "the alert fired" to "this is the cause" when the cause is new.

The observability palette

In the System Design Playground, the OBSERVABILITY category has five components. Each one answers a different question, and each one is consumed by someone different — a human investigating, a human being woken up, or a machine making a routing decision.

All of them describe the same events from different angles. A single slow POST /checkout at 10:04 produces:

• a sample in the latency histogram of order (metric) — cheap, aggregated, no idea which checkout;
• a structured log line {"level":"WARN","msg":"payment call slow","trace_id":"4bf9…","duration_ms":8120} — precise, expensive at volume;
• a trace with spans for gateway → order → inventory → payment, the payment span taking 8.1 s — shows where;
• possibly a profile sample showing threads parked in the HTTP client — shows which code;
• and, if enough checkouts are slow for long enough, an alert that pages the on-call engineer.

The glue between them is context: the same trace_id in the log line, in the trace and attached to the metric sample (an exemplar). Without shared context, you have three disconnected tools; with it, you have one investigation.

The investigation loop

Observability pays off as a repeatable loop that moves from cheap, aggregated signals to expensive, detailed ones — narrowing the search at each step. Play the diagram:

flowchart LR
    A["Alert<br/>checkout SLO burning 14×"] --> D["RED dashboard<br/>which service? which route?"]
    D --> E["Exemplar<br/>one slow request's trace_id"]
    E --> T["Trace<br/>which span is slow?"]
    T --> L["Logs of that span<br/>why? (error, retry, timeout)"]
    L --> P["Profile<br/>which code path?"]
    P --> M["Mitigate<br/>rollback, breaker, failover"]
    M --> PM["Postmortem<br/>prevent the class of failure"]

1. Detect with an SLO-based alert on a symptom users feel (Alerting).
2. Localize with metrics: the RED dashboard shows POST /checkout p99 went from 400 ms to 9 s; other routes are normal.
3. Pick an example: a latency panel with exemplars links a slow sample to its trace_id.
4. Find the hop: the trace waterfall shows 8 of the 9 seconds inside the payment span, in the call to the external provider.
5. Explain: the logs of that span show connection pool exhausted, waited 7.9 s — the provider is slow, and payment has a pool of 20 connections.
6. Mitigate first, understand later (Incident Response): open the circuit breaker, put checkouts in a queue, or switch providers.
7. Learn: the postmortem asks why a slow provider could hold the whole checkout hostage — and adds a timeout, a bulkhead, and an alert on provider latency.

See it in the simulator

Try it. At t = 50 s one of the three catalog replicas dies. Press Run as it is: every service shows its utilization as "?" (nothing is measured), and for 30 s the load balancer keeps sending a third of the requests to the dead replica — about 33% errors from t = 51 s to 80 s, 93.8% availability over the run. Add Metrics and connect it to the load balancer, the app and the database: the numbers appear, but fans see exactly the same errors — measuring fixes nothing by itself. Then add a Health Check connected to the load balancer: the dead replica leaves the rotation after 3 s instead of 30 s, and availability rises to 99.4%. What the last component, Alerting, changes is in Alerting.

The full simulator is in the System Design Playground.

Health checks

A health check is the simplest signal and the only one consumed by machines: a load balancer, an orchestrator or a service discovery system calls an endpoint and decides whether an instance should receive traffic or be restarted. Because machines act on it automatically and at scale, a badly designed health check is one of the classic causes of total outages.

Kubernetes distinguishes three probes³, and the distinction is the whole point:

stateDiagram-v2
    [*] --> Starting : container created
    Starting --> Ready : startup ok, readiness ok
    Starting --> Restarting : startup budget exceeded
    Ready --> NotReady : readiness fails (overloaded, draining)
    NotReady --> Ready : readiness ok again
    Ready --> Restarting : liveness fails (deadlock)
    NotReady --> Restarting : liveness fails
    Restarting --> Starting : new container

With Spring Boot Actuator, the probes are separate health groups, exposed at /actuator/health/liveness and /actuator/health/readiness⁴:

# application.yaml (order service)
management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        readiness:
          include: readinessState
        liveness:
          include: livenessState

# Deployment (excerpt) — the order container
startupProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  periodSeconds: 5
  failureThreshold: 24        # up to 2 minutes to start
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3         # 30 s of failures before a restart
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

Shallow vs deep checks

A shallow check answers "is my process alive?". A deep check also calls the dependencies — the database, Redis, the payment provider. Deep checks look more honest, and that is exactly the trap.

Load balancers at the edge use the same idea (see Load balancing): they remove a backend after N failed checks and add it back after M successes — the hysteresis avoids flapping.

What telemetry costs

Observability is not free, and in an interview — or in the cloud bill — you must reason about its volume like any other workload. A back-of-the-envelope estimate for BoxOffice on a normal day, with illustrative assumptions:

At 100% trace sampling that is over 5 TB of traces a month — before the flash sale multiplies the traffic by 50. Hence the three cost levers you will see in this class:

• Sampling traces (keep the interesting ones — errors and slow requests — and a small share of the rest), see OpenTelemetry.
• Bounded cardinality for metrics (no user IDs in labels), see Signals.
• Levels, sampling and retention tiers for logs (DEBUG off in production, 7–30 days hot, then cheap object storage).

The observability concept map

The five observability components describe the same requests from different angles, and shared context (exemplars, trace_id) turns them into one investigation. The map links each palette component (green) to the concepts it relies on (blue) and to the classes that continue them:

flowchart LR
    svc["app · worker<br/>(BoxOffice services)"]:::comp --> metrics["metrics"]:::comp
    svc --> tracing["tracing"]:::comp
    svc --> logs["logs"]:::comp
    metrics -->|"exemplar"| tracing
    tracing -->|"trace_id"| logs
    metrics --> alerting["alerting"]:::comp
    hc["healthcheck"]:::comp -->|"probes"| svc
    metrics -.- c1["RED, USE, golden signals,<br/>histograms, cardinality"]:::tech
    logs -.- c2["structured logs,<br/>levels, redaction"]:::tech
    tracing -.- c3["traceparent, Collector,<br/>tail sampling"]:::tech
    alerting -.- c4["SLO burn rate, multiwindow,<br/>routing, runbooks"]:::tech
    hc -.- c5["liveness, readiness, startup,<br/>shallow vs deep"]:::tech
    c4 --> c6["incident roles, mitigate first,<br/>postmortems"]:::tech
    c4 -.- d1["SLI, SLO, error budget<br/>(class 9)"]
    c5 -.- d2["Kubernetes probes, LB checks<br/>(classes 4 and 11)"]
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Related

• Hands-on 8 — Observability: OpenTelemetry auto-instrumentation and grafana/otel-lgtm on BoxOffice; RED dashboard, traces and a burn-rate alert.
• Hands-on 9 — Resilience & chaos and Hands-on 10 — Kubernetes: probes, Toxiproxy faults, and watching them in the telemetry.
• Reliability and SLOs: SLIs, SLOs and error budgets — what the alerts of this class enforce.
• Performance engineering: load tests are only useful if you can see why the system bends.
• Exercise: load-test report and the project's game day.
• Case study: Ticketing (BoxOffice flash sale).

Key takeaways

What breaks next

BoxOffice can now see itself: an SLO with a burn-rate alert, a RED dashboard, and one trace per order, from the edge through Kafka to the e-mail — in lab 8, stopping the database pages someone in about five to seven minutes (about 440 s in a test run). Seeing is not surviving. At 10:02 the payment provider answers in 4 s instead of 200 ms: the dashboard shows the order p95 climbing, the trace puts the time in the payment call, the alert fires — and checkout still dies, because every order thread is waiting on the provider and none is left even for requests that never touch it. Observability shortens the time to understand a failure; keeping one slow dependency from taking everything down — timeouts, circuit breakers, bulkheads, and chaos experiments that prove them — is class 10 — Reliability.