Observability

Observability is the ability to understand what a system is doing — and why — from the outside, using the signals it emits. It never sits on the request path: these components are side channels that watch the services. But without them you are flying blind: you cannot see which box is saturated, you cannot tell a slow dependency from a slow database, and you find out about outages from angry users. The rule of the course is simple: you can't fix what you don't measure.

The palette has three signals and two actions. Signals: metrics (numbers over time — is something wrong?), logs (detailed events — what exactly happened?) and traces (one request across services — where did the time go?). Actions: alerting (wake a human when users are affected) and health checks (let load balancers and orchestrators route around broken instances automatically). In BoxOffice, a RED dashboard per service, traces from the gateway to the payment provider, and an SLO burn-rate alert on checkout are what tell the on-call engineer, at 10:01 on sale day, that the payment provider is slow — not the database.

flowchart LR
    S1[gateway] -.->|OTLP| COL[OpenTelemetry Collector]
    S2[order] -.->|OTLP| COL
    S3[payment] -.->|OTLP| COL
    COL -.-> M[Metrics]
    COL -.-> L[Logs]
    COL -.-> T[Tracing]
    M -.->|SLO burn rate| AL[Alerting]
    AL -.->|page| ONCALL((on-call))
    LB[Load Balancer] -.->|GET /health/ready| S2

Metrics

What it is. Metrics are numeric measurements aggregated over time: counters (requests served, errors), gauges (queue depth, memory in use) and histograms (latency distribution, from which you read p50/p95/p99). Each time series is identified by a name and a set of labels (http_requests_total{service="order", route="/orders", status="500"}). They are cheap to store and query because they are pre-aggregated, which makes them the right signal for dashboards, SLOs and alerts. The standard recipes: RED for services (rate, errors, duration), USE for resources (utilization, saturation, errors) and Google's four golden signals (latency, traffic, errors, saturation).

When to use.

• Always, for every service and resource: the RED dashboard of BoxOffice's order, inventory and payment services.
• SLIs and SLOs: "share of checkout requests under 500 ms" is computed from a latency histogram.
• Capacity planning and autoscaling: CPU utilization, queue depth and consumer lag drive the autoscaler.
• Business metrics alongside technical ones: tickets sold per minute, waiting-room size, conversion.

When not to (or watch out).

• To investigate a single request: metrics are aggregates — use traces and logs.
• Cardinality: every unique combination of label values is a separate series. Labels like user_id, order_id or full URLs explode the number of series and the cost → keep labels bounded (route templates, status class).
• Averages hide the tail: record histograms and look at percentiles; never average percentiles across instances.
• Sampling interval: a 1-minute scrape smooths a 10-second spike away.

Key numbers.

Failure modes.

• Cardinality explosion — someone adds user_id as a label; memory and cost blow up, the metrics backend falls over → label allow-lists, limits per tenant, review metric changes.
• Missing metric where it mattered — the slow dependency had no client-side latency metric → instrument outbound calls (OpenTelemetry auto-instrumentation does).
• Misleading averages → histograms and percentiles; per-route breakdowns.
• Monitoring that dies with the system — the metrics stack runs in the same cluster that is down → monitor from outside too (synthetic checks, a separate region or a managed service).
• Dashboards nobody reads → few dashboards built around SLOs and RED/USE, linked from alerts.

In the playground. metrics is a side channel, not part of the request path. Connect services to it and their utilization becomes visible on the canvas; a service with no metrics node connected shows its utilization as "?" — you can't fix what you don't measure (a "god mode" toggle reveals everything for teaching). It adds a monthly cost. Scenarios: observability-stack, scale-out.

Real products.

Learn more: Observability › Signals · Observability › OpenTelemetry · Estimation › Percentiles · Performance › Methodology · Hands-on 8

Logs

What it is. Logs are timestamped records of discrete events: "order 81234 created", "payment provider timeout after 2000 ms", "user 77 failed login". Modern practice is structured logging — one JSON object per event with fields (level, service, trace_id, order_id, duration_ms) — so logs can be filtered and aggregated, not just grepped. Logs are shipped by an agent or the OpenTelemetry Collector to a central store where they are indexed and searched. Including the trace id in every log line links logs to traces.

When to use.

• Investigating a specific request or user: "what happened to order 81234?" — search by order_id or trace_id.
• Errors with context: stack traces, the input that caused them, which retry attempt failed.
• Audit trails for security and compliance: who refunded what, who changed permissions (kept separately, with longer retention and restricted access).
• Rare events that are not worth a metric.

When not to (or watch out).

• As your main way to detect problems: counting log lines to compute error rates is slow and expensive → use metrics, link to logs.
• Logging every request body at high volume: costs grow linearly with traffic → sample success logs, keep all errors.
• Sensitive data: passwords, tokens, card numbers and personal data must never be logged (LGPD/GDPR, PCI DSS) → redact at the source.
• Unstructured text with inconsistent formats makes search nearly impossible.

Key numbers.

Failure modes.

• Log flood — a failing dependency makes every request log a stack trace; the pipeline backs up and the bill explodes → rate-limit repeated logs, sample, drop debug in production.
• Missing correlation — logs from 6 services cannot be joined → propagate and log the trace_id.
• Leaking secrets or personal data → structured fields with redaction, reviews, scanning.
• Logging blocking the app — synchronous writes to a slow sink add latency → async appenders, local buffering, stdout + agent.
• Lost logs when a pod dies → write to stdout and let the node agent ship them; buffers with backpressure.

In the playground. logs is a side channel: connecting services to it enables the event log details (what happened to the sampled requests, which errors and retries) in the Events tab. It costs per GB, so it shows up in the cost tile. Scenarios: observability-stack, cascading-failure.

Real products.

Learn more: Observability › Signals · Observability › OpenTelemetry · Observability › Incident response · Network & security › Zero trust · Hands-on 8

Tracing

What it is. Distributed tracing follows one request across all the services it touches. Each unit of work is a span (name, start, duration, attributes, status) and spans form a tree under one trace id, which is propagated between services in request headers (the W3C traceparent header) and in message metadata. The result is a waterfall: you see that a 4 s checkout spent 3.8 s waiting for the payment provider, or that one page makes 40 sequential database calls (N+1). Because recording every request is expensive, traces are sampled: head sampling (decide at the start, e.g. 5%) or tail sampling (decide at the end, keeping all errors and slow requests).

When to use.

• Any system with more than a couple of services: finding where latency or errors come from.
• Diagnosing the critical path: what is sequential that could be parallel, which dependency dominates p99.
• Understanding fan-out and retries: a trace shows the three retries that tripled the load.
• LLM applications: a trace per agent run shows each model call, tool call, tokens and latency.

When not to (or watch out).

• Without context propagation through every hop (HTTP, gRPC, Kafka headers, async jobs), traces break into fragments → use OpenTelemetry instrumentation everywhere.
• 100% sampling at high volume is expensive; too little sampling misses the rare slow requests → tail-based sampling for errors and slow traces.
• Traces show one request; for trends use metrics (or metrics derived from spans).
• Sensitive data in span attributes is as dangerous as in logs.

Key numbers.

Failure modes.

• Broken traces — one service or the message broker drops the context → instrument every hop, propagate through Kafka headers and job payloads.
• Sampling hides the problem — the rare 10-second request was not sampled → tail sampling (keep errors and slow traces), exemplars linking metrics to traces.
• Collector overload at peak → batching, horizontal Collector scaling, sampling at the edge of the pipeline.
• Clock skew makes child spans look like they start before the parent → rely on durations within a host, NTP everywhere.

In the playground. tracing is a side channel: when present, the Trace tab shows the waterfall of one sampled slow request, with the time spent queueing and serving in each node — the fastest way to find a slow dependency or a retry storm. It adds cost. Scenarios: observability-stack, cascading-failure, retry-storm, ai-assistant.

Real products.

Learn more: Observability › OpenTelemetry · Observability › Signals · Performance › Bottlenecks · AI systems › Safety & evals · Hands-on 8

Alerting

What it is. Alerting turns signals into notifications for humans: rules evaluated continuously over metrics (or logs) that, when true for long enough, route a message to the right person — a page for urgent problems, a ticket or chat message for the rest. Good alerts are about symptoms users feel (errors, latency, SLO violations), are actionable, and come with a runbook link. The modern standard is SLO-based alerting with burn rates: alert when the error budget is being consumed much faster than sustainable, using two windows (e.g. 1 h and 5 min) so the alert is both fast and not noisy.

When to use.

• Every user-facing SLO: "checkout availability 99.9% over 28 days" with fast-burn (page) and slow-burn (ticket) alerts.
• Leading indicators of imminent outages: disk filling up within hours, certificate expiring in 7 days, consumer lag approaching retention.
• On-call rotations with escalation policies and acknowledgement.

When not to (or watch out).

• Alerting on causes instead of symptoms ("CPU > 80%") pages people for things that don't affect users → alert fatigue, and real pages get ignored.
• Alerts without an owner or a runbook.
• Static thresholds on noisy metrics flap → use windows, burn rates, hysteresis.
• Alerting depends on the monitoring stack being alive → a "dead man's switch" (an always-firing heartbeat alert that pages when it stops).

Key numbers.

Failure modes.

• Alert fatigue — too many noisy alerts, the real one is missed → delete or demote non-actionable alerts, SLO-based paging only.
• Silent failure — the metric stopped being reported, so the threshold never fires → alert on absent data, dead man's switch.
• Alerting stack down with the system → monitor the monitor from outside (synthetic checks, a separate provider).
• No runbook / wrong owner → each alert links a runbook and routes to the owning team.
• Flapping → longer evaluation windows, for durations, hysteresis.

In the playground. alerting is a side channel connected to metrics: when present, random failures are detected and fixed faster — the model reduces MTTR by 50%, which shows up as higher availability in the run report. It costs money, like the other observability nodes. Scenarios: observability-stack, multi-region.

Real products.

Learn more: Observability › Alerting · Observability › SLOs · Observability › Incident response · Estimation › Availability math · Hands-on 8

Health Check

What it is. A health check is a probe that something outside the instance — a load balancer, Kubernetes, a DNS failover service — sends periodically to decide whether that instance should receive traffic or be restarted. Kubernetes distinguishes three: liveness ("is the process stuck? if it fails, restart it"), readiness ("can it serve traffic right now? if it fails, take it out of the load balancer, don't restart it") and startup ("is it still booting? hold the other probes until it's done"). Load balancers use similar active checks (an HTTP request to /health every few seconds) and sometimes passive checks (outlier detection on real traffic errors).

When to use.

• Every replica behind a load balancer or a Kubernetes Service: without it, the LB keeps sending traffic to dead or warming-up instances.
• Readiness during startup and warm-up (JIT, caches, connection pools) and during graceful shutdown (fail readiness first, then drain).
• Regional failover: DNS health checks decide when to send users to the other region.
• Load shedding: an overloaded instance can fail readiness temporarily.

When not to (or watch out).

• Don't check dependencies in liveness: if the database is slow and every pod's liveness checks it, Kubernetes restarts all pods at once — a restart storm that makes the outage worse.
• Readiness that checks a shared dependency can take every replica out at once → check only what the instance itself needs, and degrade gracefully.
• Health endpoints must be cheap, fast and unauthenticated-but-not-public (no sensitive details).
• Too-aggressive thresholds flap; too-lax ones leave dead instances in rotation.

Key numbers.

Failure modes.

• Restart storm — liveness depends on the database; DB slows; all pods restart → liveness only checks the process itself.
• Fleet-wide removal — readiness fails everywhere because of a shared dependency; the LB has no healthy targets → degrade instead of failing readiness; LBs that "fail open" when all targets are unhealthy.
• Zombie instance — health endpoint returns 200 but real requests fail (e.g. a broken thread pool) → passive health checks / outlier detection on real traffic, deeper readiness checks.
• Slow detection → shorter intervals for critical paths, plus passive checks.
• Flapping → healthy/unhealthy thresholds with hysteresis.

In the playground. healthcheck is a side channel: connecting it to a service behind a load balancer reduces the LB's failure-detection time from 30 s to 3 s, so after a replica crash far fewer requests are sent to the dead instance. Compare the error spike after a crash with and without it. Scenarios: scale-out, multi-region, observability-stack.

Real products.

Learn more: Observability › Signals · Cloud-native › Kubernetes · Reliability › Patterns · Edge › Load balancing · Load Balancer · Hands-on 3 · Hands-on 10