System Architecture · Insper · Chapter 9

Hands-on 8 — Observability

OpenTelemetry, a RED (rate, errors, duration) dashboard, one trace across Kafka and a service level objective (SLO) that pages on time

The story so far

before · lab 7the e-mail left the request: outbox, Kafka
→
chapter 9 · lab 8OpenTelemetry, RED dashboard, SLO and alert
→
next · lab 9chapter 10: a slow provider, resilience

Asynchrony hides failures: POST /orders answers in 1.8 ms (lab 7), yet an order whose e-mail never arrives produces no 5xx. Where did that order go?

  • k6 measures from the outside; this lab adds the inside view.

"Checkout is slow"

The flash sale is live and support is flooded. Is it every fan or a few? Which of seven services?
The database, the pool, Kafka, the e-mail provider?

  • k6 measures from outside; we need the inside view.
  • Metrics: that something is wrong, and how much.
  • Traces: where the time went, request by request.
  • Logs: why, once you know where to look.

Architecture

every Java service + the agent
gateway
catalog ×3
order · notification
OTLP :4318 →
otel-lgtm (one container)
OpenTelemetry Collector
Prometheusmetrics
Tempotraces
Lokilogs
→
Grafana :3000exemplar → trace → logs; alert rules
  • The agent pushes OTLP (the OpenTelemetry Protocol); Grafana links the three signals.

Development backend in one image; production swaps it for separate systems — the services still speak OTLP.

Step 1 · The agent: one jar, no code

sh scripts/get-otel-agent.sh      # downloads infra/otel/opentelemetry-javaagent.jar (v2.32.0)
x-otel: &otel
  JAVA_TOOL_OPTIONS: -javaagent:/otel/opentelemetry-javaagent.jar
  OTEL_JAVAAGENT_ENABLED: ${OTEL_ON:-true}
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-lgtm:4318

Each Java service mounts ./infra/otel:/otel:ro and adds <<: *otel plus its own OTEL_SERVICE_NAME. No image changes.

What the agent gives you

  • Spans for HTTP server/client, JDBC, Kafka, Redis, @Scheduled.
  • Metrics with standard names: http_server_request_duration_seconds_bucket{service_name, http_route, http_response_status_code}.
  • Logs via OTLP, each with trace_id and span_id.
  • http_route is the template (/catalog/events/{id}), never the raw URL → bounded cardinality.

The hop the agent cannot see

An order produces two unrelated traces: POST /orders, and later the relay → Kafka → notification.

orderHTTP thread
INSERT →
outbox row+ traceparent (V2 migration)
SELECT →
relayscheduler thread: makeCurrent()
→
Kafka → notificationthe same trace id
  • The outbox is a database row read by another thread: the request's context must travel in it.
  • Result: one trace — 9 ms of request, the relay 22 ms later, 67 ms in notification.

Step 2 · RED and USE

Method For Signals
RED request-driven services Rate, Errors, Duration
USE resources Utilization, Saturation, Errors
  • RED says users are hurting; USE says which resource ran out.
  • Dashboard as code: infra/grafana/dashboards/boxoffice-red.json + a provider file.

Four PromQL shapes

sum by (service_name) (rate(…_count[$__rate_interval]))                     # Rate
rate(…_count{http_response_status_code=~"5.."}) / rate(…_count)             # Errors (ratio)
histogram_quantile(0.95, sum by (service_name, le) (rate(…_bucket[…])))      # Duration
db_client_connections_pending_requests                                      # Saturation

Never average percentiles. Sum the bucket rates, then take the quantile.

Step 3 · From promise to numbers

Term catalog
SLI (service level indicator) share of requests (no /actuator) not answered 5xx
SLO 99.5% over 28 days
Error budget 0.5% → ≈ 202 min of total outage per 28 days
Burn rate error ratio / 0.005 (1 = exactly on budget)

Multiwindow, multi-burn-rate

Severity Burn rate Long Short Budget spent
page 14.4 1 h 5 min ≈ 2.1%
page / ticket 6 6 h 30 min ≈ 5.4%
ticket 1 3 days 6 h ≈ 10.7%
  • Long window: the problem is significant, not a blip.
  • Short window: it is still happening → resolves soon after the fix.
  • As code: infra/grafana/alerting/slo-catalog.yaml (queries A, B + math $A > 14.4 && $B > 14.4).

Fire it

Steady traffic, one healthy hour, then docker compose stop db. When does the fast-burn rule page?

  • The 1-hour error ratio must reach 14.4 × 0.5% = 7.2%.
  • 0.072 × 60 min ≈ 4.3 min of outage, + for: 1m + evaluation ≈ 5–6 min.
  • A 30-second blip never pages; a slow 4% leak is caught by the 6-hour rule.
  • Test run: Firing about 440 s after the stop — step 2's stress test was still in the 1-hour window.
  • start db: the page clears about 5 min later, once the outage leaves the 5-minute window.

Without Docker: replay it

10:12:31stop db
→
5-min burn rate: 200within a minute
→
1-h burn rate > 14.4after 4.3 min
→
Pending → Firingfor: 1m → 5:30
→
start db → Normal5:00 later
  • The simulator on the SLO step: slo-catalog.yaml evaluated every 30 s, k6 at 50 req/s, one tick = 10 s.
  • With step 2's stress test in the window → 7:30, "Burn rate 17.0 (1 h) and 200.0 (5 min)"; 10 min of history → 2:00.
  • 4% errors never page; the 6-hour ticket fires after 3.1 h.

Step 4 · Measure it

# Experiment Record
A agent on vs off (OTEL_ON) p50 / p95 / p99, CPU, memory, startup
B sampling 100% vs 10% (parentbased_traceidratio) p95, traces per minute
C cardinality series per service, the cost of a user_id label
D time to detect stop db → Pending → Firing

k6 constant-arrival-rate on GET /catalog/events/{id} through the edge; docker stats during the run.

Pitfalls

  • Context breaks at async hops: carry traceparent in messages and rows.
  • Unbounded labels (ids, URLs, e-mails) explode the number of series.
  • No traffic, no SLI: pair SLO alerts with a probe or a "traffic dropped" alert.
  • Health checks in the SLI hide real failures.
  • Sampling applies to traces, not metrics: rates come from metrics.

Try it in the playground

  • observability-stack with the chaos profile.
  • Disconnect metrics from a service: its utilization becomes "?".
  • Which reacts first to a failure: errors, percentiles or health checks?
  • boxoffice-observability: the catalog database stops for 100 s; only Alerting shortens the outage.

You cannot fix what you do not measure — and you cannot measure what you did not instrument.

What breaks next

lab 8 · we can see itone trace per order · paged ~440 s after db stopped, in a test run
→
seeing ≠ surviving10:02: the payment provider answers in 4 s instead of 200 ms
→
lab 9 · chapter 10timeout, retry, circuit breaker, bulkhead, a game day
  • The alert fires and checkout still dies: every order thread waits on the provider.

Takeaways

  1. Metrics say that, traces say where, logs say why — linked by trace_id.
  2. The Java agent instruments HTTP, JDBC, Kafka and logs with zero code; OTLP is the contract.
  3. Async hops (outbox, queues) need the traceparent carried with the data.
  4. Dashboards and alerts are code in the repository.
  5. Alert on SLO burn rate over two windows, not on raw thresholds.
  6. Observability has a cost — measure it, and sample traces, not metrics.

Chapter 9 of the story of the course. Outline: (1) the questions k6 cannot answer; (2) otel-lgtm and the Java agent; (3) the outbox breaks the trace; (4) RED and USE as code; (5) SLO, error budget, burn rate; (6) fire the alert; (7) measure.

The story of the course (2027.1 overview, "The story of the course"): chapter 9 opens with asynchrony hiding failures. Lab 7 measured the outbox at 1.8 ms p50 against 2.7 s inline at 200 orders/s, in a test run for that lab.