8. Observability

The flash sale went live, and the support channel fills up: "checkout is slow", "the page shows an error". Is it every fan or a few? Which of the seven services? The database, the pool, Kafka, the e-mail provider? Until now the only instrument was k6, which measures from the outside. This lab adds the inside view — the three signals of the observability class: metrics say that something is wrong and how much, traces say where in the request the time went, logs say why.

This lab builds chapter 9 of the story of the course in BoxOffice: the RED dashboard of signals, the trace context carried through the outbox and Kafka of OpenTelemetry, and the SLO with the multiwindow burn-rate alert of alerting. It comes right before lab 9 for a reason: you instrument BoxOffice before you break it on purpose, otherwise the chaos experiments of lab 9 have nothing to show.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you start

Every command runs from the root of your boxoffice repository, which must be in the state lab 7 left it:

boxoffice/
├── docker-compose.yaml     # lab 6 stack + kafka, kafka-init, kafka-ui, mailpit, order, notification
├── infra/                  # nginx/, postgres/, pgbouncer/
├── load/                   # ... orders-load.js, waiting-room-spike.js
├── scripts/                # ... kafka-lag.sh
└── services/
    ├── auth/  catalog/  gateway/  inventory/
    ├── notification/
    └── order/

Every file of this lab is shown in full on these pages and published under https://insper.github.io/platform/2027.1/hands-on/8/code/<path>.

Architecture

flowchart LR
    k6[k6 / fans] --> edge[edge]
    edge --> gw[gateway]
    gw --> catalog[catalog x3]
    gw --> order[order]
    order -.->|outbox + Kafka| notification[notification]
    catalog -->|OTLP 4318| col
    gw -->|OTLP| col
    order -->|OTLP| col
    notification -->|OTLP| col
    subgraph lgtm [otel-lgtm]
        col[OTel Collector] --> prom[(Prometheus)]
        col --> tempo[(Tempo)]
        col --> loki[(Loki)]
        prom --> grafana[Grafana :3000]
        tempo --> grafana
        loki --> grafana
    end
    grafana -->|burn rate > 14.4| alert[alert: page]

Each service runs the agent and pushes OTLP (OpenTelemetry Protocol) to the collector inside otel-lgtm, which stores each signal in its own database. Grafana links them: a point on a latency chart opens its trace, a span opens the logs it wrote.

What you will measure

Prerequisites

• Labs 1 to 7 done: BoxOffice runs with the edge, the gateway, catalog replicas, inventory, order, notification and Kafka.
• Around 1.5 GB of free memory for otel-lgtm.
• Read the observability class, especially signals, OpenTelemetry and alerting.

Steps

1. Instrument everything — otel-lgtm, the Java agent in every service, logs with trace ids, and the trace context carried through the outbox.
2. RED dashboard — a provisioned dashboard: Rate, Errors, Duration per service, plus USE for JVMs and connection pools.
3. SLO and burn-rate alert — SLI, SLO and error budget for catalog; fast- and slow-burn rules as code; fire one.
4. Measure it — agent overhead, sampling, cardinality and time to detect.

Try it in the playground

Try it: run the observability-stack scenario with the chaos profile. First leave the metrics node disconnected from a service: its utilization shows as "?" — you cannot fix what you do not measure. Connect it, then watch which signal reacts first when a random failure hits: the error rate, the latency percentiles or the health check.

The full simulator is in the playground.

Measure it

The Measure it step has the k6 script and the results table to fill in and keep in results/lab8.md.

Troubleshooting

Best practices

Clean up

docker compose stop                 # keep everything for lab 9
docker compose down -v              # or remove containers and volumes (all data, including metrics)

Lab 9 uses this dashboard during its game day: keep infra/grafana/ and infra/otel/.

End state

boxoffice/
├── .gitignore              # + infra/otel/*.jar
├── docker-compose.yaml     # + x-otel anchor, agent volume in every Java service, otel-lgtm
├── infra/
│   ├── grafana/
│   │   ├── dashboards.yaml
│   │   ├── dashboards/boxoffice-red.json
│   │   └── alerting/slo-catalog.yaml
│   └── otel/opentelemetry-javaagent.jar   # downloaded, not committed
├── load/catalog-otel.js
├── scripts/get-otel-agent.sh
└── services/order/
    ├── pom.xml             # + opentelemetry-api
    └── src/main/
        ├── java/boxoffice/order/outbox/Outbox.java, OutboxRelay.java   # traceparent
        └── resources/db/migration/V2__outbox_traceparent.sql

Commit and tag: git add -A && git commit -m "lab 8: observability" && git tag lab8.

Checklist

• scripts/get-otel-agent.sh downloads the agent, and docker-compose.yaml attaches it to every Java service through the x-otel anchor and the infra/otel volume, each with its own OTEL_SERVICE_NAME
• Grafana Explore shows http_server_request_duration_seconds_* for every service, traces in Tempo and logs with trace_id in Loki
• a single trace goes from POST /orders in order through the outbox relay and Kafka to notification
• infra/grafana/ holds the dashboard provider, boxoffice-red.json and slo-catalog.yaml, loaded at startup
• the fast-burn alert fired when db was stopped, and stopped firing after it came back
• results table of the Measure it step filled in results/lab8.md and explained
• self-check: after docker compose down -v && sh scripts/get-otel-agent.sh && docker compose up -d, the dashboard and both alert rules appear without manual steps; pushed with the tag lab8

Related

• Theory: Observability · Signals: RED, USE, golden signals · OpenTelemetry · Alerting and burn rate · Incident response · SLOs and error budgets
• Previous lab: 7. Async messaging — the flow traced here.
• Next lab: 9. Resilience & chaos uses this dashboard during a game day.

Start: Instrument everything