Observability

A system you cannot observe is a system you cannot operate. Monitoring tells you that something is wrong; observability tells you why. The distinction matters in microservices: a request that touches eight services and three databases cannot be diagnosed from a single metric on a single dashboard. Observability is the property of a system that makes its internal state inferable from external outputs.

Picture a customer complaining that "checkout is slow" in the store. The request went through Nginx, the gateway, the auth check, the order service, the product service and Postgres, and maybe published an event to Kafka. Which one was slow? Was it slow for everyone or just for this customer? Without telemetry designed in advance — numbers, records and request paths emitted by every service — the only answer is guessing.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

The Three Pillars

Observability is built on three complementary data types. Each answers a different question about system behaviour.

All three describe the same events from different angles. One POST /orders that fails produces: an increment of a counter http_server_requests_seconds_count{status="500"} (metric), a JSON line {"level":"ERROR","msg":"product not found",...} (log), and a trace with one span per hop, the last one marked ERROR (trace). Metrics are cheap and tell you that something changed; logs and traces are more expensive and tell you what and where.

Metrics

Prometheus Data Model

Every time series has a metric name and a set of key-value labels:

http_requests_total{method="GET", status="200", service="order"} 4823

A label is a dimension of the series: every distinct combination of label values is a separate time series stored by Prometheus.

PromQL Basics

PromQL is Prometheus' query language. The queries below use the metric that Spring Boot (Micrometer) exports for every HTTP endpoint, http_server_requests_seconds, with the labels application, uri, method and status:

Two rules: apply rate() or increase() to a counter before summing (they handle counter resets when a pod restarts), and keep the le label when summing histogram buckets (sum by (le)), because histogram_quantile needs it. The _bucket series exist only if histograms are enabled: management.metrics.distribution.percentiles-histogram.http.server.requests=true.

RED, USE and the Golden Signals

Metrics Pipeline

flowchart LR
    app["Spring Boot\n(Actuator + Micrometer)"]
    prom["Prometheus"]
    grafana["Grafana\n(dashboards + alerts)"]
    prom -->|"scrapes /actuator/prometheus (e.g. every 15 s)"| app
    prom -->|"PromQL queries"| grafana

Prometheus uses a pull model: the application only exposes its current values at an HTTP endpoint, and Prometheus fetches (scrapes) them on a schedule and stores them as time series. This is what Hands-on 6 sets up for the store.

Spring Boot Actuator configuration to expose the Prometheus endpoint (it also needs the io.micrometer:micrometer-registry-prometheus dependency):

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  metrics:
    tags:
      application: ${spring.application.name}

Alerting

Dashboards need someone looking at them; alerts notify a human when action is needed. Prometheus evaluates alerting rules and sends firing alerts to Alertmanager (or Grafana Alerting), which routes them to e-mail, Slack or a pager.

groups:
  - name: order-service
    rules:
      - alert: OrderHighErrorRatio
        expr: |
          sum(rate(http_server_requests_seconds_count{application="order", status=~"5.."}[5m]))
          /
          sum(rate(http_server_requests_seconds_count{application="order"}[5m])) > 0.05
        for: 5m                     # must hold for 5 minutes, avoids flapping
        labels:
          severity: page
        annotations:
          summary: "More than 5% of order requests are failing"

Logs

Structured vs Unstructured

Log Levels

Correlation IDs and MDC

When a request touches multiple services, a shared ID in every log line — a correlation ID — allows reconstructing the full call sequence by searching for that one value. With distributed tracing, the natural correlation ID is the trace ID.

In Java, the place to put it is the MDC (Mapped Diagnostic Context) of SLF4J/Logback: a per-thread map of key-value pairs that the logging framework can print in every line with %X{key}. Who fills the MDC depends on how the service is instrumented:

%d{ISO8601} [%thread] %-5level %logger{36} [trace_id=%X{trace_id} span_id=%X{span_id}] - %msg%n

public Order create(OrderIn in) {
    try (var ignored = MDC.putCloseable("customerId", in.customerId())) {  // removed at the end
        log.info("creating order");   // ... [trace_id=4bf9...] customerId=c-42 ... creating order
        return repository.save(...);
    }
}

%d{ISO8601} %-5level [trace_id=%X{trace_id}] customerId=%X{customerId} %logger{36} - %msg%n

Log Aggregation Pipeline

flowchart LR
    svc1["Service A\n(JSON logs → stdout)"]
    svc2["Service B\n(JSON logs → stdout)"]
    svc3["Service C\n(JSON logs → stdout)"]
    collector["Filebeat / Fluentd\n(collector)"]
    elastic["Elasticsearch\n(index + store)"]
    kibana["Kibana\n(search + dashboards)"]
    svc1 --> collector
    svc2 --> collector
    svc3 --> collector
    collector --> elastic
    elastic --> kibana

Distributed Traces

Anatomy of a Trace

A trace represents a single end-to-end request. It is composed of spans — one per operation (service call, DB query, cache lookup, etc.).

W3C traceparent Header

The traceparent header propagates trace context across service boundaries using the format version-traceId-parentSpanId-flags:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

On every downstream call, a service keeps the version, the traceId and (normally) the flags exactly as received, and replaces the parentSpanId field with the ID of its own current span. This is how trace context crosses service boundaries without any service needing to understand the full trace. The header is defined by the W3C Trace Context recommendation⁵; OpenTelemetry uses it by default.

Request Flow with Spans

sequenceDiagram
    autonumber
    actor User
    participant GW as Gateway [span: root, 120ms]
    participant OS as OrderService [span: child, 95ms]
    participant DB as PostgreSQL [span: leaf, 40ms]

    User->>+GW: GET /orders/42<br/>traceparent: 00-abc...-root-01
    GW->>+OS: GET /orders/42<br/>traceparent: 00-abc...-gw_span-01
    OS->>+DB: SELECT * FROM orders WHERE id=42
    DB-->>-OS: row data
    OS-->>-GW: 200 OK (95ms)
    GW-->>-User: 200 OK (120ms)

Sampling Strategies

Recording every span of every request is expensive at scale, so tracing systems keep only a sample of traces.

OpenTelemetry (OTel)

OpenTelemetry is the CNCF standard for generating, collecting, and exporting telemetry³. It replaces vendor-specific SDKs (Zipkin client, Jaeger client, etc.) with a single neutral API, so switching backends requires only a configuration change, not a code change.

flowchart LR
    app["Application\n(OTel SDK)"]
    collector["OTel Collector"]
    prom["Prometheus\n(metrics)"]
    jaeger["Jaeger\n(traces)"]
    loki["Loki\n(logs)"]
    app -->|"OTLP (gRPC/HTTP)"| collector
    collector --> prom
    collector --> jaeger
    collector --> loki

Java Zero-Code Instrumentation

The OTel Java agent instruments Spring Boot automatically via bytecode injection — no code changes required.

# pin a version: "latest" makes builds unreproducible
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.4.0/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar

ENTRYPOINT ["java", \
  "-javaagent:/app/opentelemetry-javaagent.jar", \
  "-jar", "/app/app.jar"]

Configure the agent via environment variables in Docker Compose:

environment:
  OTEL_SERVICE_NAME: order-service
  OTEL_EXPORTER_OTLP_PROTOCOL: grpc            # agent 2.x defaults to http/protobuf (port 4318)
  OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
  OTEL_METRICS_EXPORTER: none
  OTEL_LOGS_EXPORTER: none

Manual Spans

When automatic instrumentation is not granular enough — you want to see how long your price calculation took, not just the HTTP and SQL calls around it — add custom spans. Add the OTel API dependency (its version is managed by the opentelemetry-bom, or set it explicitly):

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
</dependency>

The simplest way, with the agent attached, is an annotation (dependency io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations):

@WithSpan("calculateTotal")
public BigDecimal calculateTotal(@SpanAttribute("order.items") int items, ...) { ... }

For full control, create the span yourself. The agent registers the global OpenTelemetry instance, so obtain the tracer from it (it is not a Spring bean unless you declare one):

private static final Tracer tracer =
    GlobalOpenTelemetry.getTracer("store.order");

public Order processOrder(OrderRequest request) {
    Span span = tracer.spanBuilder("processOrder").startSpan();
    try (Scope scope = span.makeCurrent()) {
        span.setAttribute("order.customerId", request.customerId());
        // ... business logic
        return order;
    } catch (Exception e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR, e.getMessage());
        throw e;
    } finally {
        span.end();
    }
}

Grafana Unified Stack (LGTM)

Loki + Grafana + Tempo + Mimir/Prometheus — the LGTM stack — puts all three observability signals into a single UI. The key advantage is signal correlation: you can jump from a metric spike to the traces that caused it to the log lines that explain it, all without leaving Grafana.

flowchart LR
    app["Application"]
    prom["Prometheus\n(metrics)"]
    loki["Loki\n(logs)"]
    tempo["Tempo\n(traces)"]
    grafana["Grafana"]
    app --> prom
    app --> loki
    app --> tempo
    prom --> grafana
    loki --> grafana
    tempo --> grafana

A typical drill-down workflow: a metric alert fires in Grafana → click "View Traces" to open the correlated trace in Tempo → click a span to see the correlated log lines from Loki. This cross-signal navigation is what separates an observability platform from three isolated monitoring tools.

DORA Metrics

DORA (DevOps Research and Assessment) identified four metrics that predict software delivery performance and organisational performance⁴. Two measure throughput (how fast changes flow) and two measure stability (how often changes break things and how fast you recover). Elite-performing teams score well on all four simultaneously — speed and stability are not a trade-off.

Observability Snapshot: Spike Simulation

The chart below simulates what the three observability pillars look like during a traffic spike. Panel 1 shows the request rate rising sharply at steps 10–12; Panel 2 shows error events clustering in the same window; Panel 3 shows trace span durations stretching out under load.

The three panels share a timeline: the spike visible in Panel 1 directly explains the error cluster in Panel 2, and both explain the stretched span durations in Panel 3. Without all three pillars, you would see only one piece of the picture.

Connections

• Hands-on 6 — Prometheus and Grafana for the store's metrics.
• Hands-on 10 — OpenTelemetry Java agent, Jaeger and trace_id in the logs.
• Reliability and SLI / SLO / SLA — SLIs are PromQL queries over these metrics; burn-rate alerts use them.
• Event-Driven Architecture — tracing across Kafka topics and monitoring consumer lag.
• Exercise: Bottlenecks — using observability to find where the platform is slow.

Key takeaways

• Monitoring answers known questions; observability lets you investigate new ones from metrics (that), logs (what) and traces (where).
• Prometheus scrapes /actuator/prometheus; use rate() on counters, histogram_quantile with sum by (le) for percentiles, and keep labels low-cardinality.
• Build dashboards with RED (services) and USE (resources); alert on symptoms users feel.
• Logs must be structured and carry the trace_id — filled automatically by the OTel agent; use MDC.put only for your own business keys.
• A trace is a tree of spans linked by traceparent; head sampling is cheap but blind, tail sampling keeps the interesting traces.
• OpenTelemetry is the vendor-neutral standard; measure delivery performance with the four DORA metrics.