Observability

A system you cannot observe is a system you cannot operate. Monitoring tells you that something is wrong; observability tells you why. The distinction matters in microservices: a request that touches eight services and three databases cannot be diagnosed from a single metric on a single dashboard. Observability is the property of a system that makes its internal state inferable from external outputs.

The Three Pillars

Observability is built on three complementary data types. Each answers a different question about system behaviour.

Metrics

Prometheus Data Model

Every time series has a metric name and a set of key-value labels:

http_requests_total{method="GET", status="200", service="order"} 4823

PromQL Basics

RED and USE Methods

Metrics Pipeline

flowchart LR
    app["Spring Boot\n(Actuator + Micrometer)"]
    prom["Prometheus"]
    grafana["Grafana\n(dashboards + alerts)"]
    app -->|"exposes /actuator/prometheus"| prom
    prom -->|"scrape every 15 s"| prom
    prom -->|"PromQL queries"| grafana

Spring Boot Actuator configuration to expose the Prometheus endpoint:

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  metrics:
    tags:
      application: ${spring.application.name}

Logs

Structured vs Unstructured

Log Levels

Correlation IDs and MDC

When a request touches multiple services, a shared traceId in every log line allows reconstructing the full call sequence. Spring's Mapped Diagnostic Context (MDC) propagates this automatically with OpenTelemetry. Add the trace ID to every log line by placing it in MDC at the entry point and referencing it in the Logback pattern:

MDC.put("traceId", traceId);
// ... process request
MDC.remove("traceId");

Logback pattern to include the trace ID:

%d{ISO8601} [%X{traceId}] %-5level %logger{36} - %msg%n

Log Aggregation Pipeline

flowchart LR
    svc1["Service A\n(JSON logs → stdout)"]
    svc2["Service B\n(JSON logs → stdout)"]
    svc3["Service C\n(JSON logs → stdout)"]
    collector["Filebeat / Fluentd\n(collector)"]
    elastic["Elasticsearch\n(index + store)"]
    kibana["Kibana\n(search + dashboards)"]
    svc1 --> collector
    svc2 --> collector
    svc3 --> collector
    collector --> elastic
    elastic --> kibana

Distributed Traces

Anatomy of a Trace

A trace represents a single end-to-end request. It is composed of spans — one per operation (service call, DB query, cache lookup, etc.).

W3C traceparent Header

The traceparent header propagates trace context across service boundaries using the format version-traceId-parentSpanId-flags:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Every service propagates this header unchanged on downstream calls — only the parentSpanId changes to the current span's ID. This is how trace context crosses service boundaries without any service needing to understand the full trace.

Request Flow with Spans

sequenceDiagram
    autonumber
    actor User
    participant GW as Gateway [span: root, 120ms]
    participant OS as OrderService [span: child, 95ms]
    participant DB as PostgreSQL [span: leaf, 40ms]

    User->>+GW: GET /orders/42<br/>traceparent: 00-abc...-root-01
    GW->>+OS: GET /orders/42<br/>traceparent: 00-abc...-gw_span-01
    OS->>+DB: SELECT * FROM orders WHERE id=42
    DB-->>-OS: row data
    OS-->>-GW: 200 OK (95ms)
    GW-->>-User: 200 OK (120ms)

Sampling Strategies

OpenTelemetry (OTel)

OpenTelemetry is the CNCF standard for generating, collecting, and exporting telemetry. It replaces vendor-specific SDKs (Zipkin client, Jaeger client, etc.) with a single neutral API, so switching backends requires only a configuration change, not a code change.

flowchart LR
    app["Application\n(OTel SDK)"]
    collector["OTel Collector"]
    prom["Prometheus\n(metrics)"]
    jaeger["Jaeger\n(traces)"]
    loki["Loki\n(logs)"]
    app -->|"OTLP (gRPC/HTTP)"| collector
    collector --> prom
    collector --> jaeger
    collector --> loki

Java Zero-Code Instrumentation

The OTel Java agent instruments Spring Boot automatically via bytecode injection — no code changes required.

ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar

ENTRYPOINT ["java", \
  "-javaagent:/app/opentelemetry-javaagent.jar", \
  "-jar", "/app/app.jar"]

Configure the agent via environment variables in Docker Compose:

environment:
  OTEL_SERVICE_NAME: order-service
  OTEL_EXPORTER_OTLP_ENDPOINT: http://jaeger:4317
  OTEL_METRICS_EXPORTER: none
  OTEL_LOGS_EXPORTER: none

Manual Spans

When automatic instrumentation is not granular enough, add custom spans programmatically. Add the OTel API dependency:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
</dependency>

Wrap business logic in a span to capture timing, attributes, and exceptions:

@Autowired
private Tracer tracer;

public Order processOrder(OrderRequest request) {
    Span span = tracer.spanBuilder("processOrder").startSpan();
    try (Scope scope = span.makeCurrent()) {
        span.setAttribute("order.customerId", request.customerId());
        // ... business logic
        return order;
    } catch (Exception e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR, e.getMessage());
        throw e;
    } finally {
        span.end();
    }
}

Grafana Unified Stack (LGTM)

Loki + Grafana + Tempo + Mimir/Prometheus — the LGTM stack — puts all three observability signals into a single UI. The key advantage is signal correlation: you can jump from a metric spike to the traces that caused it to the log lines that explain it, all without leaving Grafana.

flowchart LR
    app["Application"]
    prom["Prometheus\n(metrics)"]
    loki["Loki\n(logs)"]
    tempo["Tempo\n(traces)"]
    grafana["Grafana"]
    app --> prom
    app --> loki
    app --> tempo
    prom --> grafana
    loki --> grafana
    tempo --> grafana

A typical drill-down workflow: a metric alert fires in Grafana → click "View Traces" to open the correlated trace in Tempo → click a span to see the correlated log lines from Loki. This cross-signal navigation is what separates an observability platform from three isolated monitoring tools.

DORA Metrics

DORA (DevOps Research and Assessment) identified four metrics that predict software delivery performance and organisational performance. Elite-performing teams score well on all four simultaneously.

Observability Snapshot: Spike Simulation

The chart below simulates what the three observability pillars look like during a traffic spike. Panel 1 shows the request rate rising sharply at steps 10–12; Panel 2 shows error events clustering in the same window; Panel 3 shows trace span durations stretching out under load.

The three panels share a timeline: the spike visible in Panel 1 directly explains the error cluster in Panel 2, and both explain the stretched span durations in Panel 3. Without all three pillars, you would see only one piece of the picture.