OpenTelemetry

BoxOffice is written in Java (gateway, order, inventory, payment, catalog), Python (the AI assistant), runs Nginx at the edge, Kafka in the middle and PostgreSQL at the bottom. Ten years ago, tracing a checkout across all of that meant a vendor agent per language, a proprietary header per vendor, and a full re-instrumentation the day the company changed vendors. Worse, a trace broke at the first component that spoke a different dialect. OpenTelemetry (OTel) fixed this by standardising the boring parts — the API, the wire protocol, the context header and the names of things — so that every library and every backend can speak the same language¹. (The ideas themselves — trace IDs, spans, parent links, sampling — come from Google's Dapper paper⁴.)

What OpenTelemetry is (and is not)

OpenTelemetry is a CNCF project that defines how telemetry is produced and transported. It is not a backend: it does not store or display anything. You still choose where the data goes — Prometheus, Tempo, Jaeger, Loki, a cloud service or a vendor — and you can change that choice without touching application code.

The signals have matured at different speeds: traces and metrics are stable, logs became stable later (OTel mostly bridges existing logging libraries, adding the trace context), and profiles were added as the fourth signal and are still maturing.

flowchart LR
    subgraph services ["BoxOffice services"]
        J["Java services<br/>OTel Java agent"]
        PY["assistant (Python)<br/>OTel SDK"]
        NG["Nginx<br/>OTel module"]
    end
    subgraph col ["OTel Collector"]
        R["receivers<br/>otlp"] --> PR["processors<br/>memory_limiter, k8s attrs,<br/>redaction, tail sampling, batch"] --> E["exporters"]
    end
    J -- OTLP --> R
    PY -- OTLP --> R
    NG -- OTLP --> R
    E --> TEMPO[("Tempo / Jaeger<br/>traces")]
    E --> PROM[("Prometheus / Mimir<br/>metrics")]
    E --> LOKI[("Loki<br/>logs")]

Context propagation

A trace exists only if every hop knows which trace it belongs to. Context propagation is the mechanism: the caller injects the current trace context into the outgoing request, and the callee extracts it and creates its own span as a child.

The traceparent header

OpenTelemetry uses the W3C Trace Context standard² by default. One header carries the context:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                └─ flags: 01 = sampled
             │  │                                └─ parent-id: the caller's current span (64 bits)
             │  └─ trace-id: same for the whole request (128 bits)
             └─ version

On every outgoing call, a service keeps the trace-id and the flags and replaces the parent-id with the ID of its own current span. No service needs to understand the whole trace; the tree is rebuilt by the backend from the parent links. A second header, tracestate, carries vendor-specific data, and baggage (baggage: tenant=festival-x,channel=mobile) carries business key-value pairs that every downstream service can read — useful, but every byte travels on every request, so keep it small and never put secrets in it.

Following a checkout

Play the diagram and watch the parent-id change at each hop while the trace-id stays the same — including across Kafka, where the context travels in a message header:

sequenceDiagram
    autonumber
    participant GW as gateway
    participant O as order
    participant I as inventory
    participant P as payment
    participant X as payment provider
    participant K as Kafka
    participant N as notification
    GW->>O: POST /orders<br/>traceparent: 00-4bf9…-a1a1…-01
    O->>I: PUT /holds/h7/confirm<br/>traceparent: 00-4bf9…-b2b2…-01
    I-->>O: 200 OK (90 ms)
    O->>P: POST /charges<br/>traceparent: 00-4bf9…-c3c3…-01
    P->>X: POST /v1/charges (CLIENT span recorded by payment)
    X-->>P: 200 OK (1.0 s)
    P-->>O: 201 Created
    O->>K: produce OrderConfirmed<br/>header traceparent: 00-4bf9…-d4d4…-01
    O-->>GW: 201 Created
    K->>N: consume OrderConfirmed (CONSUMER span, parent d4d4…)
    N->>N: send e-mail and push

Three details matter in practice:

• The external provider is a black box. It will not send its spans to you. The CLIENT span recorded by payment around the call is all you will ever see of it — which is why that span's duration and status are worth an alert of their own. Many teams also strip traceparent and baggage at egress, so internal IDs do not leak to third parties.
• Asynchronous hops propagate through message headers. A consumer that handles one message continues the producer's trace (parent-child). A consumer that processes a batch of 500 messages cannot have 500 parents; it starts its own trace and adds span links to each producer's span.
• The trace ends where propagation breaks. One component that drops the header splits every trace in two.

@Transactional(propagation = Propagation.MANDATORY)
public void append(UUID eventId, String aggregateType, UUID aggregateId, String eventType, Object payload) {
    String json = mapper.writeValueAsString(payload); // Jackson 3: exceptions are unchecked
    // capture the trace context of the current request (null without the agent)
    Map<String, String> carrier = new HashMap<>();
    W3CTraceContextPropagator.getInstance().inject(Context.current(), carrier, (c, k, v) -> c.put(k, v));

    jdbc.sql("""
            INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload, traceparent)
            VALUES (?, ?, ?, ?, ?::jsonb, ?)
            """)
            .params(eventId, aggregateType, aggregateId, eventType, json, carrier.get("traceparent"))
            .update();
}

for (Pending p : batch) {
    ProducerRecord<String, String> record =
            new ProducerRecord<>(topic, p.aggregateId().toString(), p.payload());
    record.headers().add("eventId", p.id().toString().getBytes(StandardCharsets.UTF_8));
    record.headers().add("eventType", p.eventType().getBytes(StandardCharsets.UTF_8));
    // continue the trace of the request that created the event
    Context origin = W3CTraceContextPropagator.getInstance().extract(Context.root(), p, ROW);
    try (Scope ignored = origin.makeCurrent()) {
        acks.add(kafka.send(record)); // the agent's producer span is a child of the request
    }
}

Instrumenting BoxOffice

Zero-code: the Java agent

The OpenTelemetry Java agent attaches to the JVM and instruments hundreds of libraries — Spring MVC/WebFlux, JDBC, HikariCP, Kafka clients, HTTP clients, Logback — by bytecode injection, with no code changes. It also puts trace_id and span_id into the logging context (MDC), so structured logs carry them automatically.

# Dockerfile (excerpt) — pin a version; check the latest release before upgrading
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.32.0/opentelemetry-javaagent.jar /otel/opentelemetry-javaagent.jar
ENTRYPOINT ["java", "-javaagent:/otel/opentelemetry-javaagent.jar", "-jar", "/app/app.jar"]

# docker-compose.yaml (excerpt) — the order service
environment:
  OTEL_SERVICE_NAME: order
  OTEL_RESOURCE_ATTRIBUTES: service.namespace=boxoffice,service.version=1.4.2,deployment.environment.name=dev
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318   # agent 2.x default protocol: http/protobuf
  OTEL_TRACES_SAMPLER: parentbased_always_on                 # keep all; the Collector decides (tail sampling)
  OTEL_METRIC_EXPORT_INTERVAL: "15000"

service.name is the most important attribute in OpenTelemetry: without it, every service appears as unknown_service:java and no dashboard can tell them apart. service.version and deployment.environment.name let you compare a canary with the stable version, or staging with production.

# lab 8: OpenTelemetry settings shared by every Java service (merged into each "environment")
x-otel: &otel
  JAVA_TOOL_OPTIONS: -javaagent:/otel/opentelemetry-javaagent.jar
  OTEL_JAVAAGENT_ENABLED: ${OTEL_ON:-true}          # false = same image, agent off
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-lgtm:4318
  OTEL_METRIC_EXPORT_INTERVAL: "10000"              # 10 s instead of 60 s
  OTEL_RESOURCE_ATTRIBUTES: service.namespace=boxoffice,deployment.environment.name=local
  OTEL_TRACES_SAMPLER: ${OTEL_TRACES_SAMPLER:-parentbased_always_on}
  OTEL_TRACES_SAMPLER_ARG: ${OTEL_TRACES_SAMPLER_ARG:-1.0}
  LOGGING_PATTERN_CORRELATION: "[%X{trace_id:-},%X{span_id:-}] "   # trace id in console logs
x-catalog: &catalog
  build: ./services/catalog
  image: boxoffice-catalog
  volumes:
    - ./infra/otel:/otel:ro          # lab 8: the Java agent
  environment:
    <<: *otel
    OTEL_SERVICE_NAME: catalog       # three replicas, ONE service

Zero-code: the Python assistant

uv add opentelemetry-distro opentelemetry-exporter-otlp
uv run opentelemetry-bootstrap -a requirements > otel-instrumentations.txt   # instrumentations for the libraries found (FastAPI, httpx, psycopg…)
uv add -r otel-instrumentations.txt
OTEL_SERVICE_NAME=assistant \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
uv run opentelemetry-instrument uvicorn app.main:app --host 0.0.0.0 --port 8000

Manual spans where the agent cannot see

Automatic instrumentation sees I/O. It cannot see business steps or waiting inside your code — the connection pool of the payment adapter in the Signals waterfall. A few manual spans in the right places are worth more than hundreds of automatic ones:

private static final Tracer tracer = GlobalOpenTelemetry.getTracer("boxoffice.payment");

ProviderConnection acquire() throws TimeoutException {
    Span span = tracer.spanBuilder("pool.acquire").startSpan();
    try (Scope ignored = span.makeCurrent()) {
        long start = System.nanoTime();
        ProviderConnection c = pool.borrow(Duration.ofMillis(500));   // bounded wait, never forever
        span.setAttribute("pool.wait_ms", (System.nanoTime() - start) / 1_000_000);
        return c;
    } catch (TimeoutException e) {
        span.recordException(e);
        span.setStatus(StatusCode.ERROR, "pool exhausted");
        throw e;
    } finally {
        span.end();
    }
}

With the agent attached, the simplest alternative is an annotation on a method: @WithSpan("seatmap.render"), with @SpanAttribute on the parameters you want recorded.

The Collector

Services could export directly to each backend, but then every service must know every backend, its credentials and its retry policy. The Collector decouples them: services send OTLP to one place, and the Collector's pipelines decide what happens next. A pipeline has three stages per signal: receivers (how data gets in), processors (what is done to it, in order), exporters (where it goes).

# otel-collector.yaml — gateway Collector for BoxOffice (sketch)
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:                 # first: refuse data instead of being OOM-killed
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  k8sattributes: {}               # add k8s.pod.name, k8s.namespace.name, k8s.deployment.name
  attributes/redact:
    actions:
      - key: payment.card_number
        action: delete
      - key: enduser.email
        action: hash
  tail_sampling:
    decision_wait: 10s            # wait for the trace's spans before deciding
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: some-of-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  batch: {}                       # last: send in batches

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }
  otlphttp/prometheus:
    endpoint: http://prometheus:9090/api/v1/otlp
  otlphttp/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, attributes/redact, tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp/prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, attributes/redact, batch]
      exporters: [otlphttp/loki]

The order of processors matters: memory_limiter first, so an overloaded Collector sheds data instead of crashing; batch last, so it batches what survived.

Deployment patterns

Sampling

Keeping every span of every request is rarely affordable (see what telemetry costs). Sampling keeps a subset of traces — complete traces, not random spans, otherwise the trees break.

Head and tail sampling are applied in sequence: a trace dropped at the head can never be rescued by the tail. The comparison below uses illustrative numbers for a busy BoxOffice day — 2,000 requests/s, 8 spans per request of 500 bytes, 1% of requests failing and 2% slower than the 2 s threshold:

Tail sampling stores less than 10% head sampling and still keeps every error trace — at the price of shipping 100% of the spans over the network to the Collector and buffering them for the decision window. A common production recipe: head sampling at 100% (or a high ratio) in the services, tail sampling in a gateway Collector, and metrics computed from all requests before sampling, so dashboards and SLOs are exact even though most traces are discarded.

Backends and a local stack

OpenTelemetry leaves the backend to you. The same OTLP data can go to any of them:

For development — and in Hands-on 8 — a single container bundles a Collector with the whole LGTM stack:

# docker-compose.yaml (excerpt)
services:
  otel-lgtm:
    image: grafana/otel-lgtm
    ports:
      - "3000:3000"   # Grafana
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

Trade-offs

How it connects

flowchart LR
    KH["Kafka message headers,<br/>span links → Messaging"] --> CTX["context propagation:<br/>traceparent, baggage"]
    CTX --> SVC["services: Java agent,<br/>Python SDK, Nginx"]
    SVC -->|"OTLP 4317 / 4318"| COL["Collector:<br/>receivers → processors → exporters"]
    COL --> RD["redaction:<br/>no cards, no e-mails"]
    COL --> TS["tail sampling:<br/>errors + slow + 5%"]
    TS --> TR["traces backend<br/>Tempo / Jaeger"]
    COL --> MET["metrics from all traffic<br/>→ SLOs, Alerting"]
    COL --> LG["logs backend<br/>Loki"]
    MET -->|"exemplars"| TR
    COL --> OPS["memory_limiter, gateway replicas<br/>→ Cloud-native"]

Palette components: tracing · metrics · logs · app (instrumented services) · apigw (must forward traceparent) · kafka (context in message headers) · payment (a black box: only the CLIENT span). Scenarios: observability-stack, event-fanout (traces that cross Kafka). Continues in: Signals, Alerting, Messaging — Kafka, Cloud-native — Kubernetes (Collector as DaemonSet or Deployment), Hands-on 8 — instrument everything.

Key takeaways