a. Overview
Goal
Add OpenTelemetry instrumentation to every Spring Boot service so that a single user request β traversing Gateway β OrderService β ProductService β generates a unified distributed trace visible in Jaeger, with trace IDs automatically injected into log lines.
You already have Prometheus scraping metrics and Grafana displaying dashboards. Metrics answer "how much" and "how often". They cannot tell you which service inside a multi-hop request chain is responsible for a 2-second tail latency. Distributed tracing answers that question.
Slides
Click the slides and use β β to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab
The Problem This Solves
Consider a slow /orders response. Prometheus shows elevated http_server_requests_seconds on the gateway, but the gateway delegates to OrderService which in turn calls ProductService and the database. The metric cannot tell you where the time went.
Without tracing, the only option is to stare at logs across four separate services and guess.
sequenceDiagram
participant C as Client
participant G as Gateway
participant O as OrderService
participant P as ProductService
participant D as Database
C->>G: POST /orders (? ms)
G->>O: POST /orders (? ms)
O->>P: GET /products/{id} (? ms)
P->>D: SELECT ... (? ms)
D-->>P: row
P-->>O: ProductDTO
O-->>G: OrderDTO
G-->>C: 200 OK (total: 2 340 ms) With tracing, every hop is measured as a span and all spans for one request share a trace ID. The Jaeger UI renders the full call graph as a flame chart so you can see the slow span immediately.
| Term | What it is |
|---|---|
| Span | One timed operation: an HTTP call, a query, a business step |
| Trace | All the spans of one request, grouped by a shared trace ID |
| Parent / child | Each span points to the span that called it |
traceparent | The W3C header that carries the context to the next service |
| OTLP | The protocol the agent uses to export spans |
| Flame chart | The Jaeger view: one bar per span, width = duration |
sequenceDiagram
participant C as Client
participant G as Gateway
participant O as OrderService
participant P as ProductService
participant D as Database
C->>G: POST /orders<br/>traceparent: 00-4bf9...36-01
note over G: span: gateway.POST /orders (2 340 ms)
G->>O: POST /orders<br/>traceparent: 00-4bf9...36-01
note over O: span: order.POST /orders (2 250 ms)
O->>P: GET /products/7<br/>traceparent: 00-4bf9...36-01
note over P: span: product.GET /products/7 (2 190 ms)
P->>D: SELECT ...
note over D: span: jdbc.query (2 180 ms)
D-->>P: row
P-->>O: ProductDTO
O-->>G: OrderDTO
G-->>C: 200 OK trace_id=4bf9...36 The bottleneck is immediately visible: the JDBC query inside ProductService accounts for almost all of the latency.
Try it
Before the lab, the same trace in the browser. It opens with the page's curl already sent: 5 spans, 180 ms. Play or β replays the page: with the slow query, the trace takes 2 340 ms and SELECT products 2 180 ms (93%); the same delay in product-service's own code leaves the SELECT at 20 ms; with product-service exporting to localhost:4317, order-service shows 2 225 ms of self time; with the 10% sampler, Jaeger keeps 6 of 100 requests. Click a span to see its tags; the headers and log lines under the waterfall carry the same trace id.
Architecture
This hands-on adds Jaeger to the observability layer and wires the OpenTelemetry Java agent into each service container:
flowchart LR
subgraph obs [Observability Layer]
prometheus@{ shape: cyl, label: "Prometheus\n:9090" }
grafana[Grafana\n:3000]
jaeger["Jaeger\n:16686 UI\n:4317 OTLP"]
prometheus --> grafana
end
subgraph api [Trusted Layer]
loadbalancer --> gateway
gateway --> order
gateway --> account
order --> product
order --> db@{ shape: cyl, label: "PostgreSQL" }
end
gateway & order & account & product e1@-->|"OTLP traces"| jaeger
prometheus e2@-.->|"scrape /actuator/prometheus"| gateway & order & account & product
internet e0@==>|":80"| loadbalancer
e0@{ animate: true }
e1@{ animate: true }
e2@{ animate: true } Two pipelines run side by side: Prometheus pulls (scrapes) the metrics from each service, while each service's agent pushes its spans to Jaeger over OTLP.
Adding Jaeger to Docker Compose
Jaeger ships as a single all-in-one container that includes the collector, query engine, and UI. For production you would run these as separate components, but for local development the all-in-one image is sufficient.
Add the following service to compose.yaml:
jaeger:
image: jaegertracing/all-in-one:1.57
container_name: jaeger
environment:
COLLECTOR_OTLP_ENABLED: "true"
ports:
- "16686:16686" # Jaeger UI
- "4317:4317" # OTLP gRPC receiver
- "4318:4318" # OTLP HTTP receiver
networks:
- store-net
Port 4317 vs 4318
The OTel Java agent defaults to gRPC on port 4317. Port 4318 is for HTTP/protobuf. Both carry the same OTLP protocol β use 4317 unless a firewall forces HTTP.
OpenTelemetry Java Agent
The OpenTelemetry Java agent is a JVM instrumentation agent β a single JAR you attach with -javaagent. It uses Java's bytecode instrumentation API to intercept Spring MVC, Spring WebClient, RestTemplate, JDBC drivers, and dozens of other libraries without any code changes.
Download the agent
Add a download step to each service's Dockerfile:
FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.4.0/opentelemetry-javaagent.jar \
/app/opentelemetry-javaagent.jar
COPY target/*.jar app.jar
ENTRYPOINT ["java", \
"-javaagent:/app/opentelemetry-javaagent.jar", \
"-jar", "/app/app.jar"]
Pin the agent version
The OTel agent version and the Spring Boot version must be compatible. v2.4.0 works with Spring Boot 3.x. Check the compatibility matrix before upgrading either dependency.
Configure each service in Docker Compose
The agent is configured entirely through environment variables. Add the following environment block to each service in compose.yaml:
Why OTEL_METRICS_EXPORTER: none?
By default the OTel agent also tries to export metrics via OTLP. You are already collecting metrics with Prometheus and Micrometer β exporting them a second time to Jaeger would create duplicates and waste resources. Setting none disables the OTel metric pipeline while leaving trace export fully enabled.
localhost inside a container
OTEL_EXPORTER_OTLP_ENDPOINT must use the Compose service name, http://jaeger:4317. Inside a container, localhost is the container itself, so an endpoint like http://localhost:4317 sends the spans nowhere and no trace reaches Jaeger.
flowchart LR
subgraph oc ["order container"]
agent["OTel agent\nexports its spans"]
lo["localhost:4317\nnobody listens here"]
end
jaeger["jaeger:4317\nthe Jaeger container"]
agent -->|"http://localhost:4317"| lo
agent -->|"http://jaeger:4317"| jaeger Name every service
Without OTEL_SERVICE_NAME, the agent reports the service as unknown_service:java. The four services end up under one name in Jaeger's Service dropdown and you cannot tell them apart.
Trace Propagation β How It Works
When the agent intercepts an outgoing HTTP call, it injects the W3C traceparent header into the request. The receiving service's agent reads that header and creates a child span under the same trace.
The traceparent format is:
00-{traceId}-{parentSpanId}-{flags}
β β β β
β β β βββ 01 = sampled, 00 = not sampled
β β βββ 16-char parent span ID (hex)
β βββ 32-char trace ID (hex) β identical on every hop
βββ version byte (always 00)
Example value seen in HTTP headers:
The trace ID 4bf92f3577b34da6a3ce929d0e0e4736 is present on every service in the chain. All spans carrying that ID are grouped into one trace in Jaeger.
sequenceDiagram
participant G as Gateway Agent
participant O as Order Agent
participant P as Product Agent
G->>G: start root span<br/>traceId=4bf9... spanId=aaa1
G->>O: POST /orders<br/>traceparent: 00-4bf9...-aaa1-01
O->>O: start child span<br/>traceId=4bf9... spanId=bbb2 parent=aaa1
O->>P: GET /products/7<br/>traceparent: 00-4bf9...-bbb2-01
P->>P: start child span<br/>traceId=4bf9... spanId=ccc3 parent=bbb2
P-->>O: 200
O-->>G: 200
P->>P: end child span β export to Jaeger
O->>O: end child span β export to Jaeger
G->>G: end root span β export to Jaeger Sampling
The 01 flags byte means the trace is sampled and will be exported. The default sampler exports 100% of traces. For production, switch to a rate-limiting sampler: set OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 to sample 10% of requests.
The decision is parent-based: the gateway decides once, at the root span, and writes the answer in the flags; every service downstream follows the flags, so a trace is never half-sampled. A request that was not sampled still has a trace ID: it shows up in the log lines, not in Jaeger.
flowchart LR
req["POST /orders\nno traceparent yet"] --> root{"gateway-service\nroot span: sample?"}
root -->|"yes, 1 in 10"| s1["traceparent ...-01\norder and product\nrecord and export"]
s1 --> j["Jaeger has\nthe whole trace"]
root -->|"no, 9 in 10"| s0["traceparent ...-00\nno service exports"]
s0 --> l["the log lines\nstill carry the trace_id"] Default (parentbased_always_on) | parentbased_traceidratio, 0.1 | |
|---|---|---|
| Traces in Jaeger | β every request | β οΈ about 1 in 10 |
| Export and storage cost | β οΈ grows with the traffic | β about a tenth |
| A rare slow request | β always there | β οΈ may not have been kept |
Correlating Traces with Logs
A trace ID in Jaeger is only useful if you can find the matching log lines. The OTel agent bridges both worlds: it injects the active trace ID and span ID into SLF4J's MDC, so every log statement emitted during a traced request automatically carries the trace context.
Add the logback bridge dependency
In each service's pom.xml:
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-logback-appender-1.0</artifactId>
<version>2.4.0-alpha</version>
<scope>runtime</scope>
</dependency>
Configure logback-spring.xml
Create src/main/resources/logback-spring.xml in each service:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>
%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36}
[trace_id=%X{trace_id} span_id=%X{span_id}] - %msg%n
</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
MDC key names
The OTel agent writes trace context under the keys trace_id and span_id in SLF4J MDC. These match the OTLP/W3C naming convention and are directly searchable in Loki, CloudWatch Insights, Splunk, and most other log aggregators.
With this configuration, every log line during a traced request will look like:
14:23:01.442 [http-nio-8080-exec-3] INFO c.example.order.OrderService
[trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7] - creating order for account 1
14:23:01.462 [http-nio-8080-exec-3] INFO c.example.order.OrderService
[trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7] - calling ProductService for product id 7
Workflow: open the slow trace in Jaeger β click the problematic span β copy the trace_id value β paste it into your log aggregator's search bar β all log lines from that exact request appear together.
flowchart LR
a["Jaeger\nslow trace"] --> b["click the\nslow span"]
b --> c["copy its\ntrace_id"]
c --> d["log aggregator\nsearch the trace_id"]
d --> e["every log line\nof that request"] Viewing Traces in Jaeger UI
Rebuild and start the full stack:
Verify Jaeger started:
Expected output:
NAME IMAGE STATUS PORTS
jaeger jaegertracing/all-in-one:1.57 running 0.0.0.0:4317->4317/tcp, 0.0.0.0:16686->16686/tcp
Generate some traffic so there are traces to explore:
curl -s -X POST http://localhost:8080/orders \
-H "Content-Type: application/json" \
-d '{"accountId": 1, "productId": 7, "quantity": 2}' | jq
Open http://localhost:16686 in your browser.
Finding a trace
- In the Service dropdown, select
gateway-service. - Leave Operation as
alland click Find Traces. - The results list recent traces sorted by timestamp. Each row shows the root operation name, total duration, and span count.
- Click any trace to open the flame chart view.
Reading the flame chart
The flame chart renders each span as a horizontal bar. The root span (gateway) is at the top. Child spans are indented below their parent. Bar width is proportional to duration β a wide bar is a slow operation.
What to look for
- A span nearly as wide as the root span is the primary latency contributor.
- Nested spans are all that wide when the slow part is deep: the culprit is the span whose bar its children do not cover (its own, or self, time).
- Spans labelled
SELECTorjdbc.querywith long duration indicate slow database queries. - Spans labelled
HTTP GETorHTTP POSTare outbound HTTP calls captured by the agent. - A span with a red
!icon recorded an exception β click it to see the full stack trace attached as a span event.
Span attributes
Clicking a span expands its tags. The agent records these automatically:
| Tag | Example value |
|---|---|
http.method | POST |
http.url | http://order:8080/orders |
http.status_code | 200 |
db.system | postgresql |
db.statement | SELECT * FROM products WHERE id = ? |
net.peer.name | db |
Adding Custom Spans
The Java agent instruments framework code automatically, but it knows nothing about your business logic. When validation, pricing calculation, or any significant processing step needs to be traced individually, add a manual span.
Add the OpenTelemetry API dependency
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
</dependency>
Spring Boot's dependency management includes opentelemetry-bom, so no version is needed here.
Instrument business logic
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
@Service
public class OrderService {
@Autowired
private Tracer tracer;
public OrderOut create(OrderIn orderIn) {
Span span = tracer.spanBuilder("validateOrder").startSpan();
try (Scope scope = span.makeCurrent()) {
span.setAttribute("order.account_id", orderIn.getAccountId());
span.setAttribute("order.product_id", orderIn.getProductId());
validateAccount(orderIn.getAccountId());
checkInventory(orderIn.getProductId(), orderIn.getQuantity());
} catch (Exception e) {
span.recordException(e);
throw e;
} finally {
span.end();
}
return persist(orderIn);
}
}
With the manual span, the call to ProductService (inside checkInventory) becomes its child: in the flame chart, validateOrder sits between order-service and product-service.
flowchart TB
g["gateway-service\nPOST /orders"] --> o["order-service\nPOST /orders"]
o --> v["order-service\nvalidateOrder (manual span)"]
v --> p["product-service\nGET /products/7"]
p --> q["product-service\nSELECT products"]
o --> i["order-service\nINSERT orders"] Always call span.end() in a finally block
If span.end() is never called, the span is never exported to Jaeger. The try-finally pattern guarantees cleanup even when an exception is thrown. The agent handles this automatically for framework code; manual spans are your responsibility.
When to add manual spans
Add manual spans when: (1) a method takes more than ~50 ms and you want to measure it independently, (2) you need to attach business-level attributes (order.id, account.tier) that are not captured by framework instrumentation, or (3) you need to record a caught exception as a span event. Do not add manual spans to trivial getters or field assignments β the noise obscures the useful data.
Checklist
Before moving on, verify:
-
docker compose psshowsjaegerrunning with ports 16686 and 4317 exposed -
http://localhost:16686loads the Jaeger UI without errors - After sending a request,
gateway-serviceappears in the Jaeger service dropdown - A trace for
POST /orderscontains at least 3 spans: gateway, order-service, product-service - Each span has
http.methodandhttp.status_codetags set by the agent automatically - Log output includes
trace_id=fields that match the trace ID shown in Jaeger