Signals

The first question during an incident is never "why?" — it is "is something wrong, and where should I look?". The second is "what exactly happened to this request?". The third is "which line of code is burning the time?". Each question needs a different kind of data, with a different cost. Emitting every detail of every request answers all of them and bankrupts you; emitting only CPU graphs is cheap and answers none. This page is about choosing the right signal for each question.

Four signals, one spectrum

Think of it as a spectrum from aggregated and cheap to detailed and expensive. Metrics compress a million requests into a few numbers per interval — you can keep them for months, but you cannot ask about one request. Logs and traces keep each request — you can ask anything about it, but you must sample, filter or expire them.

flowchart LR
    M["Metrics<br/>aggregate · cheap · long retention"] --> T["Traces<br/>per request · sampled"]
    T --> L["Logs<br/>per event · verbose"]
    L --> P["Profiles<br/>per code path"]

Metrics

The data model

A metric is a time series: a name, a set of key-value labels (also called attributes or dimensions), and a sequence of (timestamp, value) samples.

http_server_request_duration_seconds_count{job="order", http_route="/orders", http_request_method="POST", http_response_status_code="201"}  48213

Every distinct combination of label values is a separate time series that the database must index and keep in memory. This single fact explains most of the operational problems with metrics (see cardinality).

Histograms are the important one. Each bucket is a counter "how many observations were ≤ this bound": with bounds 0.1, 0.25, 0.5, 1, 2.5 s, a 0.3 s checkout increments the buckets 0.5, 1, 2.5 and +Inf. Because buckets are counters, histograms from 20 pods can be summed and a percentile estimated from the total. Newer formats — Prometheus native histograms and OpenTelemetry exponential histograms — choose bucket bounds automatically with bounded relative error, which removes the most common mistake: buckets that do not cover your SLO threshold.

Pull vs push

Both are fine; modern stacks mix them. With OpenTelemetry the default is push over OTLP to a Collector, which can also scrape existing Prometheus endpoints (OpenTelemetry).

RED, USE and the four golden signals

You do not need hundreds of metrics per service. Three well-known checklists tell you which few matter:

The golden signals are essentially RED + saturation. Saturation is the subtle one: how full a resource is, usually visible as waiting — a queue, a pool with pending requests, threads blocked. Utilization near 100% is not a problem by itself; work waiting in line is.

RED in PromQL

PromQL is the query language of Prometheus (and of Mimir, Thanos and most compatible backends). Two rules prevent most wrong queries: apply rate() to a counter before summing it (it handles counter resets when a pod restarts), and keep the le label when summing histogram buckets.

# Rate: requests per second, per route, of the order service
sum by (http_route) (
  rate(http_server_request_duration_seconds_count{job="order"}[5m])
)

# Errors: fraction (0..1) of 5xx responses
sum(rate(http_server_request_duration_seconds_count{job="order", http_response_status_code=~"5.."}[5m]))
/
sum(rate(http_server_request_duration_seconds_count{job="order"}[5m]))

# Duration: p99 latency per route, aggregated across all pods
histogram_quantile(0.99,
  sum by (le, http_route) (
    rate(http_server_request_duration_seconds_bucket{job="order"}[5m])
  )
)

# SLI style: fraction of checkouts faster than 2 s (needs a bucket at le="2")
sum(rate(http_server_request_duration_seconds_bucket{job="order", http_route="/checkout", le="2"}[5m]))
/
sum(rate(http_server_request_duration_seconds_count{job="order", http_route="/checkout"}[5m]))

"title": "Rate (requests/s)",
    "expr": "sum by (service_name) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\"}[$__rate_interval]))",
"title": "Errors (% of requests answered 5xx)",
    "expr": "100 * sum by (service_name) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\", http_response_status_code=~\"5..\"}[$__rate_interval])) / sum by (service_name) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\"}[$__rate_interval]))",
"title": "Duration p95 (s)",
    "expr": "histogram_quantile(0.95, sum by (service_name, le) (rate(http_server_request_duration_seconds_bucket{service_name=~\"$service\"}[$__rate_interval])))",

Percentiles, not averages — and never average percentiles

An average latency hides the tail: if 99 checkouts take 100 ms and one takes 10 s, the average is ~200 ms and looks fine, but one fan in a hundred waits ten seconds — and a page that makes 20 calls hits that tail on most loads (see Percentiles).

A subtler mistake is to compute p99 per pod and then average across pods. Percentiles do not compose: the p99 of the whole fleet is not the mean of the pods' p99s. The simulation below has four order pods; one of them shares a node with a noisy neighbour and is slow, and it receives 10% of the traffic.

Averaging the pods' percentiles understates the tail users actually experience. Histograms avoid the problem because their buckets are counters that can be summed before the percentile is estimated — which is exactly what sum by (le) does in the query above.

Cardinality

Cardinality is the number of distinct time series a metric produces: the product of the number of values of each label. It is the main driver of the memory, cost and query speed of a metrics backend — and the most common way teams accidentally take down their monitoring.

The table below grows the checkout latency histogram label by label. Each histogram series is really 17 series (15 buckets, _sum, _count); the memory column assumes, for illustration, about 4 KB per active series in the time-series database.

The first five labels are bounded: their number of values is small and known (routes are templates such as /orders/{id}, not /orders/42). Together they produce at most about 200 thousand series (an upper bound: not every route returns every status code) — a normal load for one Prometheus server. The last label is unbounded: it grows with the number of users, and it multiplies everything before it. No metrics system survives that.

Logs

A log is a timestamped record of a discrete event. Logs are the most familiar signal and the easiest to abuse: they are free to write and expensive to keep.

Structured logs

Write logs as structured records (JSON, or the OpenTelemetry log data model) with fixed field names, not as free text.

{
  "timestamp": "2027-03-13T10:04:12.481Z",
  "level": "WARN",
  "service.name": "payment",
  "message": "provider call slow",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "order_id": "ord_9912",
  "provider": "acme-pay",
  "duration_ms": 8120,
  "pool_wait_ms": 7900
}

The trace_id and span_id fields are what turn a log line into part of an investigation: from a slow span you jump to exactly its log lines, and from an error line you jump to the full trace. With the OpenTelemetry Java agent, they are added to the logging context automatically (OpenTelemetry).

What (not) to log

• Log events, not steps. One line "checkout completed" with ten fields beats ten lines "entering method X".
• Never log secrets or personal data: passwords, tokens, card numbers, full e-mails. Mask at the source and, as a second barrier, redact in the Collector.
• Log the decision, not just the outcome: "fallback to provider B because breaker open" explains the next hour of graphs.
• Errors once: log an exception where it is handled, not at every layer it passes through.

Querying logs

With Loki (the "L" of the Grafana LGTM stack), labels select streams and the rest of the line is parsed at query time, which keeps indexing cheap:

{service_name="payment"} | json | level="WARN" | duration_ms > 5000

Traces

A trace records the path of one request through the system as a tree of spans. A span is one timed operation — an HTTP call received or sent, a database query, a message produced or consumed — with a start time, a duration, a status and attributes.

Reading a waterfall

The trace of one slow checkout during the flash sale, drawn as the waterfall that Tempo or Jaeger shows:

trace 4bf92f35…   POST /checkout                       total 9.12 s
                                         0s        3s        6s        9s
gateway    SERVER POST /checkout          |██████████████████████████████|  9.12 s
 order     SERVER POST /orders            | █████████████████████████████|  9.08 s
  inventory CLIENT PUT /holds/{id}        | ▌                            |  0.09 s
   postgres CLIENT UPDATE seat_holds      | ▏                            |  0.02 s
  payment  SERVER POST /charges           |  ████████████████████████████|  8.95 s
   payment INTERNAL pool.acquire          |  ██████████████████████████  |  7.90 s
   payment CLIENT POST provider/charges   |                          ███▌|  1.02 s
  order    PRODUCER order-events publish  |                             ▏|  0.01 s

Read it top-down and look for the critical path — the chain of spans that determines the total time. Here it is not the provider call (1 s, slow but not dramatic): it is 7.9 s waiting for a connection from the payment adapter's pool. The provider got slower, every call held its connection longer, and by Little's law (concurrency = throughput × time) the pool of 20 connections saturated — so most of the time was spent queueing inside BoxOffice. Without the pool.acquire span you would blame the provider for nine seconds; with it, you know a bounded pool plus a timeout is the fix (Resilience patterns).

Profiles

A profile samples, many times per second, what each thread of a process is executing — the full call stack — and aggregates the samples. Continuous profiling does this all the time in production with low overhead (the tools typically target around 1–2% CPU), so you can compare "10:00 today" with "10:00 last week".

The usual visualization is the flame graph⁵: each box is a function, its width is the fraction of samples in which it was on the stack, and children sit on top of their callers. Wide plateaus at the top are where the CPU actually goes.

Profiling became the fourth OpenTelemetry signal (still maturing in 2026), and eBPF-based profilers (Parca, Pyroscope, cloud profilers) can profile every process on a node without changing the application. Traces tell you which service and operation is slow; profiles tell you which function inside it.

Dashboards

A dashboard is a set of saved queries arranged to answer a question quickly. Most dashboards fail because they try to show everything; the useful ones are designed top-down, from what users feel to what resources do.

flowchart TB
    L1["1 · User journeys and SLOs<br/>search, seat hold, checkout: SLI, budget left"]
    L2["2 · Services — RED<br/>per service and route: rate, errors, p50/p99"]
    L3["3 · Dependencies<br/>provider latency, DB, cache hit ratio, Kafka lag"]
    L4["4 · Resources — USE<br/>CPU, memory, pools, queues per pod/node"]
    L1 --> L2 --> L3 --> L4

• One dashboard, one purpose. The on-call "is checkout healthy?" view fits on one screen; deep dives live elsewhere.
• Same time axis, same units: seconds for latency, ratios (0–1) or percentages for errors, never mixed on one panel.
• Show percentiles and the SLO threshold as a line on the latency panel, so the eye compares at once.
• Annotate deploys and feature flags: most incidents start at a change.
• Dashboards as code (Grafana JSON in Git, Jsonnet/Grafonnet, Terraform): reviewed, versioned and reproducible, instead of clicked by hand and lost.

Trade-offs

How it connects

flowchart LR
    REQ["one checkout:<br/>gateway → order → payment"] --> MET["metrics: RED per route,<br/>USE per pool and topic"]
    REQ --> TR["trace: spans,<br/>critical path"]
    REQ --> LOG["structured log<br/>with trace_id"]
    MET --> HIST["histograms → p99<br/>→ Estimation: percentiles"]
    MET --> CARD["bounded labels<br/>(cardinality)"]
    CARD -->|"high-cardinality detail"| TR
    TR --> WAIT["pool.acquire wait<br/>→ Little's law"]
    WAIT --> RES["timeout + bulkhead<br/>→ Reliability patterns"]
    TR --> PROF["profiles: which function<br/>→ Performance"]
    MET --> SLI["SLI per journey<br/>→ Alerting, SLOs"]

Palette components: metrics · logs · tracing · app (RED per service) · sql (USE of the connection pool) · kafka (consumer lag) · payment (client-side RED of the provider call). Scenarios: observability-stack, event-fanout (consumer lag), cascading-failure (a slow provider saturating a pool). Continues in: OpenTelemetry, Alerting, Estimation — percentiles, Estimation — queueing, Reliability — patterns, Performance — bottlenecks (flame graphs), Hands-on 8 — RED dashboard.

Key takeaways