Percentiles & Tail Latency

The BoxOffice dashboard says the seat-map endpoint has an average latency of 55 ms. Everyone is happy — except the fans, who complain on social media that the seat map "takes forever" during on-sales. Both are right. The average is 55 ms; but one request in a hundred takes more than half a second, and a fan who opens the seat map, refreshes it, zooms into two sections and tries three holds makes enough requests to hit one of those slow ones almost every visit. This page is about the slow requests: how to measure them, why they matter more than the average suggests, and why they get worse as systems become more distributed.

Why averages lie

Latency distributions are not bell curves. Most requests are fast and similar; a minority — hitting a cache miss, a garbage-collection pause, a lock, a retry, a cold connection — are much slower. The result is a long tail to the right. The plot shows a synthetic but realistic distribution for 100,000 requests: 97% are "normal" (around 40 ms) and 3% hit a slow path (around 400 ms):

The mean (about 55 ms) describes no actual request: it is pulled up by the tail, yet it hides how bad the tail is. The median says what a typical request experiences. The p99 and p99.9 say what the unlucky requests experience — and they are roughly 12× and 28× slower than the median (≈ 510 ms and ≈ 1.1 s vs ≈ 40 ms). A dashboard that shows only the average would never reveal that.

Percentiles

The p-th percentile (written pXX) of a set of latencies is the value below which XX% of them fall¹:

To compute a percentile exactly, sort the values and take the one at position ⌈p × n⌉ (the nearest-rank method): with 1,000 requests, p99 is the 990th fastest. In production, systems do not store every latency; they record histograms — counts of requests per latency bucket (0–5 ms, 5–10 ms, …) — and estimate percentiles from the buckets. Prometheus histograms, HdrHistogram and OpenTelemetry exponential histograms all work this way (class 9).

Why the tail matters more than "1%" suggests

1. Users make many requests. A fan's visit during an on-sale (event page, seat map, refreshes, holds, payment, confirmation) easily involves 50 requests. The probability that at least one is slower than the p99 is \(1 - 0.99^{50} \approx 39\%\). The "1% tail" is felt by four in ten visits.
2. The slowest users are often the best customers. Fans with many orders, organizers with large events, accounts with a lot of data — their requests do more work and land in the tail. Amazon famously set its internal latency targets at the p99.9 for this reason².
3. The tail is where overload starts. As the queueing curves show, the p99 rises long before the mean when a component approaches saturation. It is the earliest warning you get.

Tail amplification under fan-out

Modern backends rarely answer a request with a single call. The BoxOffice seat-map page calls inventory (availability), catalog (venue layout and prices), order (the fan's existing tickets) and a recommendations service, in parallel; the search service queries 20 index shards in parallel and merges the results. A request that fans out to N backends in parallel is as slow as the slowest of them.

If each backend call is slow (above its p99) with probability q = 1%, independently, the probability that the user request waits for at least one slow call is:

This is the central observation of Dean and Barroso's The Tail at Scale³: with a fan-out of 100, a backend whose p99 is 1 second makes 63% of user requests take more than 1 second. In a large distributed system, the backend's tail becomes the user's median. The more you split a system into parallel pieces (shards, microservices), the more the rare slowness of each piece dominates.

Where the tail comes from

Tail latency is rarely caused by the "normal" code path. Typical causes³:

Taming the tail

You cannot eliminate variability; you can design so that it does not reach the user.

• Hedged requests: send the request to one replica; if it has not answered within, say, the p95 latency, send a second copy to another replica and use whichever answers first. Only ~5% of requests are duplicated, but the tail shrinks dramatically. Dean and Barroso report a Google case where hedging after 10 ms cut the p99.9 of a 1,000-key read from 1,800 ms to 74 ms with only 2% more requests³. Hedge only idempotent reads.
• Tied requests: send to two replicas at once, each knowing about the other; the first to start processing cancels the other. Removes queueing variability at the cost of a little coordination.
• Reduce fan-out: fewer, larger partitions for latency-critical paths; precompute aggregates (a materialized seat-map summary instead of calling five services).
• Degrade gracefully: give each backend a deadline and return a partial answer — the seat map without the recommendations panel is still a seat map.
• Take work off the critical path: what does not need to finish before the response (e-mails, analytics, search indexing) goes to a queue.
• Keep utilization low on latency-critical components, and isolate them from batch work.

Latency budgets

A user-facing target must be split across the components on the path. For BoxOffice checkout (p99 < 1.5 s):

Summing p99s is pessimistic (they rarely all happen on the same request), which is exactly what you want in a budget. When a step exceeds its budget, you know whose problem it is. Each budget also sets that component's timeout — a timeout longer than the caller's whole budget protects nothing.

Measuring percentiles honestly

Two traps make tail latency look better than it is:

• Coordinated omission: a load generator that sends the next request only after the previous one returns stops sending when the server stalls — so it never records the requests that would have arrived during the stall, and the tail disappears from the results. Use tools that keep a constant arrival rate (open model: k6's constant-arrival-rate, wrk2) and compare intended vs actual send times⁴. Class 15 covers it in depth.
• Measuring at the wrong place: the server's own timer misses the time spent in the accept queue, the load balancer and the network. Measure where the user is (client-side, or at the edge), and compare with server-side numbers.

export const options = {
  scenarios: {
    open: {
      executor: 'constant-arrival-rate',
      rate: RATE,
      timeUnit: '1s',
      duration: DURATION,
      preAllocatedVUs: Math.max(20, RATE),   // VUs ready to start requests
      maxVUs: Math.max(200, RATE * 10),      // k6 may add more if requests get slow
    },
  },
  summaryTrendStats: ['avg', 'med', 'p(95)', 'p(99)', 'max'],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    'http_req_duration{name:detail}': ['p(99)<500'],
    'http_req_duration{name:list}': ['p(99)<1000'],
  },
};

Try it: in the scale-out scenario, run the load profile and note the p50 and p99. Then use chaos at low to make one replica occasionally slow. Which percentile moves first? By how much does the p99 move compared with the p50?

How it connects

flowchart LR
    DIST["long-tailed latency"] --> PCT["p50, p99, p99.9<br/>from histograms"]
    PCT --> SLO["latency SLOs<br/>→ Reliability"]
    PCT --> OBS["merge histograms,<br/>never average → Observability"]
    FAN["fan-out N<br/>1 − (1 − q)ᴺ"] --> AMP["the backend tail becomes<br/>the common case (large N)"]
    DIST --> SRC["sources: GC, cache misses,<br/>cold starts, slow dependencies"]
    AMP --> TAME["hedging, partial results,<br/>less fan-out, headroom"]
    SRC --> TAME
    TAME --> BUD["latency budgets<br/>and timeouts → Reliability"]
    PCT --> CO["measure at the client,<br/>coordinated omission → Performance"]

Palette components: search (20 shards: the fan-out tail) · app (a page that aggregates several services; hedging to another replica) · cache (misses are a tail source) · serverless (cold starts) · apigw (where the SLO is measured) · payment (the provider dominates the checkout budget) · metrics (histograms, never averaged percentiles). Scenarios: scale-out (with chaos), search-autocomplete. Continues in: Availability Math, Observability — SLOs, Reliability — timeouts, Observability — signals, Performance — load testing, Hands-on 2.

Key takeaways