Skip to content

Overview

Last year BoxOffice's biggest on-sale went down at 10:02. The team had load-tested: 200 virtual users against staging, p95 of 180 ms, "all green". In the post-mortem they found five problems with that test. Staging had a tenth of production's data, so every query hit a warm cache. The virtual users waited for each response before sending the next request, so when the system slowed down, the test politely slowed down with it and never saw the queue that real fans built. The test lasted ten minutes and never reached the point where the database connection pool ran out. Nobody had written down what "good enough" meant before running it. And the report showed averages.

Performance engineering is the discipline that prevents that post-mortem. It is not "run a tool and look at a graph": it is a method — state the objective, model the workload, generate load that behaves like real users, measure what users feel, find the one resource that limits the system, fix it, and measure again. This class teaches the method and the tools, using BoxOffice's flash sale as the running test case.

The story so far

Chapter 15 of the story of the course. Every chapter added a box to BoxOffice because a number asked for it: an edge and stateless replicas (class 4, class 5), a database that sells each seat once (class 6), a cache (class 7), a waiting room (class 8), SLOs and breakers (class 9, class 10), Kubernetes (class 11) — and, since class 14, changes arrive faster than anyone can re-check them by hand. Each box was measured alone, in its own lab. This class answers: does the whole design hold the next on-sale — 2 M fans at 10:00, p99 under 300 ms — how do we prove it, and how do we keep it proven on every change?

Learning objectives

By the end of this class you should be able to:

  • Define latency, service time, throughput, concurrency, utilization and saturation, and relate them with Little's Law.
  • Read a throughput–latency curve, locate the knee and the saturation point, and explain why latency explodes as utilization approaches 100 %.
  • Apply a performance methodology — workload characterization, the USE method, RED metrics — to find the bottleneck of a system instead of guessing.
  • Design and write smoke, load, stress, spike, soak and breakpoint tests in k6, choosing between open (arrival-rate) and closed (virtual-user) workload models, with thresholds that encode the SLOs.
  • Detect coordinated omission and other measurement errors (unrealistic data, cold caches, a saturated load generator) that make a test lie.
  • Diagnose a bottleneck with profiling (flame graphs from async-profiler and py-spy), database tools (EXPLAIN ANALYZE, pg_stat_statements) and pool sizing.
  • Plan capacity with Little's Law and headroom, and enforce performance budgets in CI.

Roadmap

  1. This overview — the vocabulary, the throughput–latency curve and its knee, and the performance-engineering loop.
  2. Methodology — anti-methods, workload characterization, the USE and RED methods, test environments, test data and warm-up.
  3. Load testing — the six test types in k6, open vs closed models, thresholds as SLOs, coordinated omission, chaos under load, and the playground profiles.
  4. Finding bottlenecks — reading results, profiling and flame graphs, database bottlenecks, pool sizing.
  5. Capacity planning — Little's Law and headroom, autoscaling lag, a capacity model for the on-sale, and performance budgets in CI.
  6. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

The vocabulary of performance

Precise words prevent most performance arguments. The Estimation class introduced them; here they become measurement targets.

Term Definition BoxOffice example
Service time (S) Time a resource spends actually working on one request 4 ms of CPU and 6 ms of database time to place a seat hold
Waiting (queueing) time Time a request spends waiting for a busy resource Waiting for a free connection in the pool
Latency / response time (R) What the client measures: waiting + service + network POST /holds took 230 ms end to end
Throughput (X) Completed requests per unit of time 1 800 holds per second
Concurrency (N) Requests in the system at the same moment (in service or waiting) 120 holds in flight
Utilization (U) Fraction of time a resource is busy Database CPU busy 82 % of the time
Saturation Work the resource cannot serve yet — the queue length 40 requests waiting for a connection
Percentile (p50, p99, p99.9) The value below which that fraction of observations falls p99 = 300 ms: 1 hold in 100 is slower than 300 ms
Error rate Fraction of failed requests — including timeouts 0.3 % of holds returned 503

Three relations hold for any stable system and are worth memorizing:

  • Little's Law: \(N = X \times R\) — concurrency equals throughput times response time1. At 1 800 holds/s and 0.12 s each, there are on average 216 holds in flight: that is how many threads, connections or async slots the path needs.
  • Utilization law: \(U = X \times S\) — a database spending 0.4 ms of CPU per query (per core) at 2 000 queries/s on one core is 80 % busy.
  • Response time = service time + waiting time: tuning code shortens service time; adding capacity or reducing load shortens waiting time. Near saturation, waiting dominates.

Averages lie

A mean of 120 ms is compatible with "everyone waits 120 ms" and with "99 % wait 20 ms and 1 % wait 10 s". Users — and SLOs — live in the tail. Report percentiles from histograms (HDR or native histograms), never the average of percentiles from different machines or intervals: percentiles cannot be averaged. See Percentiles and tail latency.

The throughput–latency curve and its knee

Every system, from one thread to a data centre, shows the same shape when you increase the load. The left plot models one inventory pod with 8 worker threads and 20 ms of mean service time — capacity 400 requests per second — as an M/M/8 queue. The right plot uses the Universal Scalability Law (USL), which adds contention (σ, waiting for shared resources) and coherency (κ, the cost of keeping shared state consistent) to the ideal linear scaling2:

\[ X(N) = \frac{\lambda N}{1 + \sigma (N-1) + \kappa N (N-1)} \]
2026-11-01T21:54:32.538074 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 50 100 150 200 250 300 350 400 offered load (req/s); capacity = 400 0 50 100 150 200 250 300 350 400 response time (ms) the knee: > 75 % busy Latency vs load (M/M/8, 20 ms) mean p99 0 20 40 60 80 100 120 concurrent requests N 0 100 200 300 400 500 600 700 throughput (req/s) peak at N ≈ 40 then retrograde Throughput vs concurrency (USL) USL throughput linear (ideal)

Read the two plots together:

  1. The flat region. At low load, latency is just service time. Doubling the load barely changes it. Most systems spend most of their life here, which is why they seem fast.
  2. The knee. Past roughly 70–80 % utilization, requests start finding all servers busy and queue. Latency — and the p99 first — bends upward. The knee is the useful capacity of the system: the highest throughput at which the latency SLO still holds.
  3. Saturation. As offered load approaches capacity, queues grow without bound; latency goes vertical and timeouts turn into errors. Throughput stops growing.
  4. Retrograde scaling. With contention and coherency costs, adding more concurrency past the peak reduces throughput — threads fight over locks, caches thrash, connections time out and retry. A system pushed beyond its peak can serve fewer tickets per second than one held just below it, which is the core argument for admission control and waiting rooms.

BoxOffice: the knee is the number you plan with

If one inventory pod keeps p99 under 300 ms up to 275 req/s and collapses at 360, the capacity number for planning is 275, not 360. For an on-sale expecting 4 500 hold requests per second at peak, that means ⌈4 500 / 275⌉ = 17 pods before headroom — 22 at 75 % of the knee, 33 to survive the loss of one of three zones — and a database that can take 33 pods' worth of connections. The Capacity planning page builds this model end to end.

In the BoxOffice code: the knee of catalog, measured (lab 2)

Hands-on 2 — Measure first finds the knee exactly as this section describes it: sweep.sh runs the open-model load.js once per arrival rate, one minute each, after a 30 s warm-up that is not recorded, and waits 10 s between steps so that queues drain. In the lab's example run (catalog limited to 1 CPU, a pool of 10 database connections), p99 is 38.6 ms at 300 req/s, 223.7 ms at 400 and 2,056 ms at 500 — where k6 also dropped 410 iterations. The knee is between 300 and 400 req/s, and the resource that saturated was the connection pool, not the CPU.

scripts/sweep.sh (one load test per arrival rate)
#!/usr/bin/env bash
# Lab 2: run load.js once per arrival rate and save one k6 summary per run.
#   bash scripts/sweep.sh 25 50 100 150 200 300 400
# Then:  uv run scripts/summarize.py results/lab2/rate-*.json
set -euo pipefail
OUT=${OUT:-results/lab2}
DURATION=${DURATION:-1m}
mkdir -p "$OUT"
echo ">>> warm-up: 30 s at RATE=$1 (not recorded)"
k6 run -q -e RATE="$1" -e DURATION=30s load/load.js > /dev/null || true
for rate in "$@"; do
  echo ">>> RATE=$rate req/s for $DURATION"
  # || true: a run that crosses a threshold still produces a summary; keep sweeping
  k6 run -q -e RATE="$rate" -e DURATION="$DURATION" \
     --summary-export "$OUT/rate-$rate.json" load/load.js > /dev/null || true
  sleep 10   # let queues drain before the next step
done

Try it: this playground scenario is the simplest possible system — client → app server → SQL database — running the breakpoint profile, which increases the arrival rate step by step until something breaks. Press Run and watch three numbers together: throughput, p99 latency and the utilization of each component. Note the rate at which p99 starts to bend (the knee) and which component reaches 100 % first (the bottleneck). Then give that component more capacity and run again: does the knee move, and which component becomes the new bottleneck?

Open the full playground to add a cache or read replicas and see how the curve changes.

The performance-engineering loop

Performance work is a scientific loop, not a one-off event before launch3:

flowchart LR
    slo["1. Objectives<br/>SLOs per endpoint,<br/>expected peak"] --> model["2. Workload model<br/>mix, arrival pattern,<br/>data"]
    model --> test["3. Test<br/>k6 scenario in a<br/>production-like env"]
    test --> analyze["4. Analyze<br/>curve, USE, RED,<br/>profiles"]
    analyze --> fix["5. Change ONE thing<br/>(code, config,<br/>capacity)"]
    fix --> test
    analyze --> ship["6. Ship + budget in CI<br/>monitor in production"]
    ship --> slo
  • Objectives first. "Fast" is not a requirement. "POST /holds: p99 < 300 ms and errors < 0.1 % at 4 500 req/s" is. Without it, no test can pass or fail.
  • Change one thing at a time and re-run the same test — otherwise you cannot attribute the improvement.
  • Production closes the loop. Real traffic is the best workload model: production metrics and traces feed the next test, and the performance budget in CI keeps regressions from coming back.
Level Question Tools
Micro-benchmark Is this function/data structure faster? JMH (Java), pytest-benchmark
Component test How many req/s does one inventory pod sustain within the SLO? k6 against one pod, profiler attached
System test Does the whole purchase path hold at the on-sale peak? k6 scenarios against a production-like environment4
Production Is it holding right now? Where is the tail coming from? RED dashboards, traces, continuous profiling, SLO alerts

In the interview

When an interviewer asks "how would you know this design handles the peak?", do not say "we would load-test it". Say what you would test and how you would read it: "An open-model k6 test at the expected peak arrival rate with the realistic endpoint mix, thresholds on p99 and error rate per endpoint, run long enough to reach steady state on warm caches; then a breakpoint test to find the knee and compute the headroom; the USE method on the database and the connection pool to find what saturates first."

Best practices

  • Keep the load tests in the service repository, versioned with the code — a script that drifts from the API measures an endpoint that no longer exists. In BoxOffice: the load/ folder of the repository, from lab 2 to lab 11.
  • Record every run with its conditions — commit, configuration (pool size, CPU limit), data snapshot and machine next to the numbers; a p99 without its conditions cannot be compared with the next one. In BoxOffice: results/lab2.md and the sweep folders of lab 2.
  • Publish capacity as a number per replica, with its SLO — "catalog: 300 req/s at p99 < 300 ms on 1 CPU" is what capacity planning, autoscaling and the waiting room's admission rate are computed from. In BoxOffice: the knee measured in lab 2.
  • Read the client and the server on one time axis — k6 says that the p99 broke, the services' RED and USE metrics say where; tag each run and put both in the same Grafana. In BoxOffice: the otel-lgtm stack of lab 8.
  • Gate every change on a short budget; run the full campaign before each big event — a 1-minute test on every pull request catches the missing index; only the long load, spike and breakpoint tests prove the on-sale. In BoxOffice: the performance budget job of lab 11.

Common pitfalls

  • No objective: a test without pass/fail criteria produces graphs, not decisions.
  • Testing the wrong system: tiny data sets, warm caches, a different instance size, no TLS, no gateway.
  • Closed-model tests for open-world traffic: the load generator slows down with the system and hides the queue (coordinated omission).
  • Averages and single runs: report percentiles from histograms; repeat runs and look at the variance.
  • Tuning without a bottleneck: optimizing a component at 20 % utilization changes nothing.
  • A saturated load generator: if the k6 machine is at 100 % CPU, you measured k6.

The performance concept map

Performance engineering does not add new boxes to the palette: it measures the ones you already have. The map links each palette component (green) to the concepts used to load it, observe it and size it (blue):

flowchart LR
    client["client · mobile<br/>(k6 generates them)"]:::comp --> edge["apigw · lb"]:::comp
    edge --> app["app"]:::comp
    app --> sql["sql"]:::comp
    app --> cache["cache"]:::comp
    queue["queue<br/>(waiting room)"]:::comp --> app
    obs["metrics · tracing"]:::comp -.- app
    client -.- c1["open vs closed model,<br/>6 test types, thresholds"]:::tech
    app -.- c2["Little's Law, the knee,<br/>USL"]:::tech
    sql -.- c3["EXPLAIN ANALYZE, hot rows,<br/>pool sizing"]:::tech
    cache -.- c4["warm-up, cold cache,<br/>hit ratio"]:::tech
    obs -.- c5["USE, RED,<br/>flame graphs"]:::tech
    queue -.- c6["headroom, autoscaling lag,<br/>admission control"]:::tech
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed
Palette component Question it answers Concepts Taught in Scenarios
client · mobile How does the real load arrive? workload model, open (arrival rate) vs closed (virtual users), think time, coordinated omission Methodology · Load testing flash-sale, monolith
app How much can one replica take within the SLO? service time, Little's Law, utilization law, the knee, USL, profiling this overview · Bottlenecks monolith (breakpoint profile), scale-out
sql Which query or lock limits the path? pg_stat_statements, EXPLAIN ANALYZE, hot rows, SKIP LOCKED, connection-pool sizing Bottlenecks flash-sale
cache Is the test measuring a warm cache that production will not have? warm-up, cold-start test, hit ratio, stampede Methodology read-heavy-cache
metrics · tracing Which resource saturates first? USE, RED and golden signals, flame graphs, continuous profiling Methodology · Bottlenecks observability-stack
queue · serverless How many replicas, and how much do we admit? headroom, autoscaling lag, cold starts, pre-scaling, waiting-room admission rate Capacity planning flash-sale, serverless-burst
thirdparty What happens when a dependency slows down under load? load and chaos together, Toxiproxy, timeouts, circuit breakers Load testing cascading-failure
(the pipeline) Will the next change regress? performance budgets in CI, thresholds as SLOs Capacity planning —

Key takeaways

  • Response time = service time + waiting; near saturation, waiting dominates. \(N = X \times R\) and \(U = X \times S\) connect the numbers.
  • Every system has a knee: plan with the throughput at which the latency SLO still holds, not with the maximum.
  • Past the peak, more concurrency can mean less throughput (USL) — admission control beats overload.
  • Performance engineering is a loop: objectives → workload model → test → analyze → change one thing → re-test → budget in CI.

What breaks next

Performance engineering proves this design for this workload: one ticketing system, one on-sale, a mix of about 80 % reads and 20 % holds and orders, a knee measured on BoxOffice's own pods. The method — workload model, open-model tests, the knee, headroom, a budget in CI — transfers to any system; the numbers and the boxes do not. A news feed fans each post out to millions of followers, a chat keeps millions of connections open for hours, a ride app writes every driver's location every few seconds: different numbers ask for different blocks. Recognizing which blocks of the previous fifteen chapters a new system needs, and which number asks for each, is class 16 — Case studies.

  1. LITTLE, J. D. C. A Proof for the Queuing Formula: L = λW. Operations Research, v. 9, n. 3, p. 383–387, 1961. ↩

  2. GUNTHER, N. J. Guerrilla Capacity Planning. Springer, 2007. ↩

  3. GREGG, B. Systems Performance: Enterprise and the Cloud. 2nd ed. Addison-Wesley, 2020. ↩

  4. Grafana Labs. k6 documentation — Test types. ↩