System Architecture · Insper

Performance engineering and load testing

Strategies to test high-performance systems — and to believe the results

The test that said "all green"

200 virtual users, p95 = 180 ms on staging. On the day, the on-sale fell over at 10:02. What went wrong?

  • Staging had a tenth of the data — every query hit a warm cache.
  • The virtual users (VUs) waited for each response: when the system slowed, the test slowed with it.
  • Ten minutes never reached the point where the connection pool ran out.
  • Nobody had written down what "good enough" meant.
  • The report showed averages.

The vocabulary

Term Meaning
Service time S time a resource actually works on a request
Waiting time time queued for a busy resource
Response time R waiting + service (+ network) — what users feel
Throughput X completed requests per second
Concurrency N requests in the system at once
Utilization U / Saturation busy fraction / queued work
  • Little's Law N = X × R — 1 800 holds/s × 0.12 s = 216 in flight.
  • Utilization law U = X × S. Report percentiles from histograms, never averages of percentiles.

Every system has a knee

  • Flat region: latency ≈ service time.
  • Knee (~70–80 % busy): queues form, p99 bends first.
  • Saturation: throughput stops growing; timeouts become errors.
  • Retrograde (Universal Scalability Law, USL): past the peak, more concurrency → less throughput.

Plan with the knee — the highest rate that still meets the service level objective (SLO) — not with the maximum.

Lab 2, catalog on 1 CPU with a pool of 10: p99 38.6 ms at 300 req/s, 223.7 ms at 400, 2,056 ms at 500 — the knee is between 300 and 400, and the pool saturated, not the CPU.

The performance-engineering loop

1 · objectivesSLOs per endpoint at the peak
→
2 · workload modelmix, arrivals, data, skew
→
3 · testproduction-like environment
→
4 · analyzecurve, RED, USE, profiles
→
5 · change ONE thingthen test again
  • 6 · Ship with a budget in continuous integration (CI); production metrics feed the next workload model.
  • Change one thing at a time — otherwise you cannot attribute the improvement.

"Fast" is not a requirement. "POST /holds p99 < 300 ms, errors < 0.1 % at 4 500 req/s" is.

Anti-methods

Anti-method Looks like
Streetlight look where the tools are easy (top), not where the problem is
Random change tweak flags until it "feels faster"
Blame someone else "it's the database" — without evidence
Ad hoc checklist replay the last incident's checklist

Scientific method: question → hypothesis → prediction → measurement → one change → re-measure.

Workload model of the on-sale (illustrative)

Endpoint Share Peak req/s p99 SLO
GET /events/{id} 35 % 10 500 200 ms
GET /events/{id}/seats 30 % 9 000 300 ms
GET /search 10 % 3 000 400 ms
POST /holds 15 % 4 500 300 ms
POST /orders 7 % 2 100 800 ms
GET /orders/{id} 3 % 900 200 ms
  • Derived from production access logs of the last on-sale.
  • Keep what tests usually lose: skew (one hot event, hot sections), think time, synchronized arrivals at 10:00:00.

USE: follow the wait, resource by resource

CPU45 %, no throttling
→
memorygarbage collection < 2 %
→
Tomcat threads200/200 — waiting on what?
→
Hikari pool10/10, 180 pending
→
PostgreSQLCPU 35 % · Lock: tuple
  • Utilization, Saturation, Errors for every resource — hardware and software (pools, locks, queues).
  • Bottleneck: row-lock contention, made worse by long transactions.

Of "add a cache, a bigger database, more pods, rewrite in Go", which one helps?

  • None: a cache does not help writes, the database CPU is idle, more pods add waiters, Go waits on the same locks.

RED + USE

RED — per service

  • Rate, Errors, Duration
  • which service is unhappy?
  • becomes the thresholds

USE — per resource

  • Utilization, Saturation, Errors
  • which resource inside it is the cause?
  • what you inspect when a threshold fails

Make the test environment tell the truth

  • Parity: same instance types, limits, pool sizes, Java virtual machine (JVM) flags, Transport Layer Security (TLS), gateway — from the same infrastructure as code.
  • Data: production-sized, skewed, synthetic (no personal data); restore a snapshot before each run.
  • Dependencies: mocks with realistic latency (payment ≈ 300 ms), not 1 ms.
  • Warm-up: the just-in-time compiler, pools, caches — a separate scenario, excluded from thresholds; test cold starts on purpose.
  • Noise: ≥ 3 runs; check the load generator (CPU, dropped_iterations).

Six test types

Type Question Shape
Smoke does it work at all? 1–2 VUs, 1 min
Load SLOs at the expected peak? ramp → plateau 20 min
Stress graceful above the peak? recovers? steps 1× → 1.5× → 2× → back
Spike absorbs 10:00:00? autoscaling lag? baseline → 1.5× in 30 s
Soak leaks, slow growth? average load, 4–24 h
Breakpoint where is the knee? what breaks first? ramp until thresholds abort

Same user flows (lib/boxoffice.js), different load shapes.

Open vs closed workload models

Response time R Closed: 200 VUs, Z = 1 s → N/(R+Z) Open: 182 req/s
0.1 s 182 req/s 182 req/s
0.5 s 133 req/s 182 — ⚠️ queue grows
2.0 s 67 req/s 182 — 🔴 queue explodes
  • Closed (constant-vus, ramping-vus): fixed populations that wait for answers.
  • Open (constant-arrival-rate, ramping-arrival-rate): internet users who do not.
  • VUs needed ≈ rate × iteration duration; dropped_iterations must be 0.

A load test in k6

const flow = (exec, share, iterSeconds) => ({
  executor: 'ramping-arrival-rate', exec, startRate: 0, timeUnit: '1s',
  preAllocatedVUs: Math.ceil(PEAK * share * iterSeconds),   // Little's Law
  maxVUs: Math.ceil(PEAK * share * iterSeconds * 3),
  stages: [ { target: PEAK * share, duration: '5m' },
            { target: PEAK * share, duration: '20m' },
            { target: 0, duration: '2m' } ],
});
export const options = {
  scenarios: { browse: flow('browse', 0.55, 0.2), hold: flow('hold', 0.20, 3.5) },
  thresholds: {
    'http_req_duration{name:POST /holds}': ['p(99)<300'],   // the SLO
    http_req_failed: ['rate<0.001'],
    dropped_iterations: ['count==0'],
  },
};

Name the URLs (tags: { name }), declare 409 = expected, count seats held.

Coordinated omission

closed loopsends, waits for the answer
server frozen 1.5 s →
1 slow samplethe other ~150 never sent
→
p99 = 1 ms ✓
open modelsends every 10 ms, regardless
server frozen 1.5 s →
~150 slow samplesmeasured from the intended send
→
p99 = 1.37 s ✗
  • Same server, four freezes a minute: the closed loop records 595 fewer samples — exactly the slow ones.
  • Fix: open-model generators (k6 arrival-rate, wrk2, vegeta, Gatling injectOpen), timeouts counted at full duration, client vs server percentiles.

Simulator: two reports of the same minute

4 freezes of 1.5 sclosed p99 1.0 ms ✓ · open p99 1.37 s ✗
→
1 freeze of 0.5 s0.8 % of the test: even the open p99 is 1.0 ms
→
8 freezesclosed p99.9 = 1.5 s, p99 still 1.0 ms
  • The load-testing page's coordinated-omission simulator: change the freezes, play the minute, compare samples and verdicts.
  • A p99 sees only a tail wider than 1 % of the requests: short stalls need p99.9 and max.

However much the server freezes, the closed loop keeps omitting the requests that would have waited.

Spike, soak, breakpoint — what to read

  • Spike: errors during the jump, minutes back to SLO; compare with and without pre-scaling.
  • Soak: trends, not totals — heap after garbage collection, connections, p99 in hour 8 vs hour 1.
  • Breakpoint: abortOnFail thresholds stop it; output = knee + first saturated resource. Never in production.
  • Stress: graceful = fast 429/503 + recovery; catastrophic = timeouts → retries → stays down.

Load + chaos together

Fault during the load test Hypothesis
payment +3 s (Toxiproxy, xk6-disruptor) breaker opens < 10 s; browsing and holds keep SLO
kill 1 of 3 inventory pods < 1 % errors for < 30 s; self-recovers
Redis down 60 s fallback to SQL; p99 < 1 s; no stampede on return

Resilience patterns are only proven at the peak. Lab 9: browse p50 9.04 s unprotected, 6.5 ms with the patterns.

Reading the curve and the symptoms

  • Latency ↑ and CPU ~100 % → CPU-bound → CPU flame graph.
  • Latency ↑ and CPU low → waiting: pool, lock, I/O, dependency → wall-clock profile, USE.
  • Throughput falls past a point → contention, retries (retrograde).
  • Bimodal histogram → two paths (cache hit/miss).
  • Slow growth at constant load → leak.
  • Client p99 ≫ server p99 → queueing before the service.

Profiling: which code?

  • Flame graph: width = share of samples; y = stack; x is not time.
  • async-profiler (JVM): asprof -d 30 -e wall -t -f wall.html 1 · -e cpu · -e lock · -e alloc.
  • py-spy (Python): py-spy record --pid 1 --idle -o wall.svg · top · dump.
  • Profile at the plateau; distroless → ephemeral debug container.

POST /holds wall-clock: 33 % row-lock wait, 28 % HTTP call inside the transaction, 24 % pool wait. A CPU profile would have blamed JSON.

The database: find, explain, fix

EXPLAIN (ANALYZE, BUFFERS) SELECT ... FROM seats
WHERE event_id = 4242 AND status = 'AVAILABLE' ORDER BY section, row_label, seat_number LIMIT 8;
-- Seq Scan ... Rows Removed by Filter: 5968520   Buffers: read=59861   412 ms

CREATE INDEX CONCURRENTLY seats_available_idx
    ON seats (event_id, section, row_label, seat_number) WHERE status = 'AVAILABLE';
-- Index Scan using seats_available_idx ...       Buffers: hit=5        0.08 ms
  • Start from pg_stat_statements, sorted by total time.
  • Hot rows: short transactions, FOR UPDATE SKIP LOCKED, a waiting room.
  • Pool = throughput × time held: 1 500/s × 46 ms = 69 → move the HTTP call out → 9. Then PgBouncer.

One change at a time: the bottleneck moves

400/sseat-map Seq Scan
index →
700/spool 10/10, 150 pending
pricing out →
1 100/srow locks
SKIP LOCKED →
1 800/sinventory CPU 85 %
10 pods + PgBouncer →
3 400/sprimary CPU 80 %
waiting room →
3 000 admittedp99 140 ms · the limit is chosen
  • Keep a log: change, new knee, what saturated next — the evidence behind every capacity number.
  • Adding capacity where the bottleneck is not buys nothing (playground monolith: 12 replicas = 6 replicas, at a higher cost).

Capacity planning

replicas = ⌈ peak / (knee × (1 − headroom)) ⌉, then survive a zone loss

  • Holds: ⌈4 500 / (275 × 0.75)⌉ = 22 pods → ⌈22 × 3/2⌉ = 33 across 3 zones.
  • Waiting room at 3 000 holds/s → 24 pods, bounded database load.
  • Headroom for forecast error, sub-minute bursts, failures, deploys, autoscaling lag.
  • Connections: 4 500 × 6 ms ≈ 27 active — PgBouncer in front of the primary.

Autoscaling arrives late

10:00:00500 → 4 500 holds/s
→
metrics15–60 s
→
Horizontal Pod Autoscaler (HPA)≤ 15 s
→
schedule / new nodes – 3 min
→
JVM ready20–60 s
  • 1–4 minutes: 4 500/s demand vs 1 100/s capacity for 120 s = 408 000 requests of backlog.
  • Known peaks: pre-scale at 09:45, admission control, warm the new pods.

Playground: boxoffice-hpa (lab 10) — raise the HPA minimum to 4 pods and the drops disappear.

Performance budgets in CI

pull request
→
compose up+ warm-up
→
absolute gatethresholds with margin
→
relative gate> 15 % slower than main?
→
merge
  • Deterministic budgets never flake: SQL statements per request (catches N+1), payload size, image size, startup time.
  • Lab 11: p95 < 300 ms against ~7 ms measured; the image job needs it — a broken budget is never built.
  • Full campaign (load, spike, breakpoint, soak) nightly or before events.

The concept map

Component Question it answers Taught in
client · mobile How does the real load arrive? Methodology · Load testing
app How much can one replica take within the SLO? Overview · Bottlenecks
sql Which query or lock limits the path? Bottlenecks
cache Are we measuring a warm cache? Methodology (warm-up)
metrics · tracing Which resource saturates first? Methodology · Bottlenecks
queue · serverless How many replicas, how much do we admit? Capacity planning
thirdparty What if a dependency slows down under load? Load testing (+ chaos)

Performance adds no boxes to the palette: it measures the ones you have. Scenarios: monolith, flash-sale, serverless-burst, cascading-failure.

Takeaways

  1. Response time = service + waiting; N = X × R; plan with the knee.
  2. Method before tools: workload model, then RED per service and USE per resource.
  3. Test a production-like system with realistic, skewed data — warmed up, and cold on purpose.
  4. Open model for internet traffic; thresholds = SLOs per endpoint; dropped_iterations = 0.
  5. Beware coordinated omission; trust percentiles from histograms — and look past p99 for short stalls.
  6. Wall-clock profiles and EXPLAIN ANALYZE find the real bottleneck; change one thing at a time.
  7. Capacity = demand / (knee × headroom) + zone loss; pre-scale known peaks; budgets in CI.

Outline: (1) vocabulary, the curve and the knee; (2) methodology: workload model, USE/RED, environments and data; (3) load testing: six test types, open vs closed, thresholds, coordinated omission, chaos under load; (4) bottlenecks: profiling and the database; (5) capacity planning and performance budgets in CI.

Collect guesses first. Each item maps to a part of the class: environment and data, open vs closed model, test types, thresholds as SLOs, percentiles.

Show the two plots of the overview page (M/M/8 latency vs load, USL throughput vs N). Then run the monolith scenario with the breakpoint profile in the playground.