Methodology

"inventory is slow during the pre-sale test." Within ten minutes the team channel has four proposals: add a Redis cache, move to a bigger database instance, add more pods, rewrite the hot path in Go. Each costs days or dollars; at most one addresses the actual cause; and nobody has looked at a single metric yet. A methodology is what replaces this guessing: a systematic way to go from a symptom to the limiting resource, and from there to the one change that matters.

Anti-methods: how not to do it

Brendan Gregg catalogues the approaches that feel productive and are not¹:

The alternative is the scientific method⁵: ask a question ("why is the p99 of POST /holds 2 s at 1 500 req/s?"), form a hypothesis ("the connection pool is exhausted"), make a prediction ("pool pending > 0 and wait time ≈ the latency increase"), test it with a measurement, and only then change something — one thing — and re-measure.

Workload characterization: what load, exactly?

A test is only as good as its model of the load. Workload characterization answers four questions¹:

From logs to a workload model

Production access logs (or the gateway's metrics) are the source of truth. A query over one past on-sale gives the mix and the arrival rate:

SELECT route,
       count(*)                                         AS requests,
       round(100.0 * count(*) / sum(count(*)) OVER (), 1) AS pct,
       max(per_second)                                  AS peak_rps
FROM (
  SELECT route, date_trunc('second', ts) AS sec, count(*) OVER (PARTITION BY route, date_trunc('second', ts)) AS per_second
  FROM gateway_access_log
  WHERE ts BETWEEN '2026-09-12 09:55' AND '2026-09-12 10:30'
) t
GROUP BY route
ORDER BY requests DESC;

The result becomes the workload model that every test in this class reuses (numbers are illustrative assumptions for a stadium on-sale):

Three properties of real workloads are routinely lost in tests and must be modelled on purpose:

• Skew. Requests are not uniform over the data. During an on-sale, one event receives almost all traffic, and the best sections receive most holds — the classic hot key / hot row problem. Popularity is often modelled with a Zipf distribution.
• Sessions and think time. A fan reads the seat map for a few seconds before holding. Think time (the pause between a user's requests) shapes concurrency: with the same arrival rate, longer think times mean more simultaneous sessions (Little's Law again).
• Synchronized arrivals. Real on-sales are not a gentle ramp: hundreds of thousands of clients refresh at 10:00:00. A test that ramps over 10 minutes misses the most dangerous second of the day.

The USE method: a checklist for every resource

The USE method says: for every resource, check Utilization, Saturation and Errors². Its power is completeness — it makes you look at resources you would otherwise forget — and speed: a few minutes of checks narrow the search to one resource.

• Utilization — how busy the resource is (percent of time, or percent of capacity).
• Saturation — how much extra work is queued waiting for it.
• Errors — error events (failed allocations, dropped packets, timeouts).

Resources are not only hardware. In a Java service, the software resources — pools, locks, queues — saturate more often than the CPU does:

A walk-through: why is POST /holds slow?

During a pre-sale test at 1 500 holds/s, the p99 of POST /holds is 2.1 s (SLO: 300 ms). Following USE, top to bottom:

flowchart TD
    s["Symptom: p99 of POST /holds = 2.1 s at 1 500 req/s"] --> cpu{"inventory CPU?"}
    cpu -->|"U 45 %, run queue low,<br/>no throttling"| mem{"Memory / GC?"}
    mem -->|"GC < 2 % of time"| tp{"Tomcat threads?"}
    tp -->|"200 / 200 busy — saturated,<br/>but busy doing what?"| pool{"Hikari pool?"}
    pool -->|"10 / 10 active,<br/>180 threads pending"| db{"PostgreSQL?"}
    db -->|"CPU 35 %, but sessions in<br/>wait_event = Lock: tuple"| root["Bottleneck: row-lock contention on hot seats,<br/>held for the whole transaction"]

1. CPU of inventory: 45 % utilization, short run queue, no throttling. Not the bottleneck — adding pods would not help.
2. Memory/GC: GC uses under 2 % of time. Fine.
3. Tomcat threads: all 200 busy. Saturated — but threads are a symptom resource: they are busy waiting for something.
4. Connection pool: 10 of 10 connections active, 180 threads pending. Every hold waits for a connection: by Little's Law, 180 waiting / 1 500 per second ≈ 120 ms of pool wait on average, and much more in the tail.
5. PostgreSQL: CPU only 35 % — so a bigger instance would not help either. But pg_stat_activity shows most sessions in wait_event = Lock: tuple: transactions wait for row locks on the same few hot seats, and each transaction holds its lock while it also calls the pricing service.

The bottleneck is lock contention, amplified by long transactions. Of the four proposals in the introduction, none would have fixed it: a cache does not help writes, a bigger database has idle CPU already, more pods would add more waiters, and Go would wait on the same locks. The fixes that work — shorter transactions (call pricing before BEGIN), SELECT … FOR UPDATE SKIP LOCKED to hand fans different free seats, and a waiting room to cap concurrent holds — are discussed in Finding bottlenecks and Databases — transactions.

SELECT wait_event_type, wait_event, state, count(*)
FROM pg_stat_activity
WHERE datname = 'boxoffice'
GROUP BY 1, 2, 3
ORDER BY count(*) DESC;

#!/usr/bin/env bash
# Lab 2: print, once per second, how many requests are INSIDE the service right now.
#   bash scripts/watch-metrics.sh http://localhost:8082
URL=${1:-http://localhost:8082}
metric() {   # metric <name>: current value of an actuator metric
  curl -s "$URL/actuator/metrics/$1" | jq -r '.measurements[0].value // "?"'
}
printf '%-8s %13s %10s %11s\n' time busy_threads db_active db_pending
while true; do
  printf '%-8s %13s %10s %11s\n' "$(date +%T)" \
    "$(metric tomcat.threads.busy)" \
    "$(metric hikaricp.connections.active)" \
    "$(metric hikaricp.connections.pending)"
  sleep 1
done

RED and the golden signals: the service view

USE looks at resources from the inside. The RED method looks at each service from the outside, the way users experience it³:

• Rate — requests per second;
• Errors — failed requests per second (or as a fraction);
• Duration — the latency distribution (histograms → percentiles).

Google's four golden signals — latency, traffic, errors, saturation — are the same idea plus saturation⁴. Use them together: RED tells you which service is unhappy; USE tells you which resource inside it is the cause. In a load test, the RED metrics of each service are the thresholds; the USE metrics are what you inspect when a threshold fails. The Observability class shows how to collect both with OpenTelemetry.

Test environment parity

A load test measures the environment it runs in. Every difference from production is a reason the result may not transfer:

Because infrastructure is code (Delivery), the most reliable option is an ephemeral performance environment: created from the same Terraform/OpenTofu modules and manifests as production for the duration of the test campaign, loaded with a production-sized synthetic data set, and destroyed afterwards.

catalog:
  build: ../../services/catalog
  environment:
    DB_HOST: db
    DB_PORT: 5432
    DB_NAME: boxoffice
    DB_USER: boxoffice
    DB_PASSWORD: boxoffice
    REDIS_HOST: redis
    REDIS_PORT: 6379
  ports:
    - "8082:8082"
  depends_on:
    db:
      condition: service_healthy
    redis:
      condition: service_healthy
  deploy:
    resources:
      limits:
        cpus: "1.0"          # the same limit as in docker-compose.yaml: budgets are per replica
        memory: 768M
  healthcheck:
    test: ["CMD-SHELL", "wget -qO- http://localhost:8082/actuator/health/readiness || exit 1"]
    interval: 5s
    timeout: 3s
    start_period: 60s
    retries: 30

Test data

• Volume: same order of magnitude as production — table sizes determine index depth, plan choices and whether the working set fits in memory.
• Distribution: reproduce the skew (one hot event, popular sections) — or you will miss lock contention and hot partitions, and overestimate cache misses on the long tail.
• Uniqueness: if every virtual user logs in as the same account or holds the same seat, you test an artificial hot spot; if every request asks for a different random event, you test an artificial cold cache. Parameterize from realistic data files.
• State reset: holds and orders change the data. Restore a database snapshot before each run so that runs are comparable.
• Privacy: never copy production personal data into a test environment; generate synthetic data with the same statistical shape.

import { SharedArray } from 'k6/data';

const users  = new SharedArray('users',  () => JSON.parse(open('./data/users.json')));    // 50 000 synthetic fans
const events = new SharedArray('events', () => JSON.parse(open('./data/events.json')));   // sorted by popularity

// Zipf-like choice: low indexes (popular events) are chosen much more often
export function pickEvent() {
  const u = Math.random();
  return events[Math.floor(events.length * Math.pow(u, 3))];
}

export function pickUser() {
  return users[Math.floor(Math.random() * users.length)];
}

Warm-up: measure the steady state — and test the cold one

Right after a deploy, a JVM service is slower than it will be in five minutes: classes are loading, the JIT compiler has not yet compiled the hot methods, connection pools are filling, Redis and PostgreSQL's buffer cache are cold, and the autoscaler is at its minimum. Measuring that period mixes two different questions.

• Steady-state tests (load, soak, breakpoint) should warm up first and exclude the warm-up from the thresholds.
• Cold-start tests deliberately measure the cold system, because production has cold moments too: a deploy during traffic, a cache flush, a failover to a cold replica, new pods added by the autoscaler during the spike.

export const options = {
  scenarios: {
    warmup: {
      executor: 'constant-arrival-rate', rate: 200, timeUnit: '1s',
      duration: '3m', preAllocatedVUs: 100, exec: 'browse',
    },
    measure: {
      executor: 'constant-arrival-rate', rate: 1000, timeUnit: '1s',
      duration: '15m', preAllocatedVUs: 500, maxVUs: 2000,
      startTime: '3m', exec: 'browse',
    },
  },
  thresholds: {
    // only the 'measure' scenario counts towards pass/fail
    'http_req_duration{scenario:measure}': ['p(99)<300'],
    'http_req_failed{scenario:measure}': ['rate<0.001'],
  },
};

Noise and repeatability

Cloud environments are noisy: neighbours share hosts, networks vary, autoscalers take different paths. One run is an anecdote.

• Repeat each configuration at least three times and report the median and the spread; a difference smaller than the run-to-run spread is not a result.
• Check the load generator: CPU below ~70 %, no network saturation, enough virtual users allocated (k6 warns when an arrival-rate scenario runs out of VUs: dropped_iterations). For large tests, run k6 distributed (k6-operator on Kubernetes or a cloud service).
• Same conditions: same data snapshot, same time window, same versions; record them in the report.
• Long enough: a 2-minute test cannot reveal a connection leak, log-file growth or a memory leak — that is the job of the soak test.

How it connects

flowchart LR
    SYM["symptom:<br/>POST /holds is slow"] --> RED["RED per service<br/>→ Observability signals"]
    RED --> USE["USE per resource: CPU,<br/>thread pool, connection pool, locks"]
    USE --> BOT["the saturated resource"]
    BOT --> FIX["change ONE thing<br/>→ Bottlenecks"]
    LOGS["gateway access logs<br/>in the warehouse"] --> WL["workload model:<br/>mix, arrival rate"]
    WL --> TEST["load test<br/>→ Load testing"]
    ENV["environment parity,<br/>test data, warm-up"] --> TEST
    NOISE["repeat runs,<br/>check the load generator"] --> TEST
    TEST --> SYM

Palette components: metrics and tracing (RED for services, USE for resources) · logs, apigw and dwh (access logs turned into a workload model) · app (thread pools, CPU) · sql (connection pool, locks) · cache (warm vs cold) · waf (bots that are not part of the workload). Scenarios: monolith (breakpoint profile), observability-stack. Continues in: Observability — signals, Load testing, Bottlenecks, Databases — transactions, Cloud-native — delivery (ephemeral performance environments).