a. Overview

"Measure first" is the most useful habit in system design. Before adding a cache, a replica or ten more servers, you need to know how much load the current system takes, where it breaks and why. Guessing is how teams spend a month on a cache for a service whose real limit was a connection pool of 10. In this lab you treat catalog as a black box with a load generator in front (k6) and a few gauges inside (Actuator), and you produce the most important chart of the course: latency against throughput.

Before you start

docker compose version        # Docker Compose v2.x (or newer)
java -version                 # 21 or newer
mvn -v                        # Apache Maven 3.9+
k6 version                    # k6 v1.0+
uv --version                  # uv (runs the Python helper scripts)

Start state — the end of lab 1, with the stack running (docker compose up -d, catalog healthy):

📁 boxoffice/
├──  docker-compose.yaml        # db + catalog (port 8082)
├──  load/hello.js  load/smoke.js
├──  results/lab0.md  results/lab1.md
├──  scripts/check-env.sh
└── 📁 services/catalog/          # the service of lab 1

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you measure: a hypothesis

The playground runs the same experiment in seconds. Form a hypothesis first, then check it against the real service.

Try it: the monolith scenario below starts with the breakpoint profile — the load grows until something gives. Watch which component turns red first and how p99 behaves just before. Then write down your guess for catalog: "it will break at about ___ req/s because ___ saturates first".

1. Closed or open workload?

In lab 0 each virtual user (VU) sent a request, waited for the answer, and sent the next one. That is a closed model: when the server slows down, the load generator slows down with it, and the test politely stops pushing. Real fans do not wait for each other: at 10:00 they arrive at whatever rate they arrive. That is an open model.

2. The scripts

The load test keeps a constant arrival rate (RATE) for a fixed time, with a realistic mix: 80% event details and 20% city listings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

// Lab 2: load test with an OPEN workload model. k6 starts RATE new requests per second,
// whether or not the previous ones have finished, like real fans arriving at a website.
//   k6 run -e RATE=100 -e DURATION=1m load/load.js
import http from 'k6/http';
import { check } from 'k6';

const BASE_URL = __ENV.BASE_URL || 'http://localhost:8082';
const RATE = Number(__ENV.RATE || 50);              // requests (iterations) per second
const DURATION = __ENV.DURATION || '1m';
const LIST_SHARE = Number(__ENV.LIST_SHARE || 0.2); // 20% listings, 80% event details
const CITIES = ['São Paulo', 'Rio de Janeiro', 'Belo Horizonte', 'Porto Alegre', 'Curitiba',
  'Salvador', 'Recife', 'Fortaleza', 'Brasília', 'Manaus'];

export const options = {
  scenarios: {
    open: {
      executor: 'constant-arrival-rate',
      rate: RATE,
      timeUnit: '1s',
      duration: DURATION,
      preAllocatedVUs: Math.max(20, RATE),   // VUs ready to start requests
      maxVUs: Math.max(200, RATE * 10),      // k6 may add more if requests get slow
    },
  },
  summaryTrendStats: ['avg', 'med', 'p(95)', 'p(99)', 'max'],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    'http_req_duration{name:detail}': ['p(99)<500'],
    'http_req_duration{name:list}': ['p(99)<1000'],
  },
};

export default function () {
  if (Math.random() < LIST_SHARE) {
    const city = CITIES[Math.floor(Math.random() * CITIES.length)];
    const page = Math.floor(Math.random() * 5);
    const res = http.get(`${BASE_URL}/catalog/events?city=${encodeURIComponent(city)}&page=${page}`,
      { tags: { name: 'list' } });
    check(res, { 'list is 200': (r) => r.status === 200 });
  } else {
    const id = 1 + Math.floor(Math.random() * 200000);
    const res = http.get(`${BASE_URL}/catalog/events/${id}`, { tags: { name: 'detail' } });
    check(res, { 'detail is 200': (r) => r.status === 200 });
  }
}

The stress test climbs a staircase of rates past the knee, holds the peak and then drops back to a low rate. The question is not only "where does it break?", but "does it recover by itself when the load goes away?".

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

// Lab 2: stress test. Climb a staircase of arrival rates past the knee, then come back
// down: does the service recover by itself when the load drops?
//   k6 run -e PEAK=600 load/stress.js
import http from 'k6/http';
import { check } from 'k6';

const BASE_URL = __ENV.BASE_URL || 'http://localhost:8082';
const PEAK = Number(__ENV.PEAK || 600);             // highest arrival rate (req/s)
const LIST_SHARE = Number(__ENV.LIST_SHARE || 0.2);
const CITIES = ['São Paulo', 'Rio de Janeiro', 'Belo Horizonte', 'Porto Alegre', 'Curitiba',
  'Salvador', 'Recife', 'Fortaleza', 'Brasília', 'Manaus'];

const step = (fraction, duration) => ({ target: Math.round(PEAK * fraction), duration });

export const options = {
  scenarios: {
    staircase: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 2000,
      stages: [
        step(0.25, '30s'), step(0.25, '30s'),   // warm-up and 1/4 of the peak
        step(0.50, '30s'), step(0.50, '30s'),
        step(0.75, '30s'), step(0.75, '30s'),
        step(1.00, '30s'), step(1.00, '1m'),    // peak, held longer
        step(0.25, '15s'), step(0.25, '1m'),    // back down: recovery?
      ],
    },
  },
  summaryTrendStats: ['avg', 'med', 'p(95)', 'p(99)', 'max'],
  thresholds: {
    http_req_failed: ['rate<0.05'],
    // a breakpoint test would abort here: { threshold: 'p(99)<2000', abortOnFail: true }
    http_req_duration: ['p(99)<2000'],
  },
};

export default function () {
  if (Math.random() < LIST_SHARE) {
    const city = CITIES[Math.floor(Math.random() * CITIES.length)];
    const res = http.get(`${BASE_URL}/catalog/events?city=${encodeURIComponent(city)}`,
      { tags: { name: 'list' }, timeout: '10s' });
    check(res, { 'list is 200': (r) => r.status === 200 });
  } else {
    const id = 1 + Math.floor(Math.random() * 200000);
    const res = http.get(`${BASE_URL}/catalog/events/${id}`, { tags: { name: 'detail' }, timeout: '10s' });
    check(res, { 'detail is 200': (r) => r.status === 200 });
  }
}

3. Watch from the inside

k6 sees the service from the outside. To find why latency grows you need gauges from the inside. Spring Boot Actuator already publishes them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

#!/usr/bin/env bash
# Lab 2: print, once per second, how many requests are INSIDE the service right now.
#   bash scripts/watch-metrics.sh http://localhost:8082
URL=${1:-http://localhost:8082}
metric() {   # metric <name>: current value of an actuator metric
  curl -s "$URL/actuator/metrics/$1" | jq -r '.measurements[0].value // "?"'
}
printf '%-8s %13s %10s %11s\n' time busy_threads db_active db_pending
while true; do
  printf '%-8s %13s %10s %11s\n' "$(date +%T)" \
    "$(metric tomcat.threads.busy)" \
    "$(metric hikaricp.connections.active)" \
    "$(metric hikaricp.connections.pending)"
  sleep 1
done

Keep three terminals open during every test: k6, bash scripts/watch-metrics.sh, and docker stats (CPU of catalog and db).

You should see one line per second (at overload, one every few seconds: the request that reads the metrics waits for a free Tomcat thread too). At rest busy_threads is 1 (the request that reads the metric); under load it follows Little's Law, and at overload the pool fills up (lines from a run for this lab, at 50, 400 and 600 req/s):

time      busy_threads  db_active  db_pending
03:43:21           1.0        0.0         0.0
03:45:19          12.0       10.0         3.0
03:46:15         200.0        9.0       190.0

4. Find the knee

Run load.js at increasing rates, one minute each, and save each summary. The sweep script does the bookkeeping (and a warm-up first: a cold JVM would spoil the first point):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

#!/usr/bin/env bash
# Lab 2: run load.js once per arrival rate and save one k6 summary per run.
#   bash scripts/sweep.sh 25 50 100 150 200 300 400
# Then:  uv run scripts/summarize.py results/lab2/rate-*.json
set -euo pipefail
OUT=${OUT:-results/lab2}
DURATION=${DURATION:-1m}
mkdir -p "$OUT"
echo ">>> warm-up: 30 s at RATE=$1 (not recorded)"
k6 run -q -e RATE="$1" -e DURATION=30s load/load.js > /dev/null || true
for rate in "$@"; do
  echo ">>> RATE=$rate req/s for $DURATION"
  # || true: a run that crosses a threshold still produces a summary; keep sweeping
  k6 run -q -e RATE="$rate" -e DURATION="$DURATION" \
     --summary-export "$OUT/rate-$rate.json" load/load.js > /dev/null || true
  sleep 10   # let queues drain before the next step
done

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# /// script
# requires-python = ">=3.12"
# dependencies = []
# ///
"""Turn k6 --summary-export files into one Markdown table (one row per file).

    uv run scripts/summarize.py results/lab2/rate-*.json

The block above is inline script metadata (PEP 723): uv reads it and runs the
script with a suitable Python, no virtual environment to manage.
"""
import json
import re
import sys


def ms(metric, stat):
    value = metric.get(stat) if metric else None
    return f"{value:.1f}" if isinstance(value, (int, float)) else "–"


def row(path):
    m = json.load(open(path))["metrics"]
    offered = re.search(r"rate-(\d+)", path)
    dur = m.get("http_req_duration", {})
    failed = m.get("http_req_failed", {}).get("value", 0)
    dropped = m.get("dropped_iterations", {}).get("count", 0)
    return [
        offered.group(1) if offered else path,
        f"{m['http_reqs']['rate']:.0f}",
        ms(dur, "med"), ms(dur, "p(95)"), ms(dur, "p(99)"),
        ms(m.get("http_req_duration{name:detail}"), "p(99)"),
        ms(m.get("http_req_duration{name:list}"), "p(99)"),
        f"{failed * 100:.1f}%",
        str(dropped),
        str(m.get("vus_max", {}).get("value", "–")),
    ]


def sort_key(path):
    n = re.search(r"rate-(\d+)", path)
    return int(n.group(1)) if n else 0


header = ["offered (req/s)", "throughput (req/s)", "p50 (ms)", "p95 (ms)", "p99 (ms)",
          "p99 detail", "p99 list", "errors", "dropped", "max VUs"]
print("| " + " | ".join(header) + " |")
print("|" + "|".join("---:" for _ in header) + "|")
for path in sorted(sys.argv[1:], key=sort_key):
    print("| " + " | ".join(row(path)) + " |")

docker compose up -d                       # catalog + db from lab 1
bash scripts/sweep.sh 50 100 200 300 400 500 600
uv run scripts/summarize.py results/lab2/rate-*.json

sweep.sh prints one line per step (>>> warm-up: 30 s at RATE=50 (not recorded), then >>> RATE=50 req/s for 1m, …) and takes about 10 minutes for seven rates. Steps past the knee also print level=error msg="thresholds on metrics ... have been crossed": the latency thresholds of load.js failed, which is expected there — the summary is saved anyway. summarize.py then prints a Markdown table like the one below — paste it into results/lab2.md.

An example run on a laptop (catalog limited to 1 CPU, pool of 10 connections):

Past 500 req/s the numbers jump around from run to run — a sign that the service is overloaded and queues decide the outcome, not the code.

How to read it:

• Below the knee, throughput equals the offered rate and p99 is roughly flat: requests rarely wait for each other.
• At the knee, a resource reaches ~100% busy. Small increases in load produce large increases in latency — the queueing curve from class 2 (W ≈ S / (1 − ρ)).
• Past the knee, throughput stops following the offered rate, dropped_iterations appears (k6 ran out of VUs to keep the rate), and p99 is measured in seconds. Users have already left.

5. Little's Law check

Little's Law holds for any stable system²: L = λ × W — the average number of requests inside (L) equals the arrival rate (λ) times the average time each one spends inside (W). Apply it to two boundaries:

1. The whole service: L should match tomcat.threads.busy. At 300 req/s with an average latency of 10 ms, L = 300 × 0.010 = 3 busy threads.
2. The database pool: each request holds a connection for its database time. The pool is a server with 10 slots, so it saturates when λ × (database time per request) = 10.

That second line predicts the knee. Measure the database time per request with one user (lab 1): if the mix averages ~8 ms of database time per request without load, the pool would allow 10 / 0.008 ≈ 1,250 req/s. The knee you measured is much lower — because under load the queries get slower (the database and the single catalog CPU are busy too), so each request holds its connection longer. Look at watch-metrics.sh around the knee: db_active sits at 10 and db_pending starts to grow. The pool is full; requests queue for it.

6. Move the bottleneck

Change one variable, measure again. The pool size is configurable without rebuilding (DB_POOL_SIZE, wired in lab 1):

DB_POOL_SIZE=30 docker compose up -d catalog        # recreate catalog with a bigger pool
OUT=results/lab2-pool30 bash scripts/sweep.sh 50 100 200 300 400 500 600
uv run scripts/summarize.py results/lab2-pool30/rate-*.json

Compare the two tables. Typically the knee moves up, but not forever: with 30 connections, the next resource saturates — the single CPU of catalog, or PostgreSQL itself (watch docker stats). A bigger pool also has a cost: every connection is a process in PostgreSQL, and in lab 6 three services × three replicas × 30 connections would exceed max_connections. Optimizing one resource always moves the bottleneck somewhere else; the job is to know where it went.

7. Stress: past the knee and back

docker compose up -d catalog       # back to the default pool (10): PEAK=600 is past its knee
K6_WEB_DASHBOARD=true K6_WEB_DASHBOARD_EXPORT=results/lab2-stress.html \
  k6 run -e PEAK=600 load/stress.js

Open http://localhost:5665 while it runs: the k6 web dashboard plots request rate, latency percentiles and VUs over time; the HTML report is saved at the end. Look for three moments:

1. The knee on the way up: p95 leaves the floor.
2. The peak: VUs climb towards maxVUs — k6 needs more concurrent users to keep the rate because each request takes longer (Little's Law again, from the client side).
3. The recovery: after the drop to 25% of the peak, how many seconds until p95 is back to normal? A service that recovers by itself is stable; one that stays slow (queues that never drain, a garbage collector that never catches up) needs load shedding — a topic for labs 3 and 9.

Measure it

Record everything in results/lab2.md:

# Lab 2 — catalog under load

Setup: catalog 1 CPU / 768 MB, Hikari pool = 10, PostgreSQL 17, k6 open model (80% detail, 20% list)

## Sweep (pool = 10)
<paste the table printed by summarize.py>

## Sweep (pool = 30)
<paste the table>

## Knee and capacity
- Knee (pool 10): ____ req/s — capacity with p99 < 300 ms: ____ req/s
- Knee (pool 30): ____ req/s — capacity with p99 < 300 ms: ____ req/s
- Bottleneck at the knee (evidence from watch-metrics / docker stats): ____

## Little's Law
| Offered (req/s) | avg latency (s) | λ × W | busy threads (measured) | db_active (measured) |
|---:|---:|---:|---:|---:|
|  100 |  |  |  |  |
|  300 |  |  |  |  |

## Stress test
- Peak reached: ____ req/s; p95 at the peak: ____ ms; dropped iterations: ____
- Recovery time after the drop: ____ s

## Hypothesis check
- My playground hypothesis was: ____ . It was right / wrong because ____ .

Troubleshooting

Clean up

docker compose up -d catalog     # if you changed DB_POOL_SIZE, recreate catalog with the default (10)
docker compose down              # when you are done for the day (data is kept)

Keep results/lab2* (the JSON summaries and your results/lab2.md): lab 3 compares against them. The HTML report is ignored by Git on purpose (it is large).

Deliverable

• load/load.js, load/stress.js, scripts/sweep.sh, scripts/summarize.py and scripts/watch-metrics.sh in your repository.
• Two sweeps (pool 10 and pool 30) with tables in results/lab2.md, the knee and the capacity of each.
• The Little's Law table with at least two rates, compared with the measured gauges.
• The stress test HTML report (results/lab2-stress.html, not committed) and its three moments described.
• One paragraph: what is the bottleneck of catalog, and what is your evidence?
• Repository tagged lab2; quiz answered.

End state

📁 boxoffice/
├──  load/load.js                 # open-model load test (RATE, DURATION, LIST_SHARE)
├──  load/stress.js               # staircase past the knee and back
├──  scripts/sweep.sh             # one load.js run per rate
├──  scripts/summarize.py         # k6 summaries -> Markdown table (uv run)
├──  scripts/watch-metrics.sh     # Tomcat threads and Hikari pool, once per second
└── 📁 results/lab2/  results/lab2-pool30/  results/lab2.md

No service changed: the stack is still db + catalog.

Next: Lab 3 — Edge, where one replica becomes three.