a. Overview

The catalog is the textbook read-heavy service: during the week of the stadium show, a few thousand popular events receive almost all the traffic, and the listing of "upcoming events in São Paulo" is requested thousands of times per minute while it changes a few times per day. Lab 2 showed that the database (through its connection pool) is what limits catalog. A cache¹ answers the repeated questions from memory, so the database only sees the first one.

Caches are also, as every interview guide warns², the most common source of subtle bugs in system design: stale data, cold starts, hot keys and stampedes. In this lab you will see each of them with your own measurements.

Before you start

docker compose version        # Docker Compose v2.x (or newer)
java -version                 # 21 or newer
mvn -v                        # Apache Maven 3.9+
k6 version                    # k6 v1.0+
uv --version                  # uv (runs the Python helper scripts)

Start state — the end of lab 4:

📁 boxoffice/
├──  docker-compose.yaml        # db, catalog-1..3, auth, gateway, edge
├──  infra/nginx/nginx.conf     # edge -> gateway
├── 📁 load/  ... gateway.js login.js
└── 📁 services/auth/  services/catalog/  services/gateway/

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Before you measure: the playground

Try it: the read-heavy-cache scenario puts a cache in front of a SQL database. Run it and note the hit ratio and the database load. Then (1) lower the cache TTL or size and watch the hit ratio fall and p99 rise; (2) restart with a cold cache and watch the first seconds; (3) switch to the spike profile and look for the moment many misses hit the database at once. Write down: "with a hit ratio of 90%, the database receives ___% of the reads".

What you will build

sequenceDiagram
    autonumber
    actor Fan
    participant C as catalog
    participant R as redis
    participant DB as db
    Fan->>C: GET /catalog/events/42
    C->>R: GET catalog:event:42
    R-->>C: (nil) — miss
    C->>DB: SELECT event 42 JOIN venue
    DB-->>C: row
    C->>R: SET catalog:event:42 <json> PX 60000..66000
    C-->>Fan: 200 + X-Cache: MISS
    Fan->>C: GET /catalog/events/42 (again)
    C->>R: GET catalog:event:42
    R-->>C: <json> — hit
    C-->>Fan: 200 + X-Cache: HIT (no database access)

Steps

Measure it: hit ratio, p99 and capacity

The load script mimics real popularity: 80% of the detail requests go to the 2,000 hottest events (1% of the catalog); listings are spread over 10 cities and the first 3 pages. It reads the X-Cache header of every response, so k6 reports the hit ratio and separate latencies for hits and misses.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

// Lab 5: a realistic read mix with POPULAR events. 80% of the detail requests go to the
// 2,000 hottest events (1% of the catalog), like a real box office the week of a big show.
//   k6 run -e RATE=300 load/cache.js
import http from 'k6/http';
import { check } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const BASE_URL = __ENV.BASE_URL || 'http://localhost';
const RATE = Number(__ENV.RATE || 300);
const DURATION = __ENV.DURATION || '1m';
const HOT_SHARE = Number(__ENV.HOT_SHARE || 0.8);
const CITIES = ['São Paulo', 'Rio de Janeiro', 'Belo Horizonte', 'Porto Alegre', 'Curitiba',
  'Salvador', 'Recife', 'Fortaleza', 'Brasília', 'Manaus'];

const cacheHit = new Rate('cache_hit');               // share of responses with X-Cache: HIT
const latencyHit = new Trend('latency_hit', true);    // latency of hits only
const latencyMiss = new Trend('latency_miss', true);  // latency of misses (and bypasses)

export const options = {
  scenarios: {
    open: {
      executor: 'constant-arrival-rate', rate: RATE, timeUnit: '1s', duration: DURATION,
      preAllocatedVUs: Math.max(20, RATE), maxVUs: Math.max(200, RATE * 10),
    },
  },
  summaryTrendStats: ['avg', 'med', 'p(95)', 'p(99)', 'max'],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    'http_req_duration{name:detail}': ['p(99)<500'],
    'http_req_duration{name:list}': ['p(99)<1000'],
    cache_hit: ['rate>=0'],                              // printed in the summary
  },
};

function record(res) {
  const outcome = res.headers['X-Cache'];
  cacheHit.add(outcome === 'HIT');
  (outcome === 'HIT' ? latencyHit : latencyMiss).add(res.timings.duration);
}

export default function () {
  if (Math.random() < 0.2) {
    const city = CITIES[Math.floor(Math.random() * CITIES.length)];
    const page = Math.floor(Math.random() * 3);              // people rarely go past page 3
    const res = http.get(`${BASE_URL}/catalog/events?city=${encodeURIComponent(city)}&page=${page}`,
      { tags: { name: 'list' } });
    check(res, { 'list is 200': (r) => r.status === 200 });
    record(res);
  } else {
    const hot = Math.random() < HOT_SHARE;
    const id = 1 + Math.floor(Math.random() * (hot ? 2000 : 200000));
    const res = http.get(`${BASE_URL}/catalog/events/${id}`, { tags: { name: 'detail' } });
    check(res, { 'detail is 200': (r) => r.status === 200 });
    record(res);
  }
}

Run the same load without and with the cache — the switch is an environment variable, no rebuild needed:

# 1. no cache (every request goes to PostgreSQL)
CACHE_ENABLED=false docker compose up -d --wait     # --wait: until the recreated replicas are healthy
k6 run -q -e RATE=300 -e DURATION=30s load/cache.js > /dev/null   # warm-up: new replicas are cold JVMs
k6 run -e RATE=300 load/cache.js

# 2. cache on, cold: warm up the new replicas, then flush Redis right before the test
CACHE_ENABLED=true docker compose up -d --wait
k6 run -q -e RATE=300 -e DURATION=30s load/cache.js > /dev/null
docker compose exec redis redis-cli FLUSHALL
k6 run -e RATE=300 load/cache.js

# 3. cache on, warm: run it again immediately
k6 run -e RATE=300 load/cache.js

# 4. capacity: repeat the lab 2 sweep, now with the cache, through the edge
BASE_URL=http://localhost OUT=results/lab5-cache bash scripts/sweep.sh 300 600 900 1200 1500

Also look at the cache from Redis's side:

docker compose exec redis redis-cli INFO stats | grep -E "keyspace_(hits|misses)"
docker compose exec redis redis-cli INFO memory | grep used_memory_human
docker compose exec redis redis-cli --scan --pattern 'catalog:event:*' | wc -l
docker compose exec redis redis-cli TTL catalog:event:1

Record in results/lab5.md:

# Lab 5 — caching

## Same load (300 req/s, 80% of details on 1% of events)
| Run | cache_hit | p50 (ms) | p99 (ms) | p99 detail | p99 list | latency_hit p99 | latency_miss p99 |
|---|---:|---:|---:|---:|---:|---:|---:|
| no cache |  |  |  |  |  |  |  |
| cold cache |  |  |  |  |  |  |  |
| warm cache |  |  |  |  |  |  |  |

## Capacity (sweep through the edge, 3 replicas)
| Setup | Capacity with p99 < 300 ms (req/s) |
|---|---:|
| lab 3, no cache | |
| lab 5, cache | |

## Redis
- keyspace_hits / keyspace_misses: ____ / ____ → hit ratio ____ %
- keys cached: ____ ; memory used: ____

## Stampede (see step 2)
| CACHE_STAMPEDE_PROTECTION | cache_misses in 1 min | p99 (ms) | max (ms) |
|---|---:|---:|---:|
| false | | | |
| true  | | | |

## Questions
1. With hit ratio h, the database receives (1 − h) of the reads. Which hit ratio did you need to double the capacity?
2. Change HOT_SHARE to 0.2 (popularity almost uniform). What happens to the hit ratio, and why?
3. An admin changes the price of event 42. For how long can a fan see the old price on the listing page? On the event page?

Troubleshooting

Clean up

CACHE_TTL_LIST=10s CACHE_STAMPEDE_PROTECTION=false docker compose up -d   # back to the defaults
docker compose exec redis redis-cli FLUSHALL                               # optional: empty the cache
docker compose down

Keep everything: lab 6 adds inventory, a read replica and PgBouncer to this stack.

Deliverable

• Redis in docker-compose.yaml; EventCache and the new EventController in catalog.
• X-Cache visible: curl -sI localhost/catalog/events/1 twice shows MISS then HIT.
• An update through the gateway (admin token) evicts the event: the next read is a MISS with the new data.
• Stopping Redis (docker compose stop redis) does not break the catalog: requests answer with X-Cache: MISS.
• results/lab5.md complete: three runs, capacity, Redis stats, stampede table and the three answers.
• Repository tagged lab5; quiz answered.

End state

📁 boxoffice/
├──  docker-compose.yaml        # replaced: + redis (6379), catalog gets REDIS_* and CACHE_* variables
├──  load/cache.js  load/stampede.js                       # new
├──  services/catalog/pom.xml                              # replaced: + spring-boot-starter-data-redis
├──  services/catalog/src/main/resources/application.yaml  # replaced: Redis + cache settings
├──  services/catalog/src/main/java/boxoffice/catalog/EventCache.java        # new
├──  services/catalog/src/main/java/boxoffice/catalog/EventController.java   # replaced
└──  results/lab5.md

Next: Lab 6 — Data at scale.