c. Stampede

A cache stampede (also thundering herd or dog-piling) happens when a hot key expires: every request that arrives before the value is rebuilt misses, and all of them run the same expensive query at the same time. The database, which the cache was protecting, receives a burst of identical work exactly when traffic is highest.

sequenceDiagram
    autonumber
    participant F1 as Fan 1
    participant F2 as Fan 2..200
    participant C as catalog ×3
    participant R as redis
    participant DB as db
    Note over R: key "São Paulo, page 0" expires
    F1->>C: GET listing
    C->>R: GET → miss
    C->>DB: heavy query (join, sort, count)
    F2->>C: GET listing (same key, same moment)
    C->>R: GET → miss
    C->>DB: the SAME heavy query, ×199
    Note over DB: 200 identical queries, pool of 30 connections: queueing
    DB-->>C: rows (eventually)
    C->>R: SET (200 times, same value)

1. Reproduce it

Make the problem frequent: a hot listing with a 3 s TTL, hammered by 200 closed-model users without think time.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

// Lab 5: cache stampede. 200 fans refresh the SAME expensive page non-stop.
// Every time its key expires, how many requests go to the database at once?
//   CACHE_TTL_LIST=3s docker compose up -d      (short TTL: many expirations per minute)
//   k6 run load/stampede.js
import http from 'k6/http';
import { check } from 'k6';
import { Counter } from 'k6/metrics';

const BASE_URL = __ENV.BASE_URL || 'http://localhost';
const misses = new Counter('cache_misses');          // each MISS = one heavy query in the DB

export const options = {
  vus: Number(__ENV.VUS || 200),
  duration: __ENV.DURATION || '1m',
  summaryTrendStats: ['avg', 'med', 'p(95)', 'p(99)', 'max'],
  thresholds: {
    cache_misses: ['count>=0'],                       // printed in the summary
    http_req_failed: ['rate<0.01'],
  },
};

const HOME = `${BASE_URL}/catalog/events?city=${encodeURIComponent('São Paulo')}&page=0`;

export default function () {
  const res = http.get(HOME, { tags: { name: 'home' } });
  check(res, { 'home is 200': (r) => r.status === 200 });
  if (res.headers['X-Cache'] !== 'HIT') {
    misses.add(1);
  }
}

CACHE_TTL_LIST=3s CACHE_STAMPEDE_PROTECTION=false docker compose up -d --wait
k6 run -q -e DURATION=20s load/stampede.js > /dev/null    # warm-up: the replicas were recreated
k6 run load/stampede.js

In 60 s the key expires about 20 times. Without protection, cache_misses is far above 20 — each expiry lets through every request that arrived during the reload, one heavy query each. Watch docker stats during the test: the CPU of db spikes every ~3 seconds, and so does the latency (look at max and p(99) versus med).

2. Fix it: one rebuild per key

The fix implemented in EventCache (already in your code, behind a switch) is a distributed lock per key:

1. Miss → try SET key:lock <token> NX PX 2000. Redis guarantees only one request in the whole cluster (all three replicas) gets it.
2. The winner double-checks the cache, runs the query, writes the value, releases the lock.
3. The losers do not query the database: they poll the cache every 25 ms for up to 500 ms and return the value as soon as the winner writes it (a HIT).
4. If the winner dies, the lock expires after 2 s, so nobody waits forever.

if (stampedeProtection) {
    var lockKey = key + ":lock";
    var token = UUID.randomUUID().toString();
    if (tryLock(lockKey, token)) {
        try {
            cached = read(key);               // someone may have filled it meanwhile
            if (cached != null) {
                return count(new Result(cached, "HIT"));
            }
            return count(load(key, ttl, loader));
        } finally {
            unlock(lockKey, token);
        }
    }
    // another request is already loading this key: wait for it instead of
    // sending one more identical query to the database
    for (int i = 0; i < 20; i++) {
        sleep(Duration.ofMillis(25));
        cached = read(key);
        if (cached != null) {
            return count(new Result(cached, "HIT"));
        }
    }
}

CACHE_TTL_LIST=3s CACHE_STAMPEDE_PROTECTION=true docker compose up -d --wait
k6 run -q -e DURATION=20s load/stampede.js > /dev/null    # warm-up: the replicas were recreated
k6 run load/stampede.js

Now cache_misses should be close to the number of expirations (~20 per minute). In a test run of this script as it is (200 VUs, 60 s, TTL 3 s), the misses dropped from 2,238 to 19 — one database query per expiration instead of about a hundred and ten. In the same run p99 fell from about 1 s to about 0.3 s and the average CPU of db from 234% to 4%. The latency gain varies from machine to machine (Redis, the edge and the gateway also weigh on each request), but the database load during each expiration drops by about two orders of magnitude, and that is what protects it during a real spike. Fill in the stampede table of results/lab5.md.

3. Other cures, and their trade-offs

Back to the lab overview to record the results.