Failure Modes

A cache changes how the system fails. Without it, the database degrades smoothly as load grows. With it, the database is sized for the misses — and anything that suddenly turns hits into misses (a restart, an expiry, a new kind of request, a cache node dying) sends the full load to a database that was never sized for it. Most cache incidents are not bugs in the cache; they are the moment the system discovers how much it depended on the cache.

This page goes through the classic failure modes in the order they tend to appear in a flash sale, each with its symptom and its fix.

Cold cache

A cold cache is an empty one: after a restart of Redis without persistence, a deploy that changes the key version (:v3 → :v4), a new region, or a cache node replaced by autoscaling. The hit ratio starts at zero and climbs as keys are read.

The danger is the gap between the load the database was sized for (the misses) and the load it receives (everything). With 50,000 reads/s and a usual 98% hit ratio, the database is sized for ~1,000 QPS; a cold start sends it 50,000.

Fixes:

• Warm-up: before a known peak, preload the keys you know will be hot — the events going on sale today, the home page lists — with a job that reads them through the normal path.
• Gradual traffic shift: send 5%, then 25%, then 100% of traffic to a new cache node or region (canary), so it warms under a fraction of the load.
• Persistence for big caches: a Redis restart that reloads an RDB snapshot comes back warm (Redis — persistence).
• Never restart all cache nodes at once.
• Capacity floor: know how many QPS the database can take with an empty cache, and protect it with load shedding (see cache down). The Google SRE book lists cold caches among the classic triggers of cascading failures¹.

Cache stampede

A cache stampede (also thundering herd or dog-piling) happens when a hot key expires or is deleted. Every request that arrives before someone puts the value back misses — and each one goes to the database to recompute the same value.

sequenceDiagram
    participant A as 4,000 requests
    participant R as Redis
    participant D as PostgreSQL
    Note over R: event:42 expires at 10:00:00.000
    A->>R: GET event:42 (×4,000 within 200 ms)
    R-->>A: miss (×4,000)
    A->>D: same SELECT (×4,000)
    Note over D: CPU 100%, query time 3 ms to 2 s
    D-->>A: rows, slowly
    A->>R: SET event:42 (×4,000, all identical)

The arithmetic is Little's law. If event 42 is read 20,000 times per second and recomputing it takes 200 ms, then about \(20{,}000 \times 0.2 = 4{,}000\) requests miss before the first one finishes. Worse: under that load the query slows down, the window grows, and more requests pile in. A stampede can take down a database that was serving the same page comfortably one second before.

There are four standard fixes. They are complementary.

1. Lock (or lease): only one recomputes

On a miss, a request tries to acquire a short lock for the key. The winner recomputes; the others wait a few milliseconds and read the cache again (or serve a stale copy).

import json
import time
import uuid

RELEASE = """
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0
"""


def get_event(event_id: int, db, r) -> dict:
    key = f"boxoffice:catalog:event:{event_id}:v3"
    lock = f"lock:{key}"
    for _ in range(50):                                   # wait at most ~1 s
        cached = r.get(key)
        if cached is not None:
            return json.loads(cached)
        token = str(uuid.uuid4())
        if r.set(lock, token, nx=True, px=5_000):          # only one winner
            try:
                event = db.fetch_event_with_venue_and_prices(event_id)
                r.set(key, json.dumps(event), ex=300)
                return event
            finally:
                r.eval(RELEASE, 1, lock, token)             # release only our own lock
        time.sleep(0.02)                                    # losers wait and re-read
    return db.fetch_event_with_venue_and_prices(event_id)   # last resort

• SET ... NX PX 5000 is atomic: set only if absent, with a 5 s expiry so a crashed winner does not block the key forever.
• The release checks the token, so a slow winner whose lock already expired does not delete someone else's lock.
• Cost: losers wait (latency) and poll the cache. Facebook's memcache leases are a refined version of this, built into the cache server².

    if (stampedeProtection) {
        var lockKey = key + ":lock";
        var token = UUID.randomUUID().toString();
        if (tryLock(lockKey, token)) {
            try {
                cached = read(key);               // someone may have filled it meanwhile
                if (cached != null) {
                    return count(new Result(cached, "HIT"));
                }
                return count(load(key, ttl, loader));
            } finally {
                unlock(lockKey, token);
            }
        }
        // another request is already loading this key: wait for it instead of
        // sending one more identical query to the database
        for (int i = 0; i < 20; i++) {
            sleep(Duration.ofMillis(25));
            cached = read(key);
            if (cached != null) {
                return count(new Result(cached, "HIT"));
            }
        }
    }
private boolean tryLock(String lockKey, String token) {
    try {
        // SET lockKey token NX PX 2000: only one request in the whole cluster gets it
        return Boolean.TRUE.equals(redis.opsForValue().setIfAbsent(lockKey, token, Duration.ofSeconds(2)));
    } catch (DataAccessException e) {
        return true;                              // no Redis, no lock: just load
    }
}

private void unlock(String lockKey, String token) {
    try {
        // delete only our own lock (it may have expired and been taken by someone else);
        // get + delete is not atomic: production code uses a Lua script for this
        if (token.equals(redis.opsForValue().get(lockKey))) {
            redis.delete(lockKey);
        }
    } catch (DataAccessException e) {
        log.debug("unlock failed: {}", e.getMessage());
    }
}

2. Request coalescing (single-flight)

Inside one process, concurrent misses for the same key can share one in-flight computation: the first caller starts it, the others wait for the same future. Caffeine's LoadingCache and AsyncLoadingCache do this automatically; Go has singleflight; in Java you can use a ConcurrentHashMap<Key, CompletableFuture<V>>.

Coalescing does not need Redis: with 20 replicas, a stampede of 4,000 requests becomes 20 database queries — usually harmless. It is the cheapest fix and should be the default for hot paths.

3. Stale-while-revalidate

Keep two expirations: a soft TTL stored inside the value (e.g. 60 s) and a longer hard TTL on the key (e.g. 10 min). After the soft TTL, the first reader triggers a background refresh (guarded by a lock) and everyone, including that reader, keeps getting the stale value until the new one arrives. Readers never wait; the database sees one query. The same idea exists in HTTP as Cache-Control: max-age=60, stale-while-revalidate=600.

4. Probabilistic early expiration (XFetch)

Instead of a lock, let each reader decide at random to refresh the value a little before it expires, with a probability that grows as expiry approaches. Vattani, Chierichetti and Lowenstein showed that the following rule is optimal in a precise sense³. Store with the value the time \(\delta\) it took to compute it and its expiry time; on each read, recompute if

Since \(-\ln u\) is an exponential random variable with mean 1, each read "looks ahead" by a random amount of about \(\delta \beta\). The parameter \(\beta\) (default 1) trades earlier refreshes for fewer stampedes. The probability that a single read refreshes, with \(r\) seconds left before expiry, is \(e^{-r/(\delta \beta)}\).

The right panel is the elegant part. If reads arrive at rate \(\lambda\), the probability that nobody refreshed early before expiry is \(e^{-\lambda \delta \beta}\). A hot key (hundreds of reads per second) is almost surely refreshed just before it expires — by one reader, with no lock and no coordination — while a cold key, where a stampede cannot happen anyway, simply expires. The algorithm adapts to the traffic by itself.

import json
import math
import random
import time


def xfetch(r, key: str, recompute, ttl: int = 300, beta: float = 1.0):
    entry = r.hgetall(key)                     # {"data": ..., "delta": ..., "expiry": ...}
    now = time.time()
    if entry:
        delta, expiry = float(entry["delta"]), float(entry["expiry"])
        u = 1.0 - random.random()              # in (0, 1], so log(u) is defined
        if now - delta * beta * math.log(u) < expiry:
            return json.loads(entry["data"])   # serve the cached value
    start = time.time()
    value = recompute()                        # miss, or chosen to refresh early
    delta = time.time() - start
    r.hset(key, mapping={"data": json.dumps(value), "delta": delta, "expiry": start + ttl})
    r.expire(key, ttl)
    return value

Cache avalanche

A cache avalanche is a stampede on many keys at once: a large set of keys expires at the same moment, or the cache is flushed, and the misses for all of them hit the database together.

The fix is jitter: add a random component to every TTL, e.g. ttl = 600 + random(0, 120), so expirations spread over two minutes. Combine with refresh-ahead or XFetch for the very hottest keys, and with stampede protection for each individual key.

/** TTL plus up to 10% random extra, so that keys written together do not expire together. */
private static Duration withJitter(Duration ttl) {
    long extra = (long) (ttl.toMillis() * 0.10 * ThreadLocalRandom.current().nextDouble());
    return ttl.plusMillis(extra);
}

Cache penetration

Cache penetration is traffic for keys that do not exist in the database. They are never cached (there is nothing to cache), so every request goes through to the database. A bot enumerating GET /events/1, /events/2, …, /events/99999999, or a buggy client asking for an id that was deleted, can generate a database query per request at the rate of the cache.

Fixes, in layers:

1. Validate input before touching anything: malformed or out-of-range ids are rejected at the gateway.
2. Negative caching: cache "not found" with a short TTL (e.g. 30–60 s) — see patterns.
3. Bloom filter of existing ids: a compact probabilistic set that answers "definitely not present" or "probably present". A request for an id the filter says is absent never reaches the cache or the database. Sizing: about \(-\ln p / (\ln 2)^2 \approx 9.6\) bits per element for a 1% false-positive rate — ~120 KB for 100,000 events⁴. Redis Stack / Redis 8 offer Bloom filters natively (BF.ADD, BF.EXISTS).
4. Rate limiting and bot mitigation at the edge (WAF), since penetration is often abuse (Traffic & Edge).

Hot keys

A distributed cache spreads keys over nodes, but one key lives on one node. When a single key is extremely popular, that node saturates while the rest of the cluster is idle.

Fixes:

• L1 in-process cache with a short TTL (1–2 s) in each catalog replica (two-level cache). With 50 replicas and a 1 s TTL, Redis sees ~50 reads/s for that key instead of 400,000. This is by far the most effective fix.
• Key replication: store N copies under different keys (event:42#0 … event:42#7), which hash to different nodes; readers pick one at random. Writes and invalidations must touch all N copies.
• Read replicas of the cache shard (Redis replicas with READONLY reads in Cluster), accepting slightly staler reads.
• Shrink the value: send the static part (description, images) through the CDN and cache only the small dynamic part.
• Cache higher up: the public event page can be cached at the CDN or gateway for a few seconds, so the hot key never reaches the application.

Detecting hot keys: client-side metrics per key prefix, redis-cli --hotkeys (requires an LFU eviction policy), or the MONITOR command on a replica for a few seconds (never on a busy primary).

Big keys

A big key is a value of megabytes, or a collection (hash, list, set, sorted set) with millions of members. Redis executes commands on a single main thread, so reading, writing, expiring or deleting a big key blocks every other client for its duration; transferring it saturates the network.

• Find them with redis-cli --bigkeys or MEMORY USAGE key.
• Split large objects (one key per section instead of one per venue), paginate collections (HSCAN, ZRANGE ... LIMIT), compress values above a few KB.
• Delete with UNLINK (frees memory in a background thread) instead of DEL.
• Keep values small — a common guideline is to stay well below 100 KB per value and avoid collections with more than ~10,000 elements; treat these as rules of thumb, not limits.

The cache goes down

Sooner or later, the Redis primary fails, the network partitions, or a failover takes 10–30 seconds. What happens next depends on decisions made long before:

flowchart TD
    F["Redis unavailable"] --> T{"Cache calls have<br/>a short timeout?"}
    T -->|"no"| X1["threads block on Redis<br/>whole service stalls"]
    T -->|"yes"| B{"Fallback on error?"}
    B -->|"no"| X2["every read fails<br/>outage"]
    B -->|"read the DB"| C{"Can the DB take<br/>the full load?"}
    C -->|"yes"| OK1["slower but correct"]
    C -->|"no"| S["shed load and serve degraded:<br/>static page, waiting room,<br/>rate-limited misses"]

The uncomfortable truth: once the database is sized for the misses, the cache is no longer optional. It is a load-bearing dependency, and it needs the same reliability engineering as the database:

• Timeouts on every cache call (tens of milliseconds, not the client default) and a circuit breaker, so a sick cache fails fast instead of holding threads (Reliability).
• Replication with automatic failover (Redis Sentinel or Cluster; managed services do this for you) — see Redis.
• Protected fallback: falling back to the database for every miss during an outage is a self-inflicted stampede. Limit concurrent fallback queries (a bulkhead), serve degraded content for non-essential parts, and put the waiting room in front of the on-sale.
• Gutter pool: Facebook keeps a small pool of idle cache servers; when a server fails, clients send its keys to the gutter pool (with short TTLs) instead of to the database².
• Know the number: test what the database can take with the cache off. That number is your real capacity floor.

Memory pressure, evictions and stale failovers

• Eviction storms: when the working set outgrows maxmemory, the cache evicts keys that will be needed again; the hit ratio falls, the database load rises. Watch evicted_keys and the hit ratio together.
• Fragmentation: memory used by the process can exceed the data size (mem_fragmentation_ratio in INFO memory); leave headroom.
• Snapshot fork: an RDB snapshot or AOF rewrite forks the process; with copy-on-write, a write-heavy instance can need up to twice its memory during the snapshot. Size for it, or run persistence on a replica.
• Stale data after failover: Redis replication is asynchronous. If the primary acknowledged a DEL and failed before replicating it, the promoted replica still has the old value — invalidations can be lost. One more reason every key needs a TTL.

See it in the playground

Try it: this scenario runs the read-heavy-cache design under a spike. First watch the steady state: hit ratio, database QPS and p99. Then (1) flush the cache at the peak and watch the database load jump to the full request rate — a cold cache under load; (2) shorten the TTL of the hot keys and look for stampede spikes in the database QPS each time they expire; (3) enable chaos and watch what happens to p99 while the cache is unavailable.

The full playground lets you add a second cache layer or a waiting room and compare.

How it connects

flowchart LR
    COLD["cold cache<br/>warm before traffic"] --> DB["the database takes the misses"]
    STAMP["stampede: a hot key expires<br/>reads/s × recompute time"] --> FIX["lock, coalescing,<br/>stale-while-revalidate, XFetch"]
    AVA["avalanche: many keys<br/>expire together"] --> JIT["TTL jitter"]
    PEN["penetration: keys<br/>that never exist"] --> NEG["negative caching,<br/>Bloom filter"]
    HOT["hot key: one node"] --> L1["L1, key copies, CDN<br/>→ Patterns, Edge"]
    DOWN["the cache goes down"] --> RES["timeouts, circuit breaker,<br/>bulkhead → Reliability"]
    DOWN --> DB
    DB --> SHED["shed load, waiting room<br/>→ Messaging"]

Palette components: cache (a load-bearing dependency once the database is sized for the misses) · sql (what it can take with the cache off is the real capacity floor) · app (single-flight, an L1, short cache timeouts) · cdn · apigw (cache the hot public page higher up) · queue (the waiting room when the fallback would drown the database) · metrics (hit ratio, evicted_keys). Scenarios: read-heavy-cache, flash-sale, cascading-failure. Continues in: Redis — replication, Reliability — circuit breaker, Estimation — queueing (Little's law), Case study: distributed cache, Hands-on 5 (reproduce and fix a stampede).