Overview

It is 09:59 on the day a stadium show goes on sale. BoxOffice has 60,000 seats to sell and about 2 million fans with the event page open, pressing F5. At 10:00 the catalog service receives tens of thousands of GET /events/42 requests per second, and each one runs the same three SQL queries: the event, the venue, the price table. The data has not changed in a week. Still, the catalog database reaches 100% CPU, queries that took 3 ms now take 800 ms, and the checkout — which shares that database — starts timing out. Nobody bought a ticket yet, and the system is already down.

The fix is not a bigger database. It is to stop asking the database the same question two million times. That is caching: keeping a copy of the answer to an expensive question in a place that is faster and cheaper to read, and serving the copy for as long as it is good enough.

Roadmap

1. This overview — why caching works, where caches live, hit-ratio math, sizing and what never to cache.
2. Caching Patterns — cache-aside, read-through, write-through, write-behind, write-around, refresh-ahead; key design.
3. Invalidation & Eviction — TTLs, explicit invalidation, the cache-aside race, CDC, LRU/LFU/W-TinyLFU.
4. Failure Modes — cold cache, stampede, avalanche, penetration, hot keys, the cache going down.
5. Redis — data structures, atomicity, persistence, replication, Redis Cluster, managed offerings.
6. Quiz — check your understanding.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. Open the slides in a new tab

Why caching works

A cache is a bet: the data I read now will be read again soon. Two properties of real workloads make that bet pay off.

• Temporal locality — something read recently is likely to be read again. A fan who opened the event page will refresh it a dozen times before 10:00.
• Skewed popularity — a small fraction of the items receives most of the requests. BoxOffice lists 100,000 events, but on a Friday night a few hundred of them (the big shows, this weekend's games) take almost all the traffic.

Popularity on the web follows, approximately, a Zipf-like distribution: the item of rank \(i\) is requested with probability proportional to \(1/i^{s}\), where \(s\) (the skew) is typically between 0.6 and 1 for web objects¹. The higher \(s\), the more concentrated the traffic, and the more a small cache helps.

The second ingredient is the latency gap between storage layers. The numbers below are orders of magnitude — they vary with hardware and network, but the ratios between them are what matter²:

A cache does not only make a read faster. It also removes work from the system behind it: every hit is a query that the database never sees. In most real systems that second effect — protecting the source of truth — is the reason the cache exists.

Where caches live

A single BoxOffice request can be answered by six different caches before it reaches the database. Each layer is closer to the user (faster, cheaper per hit) but holds less, and is harder to invalidate.

flowchart LR
    U["Browser / app<br/>HTTP cache"] -->|"miss"| CDN["CDN edge<br/>static + public pages"]
    CDN -->|"miss"| GW["API gateway<br/>response cache"]
    GW -->|"miss"| APP["catalog service<br/>in-process cache"]
    APP -->|"miss"| R[("Redis<br/>distributed cache")]
    R -->|"miss"| DB[("PostgreSQL<br/>buffer pool + OS page cache")]
    DB -->|"row"| APP
    APP -->|"SET with TTL"| R

The deeper the layer, the more consistent it is: the database buffer pool can never return stale data, while a copy in the browser cannot be recalled at all. Choosing where to cache is therefore a choice about how much staleness each kind of data tolerates — the subject of Invalidation & Eviction. CDNs and HTTP caching are covered in depth in Traffic & Edge; this class focuses on the application side: in-process and distributed caches.

Hit-ratio math

The hit ratio \(h\) is the fraction of reads answered by the cache. Two formulas carry almost every caching decision.

Effective latency — the average time of a read:

where \(t_{\text{miss}}\) includes the failed cache lookup plus the database query (and the write-back to the cache).

Load on the database — the reads that pass through:

The plot below shows both effects. On the left, the hit ratio of an ideal cache that always keeps the \(k\) most popular items, for a catalog of 100,000 items and several skews (a real cache, with an eviction policy that must guess, does somewhat worse). On the right, the database load as a function of the hit ratio for 50,000 reads per second.

Two lessons from the left panel:

• With a strong skew (\(s \geq 1\)), a cache holding 1% of the catalog (dotted line) already answers 60–85% of the reads. Doubling the cache from there buys only a few points: the returns diminish quickly.
• With a weak skew (\(s = 0.6\), many items with similar popularity — e.g. a long tail of user profiles), a small cache barely helps. Before adding a cache, ask whether the access pattern is skewed at all.

Sizing a cache

The cache must hold the working set — the items that are read again while they are still in the cache — not the whole dataset.

Rules of thumb for sizing:

• Measure the item size, do not guess: serialize a real object and check its length (MEMORY USAGE key in Redis).
• Monitor evictions: if evicted_keys grows steadily while the hit ratio falls, the cache is too small for the working set.
• Separate by purpose: sessions (must not be evicted) and query caches (can be evicted) should not compete for the same memory — use different instances or eviction policies (Redis).

What not to cache

A cache answers "what was the value a moment ago?". Some questions need "what is the value now?".

The same logic applies to account balances, payment status, idempotency keys, and anything where acting on an old value causes a wrong, irreversible action. A good rule: cache what you display, not what you decide on.

Other things that should not go in a shared cache: data that is read once (a cache full of one-hit items only evicts useful ones), personal data without a clear TTL and access control (a cache is a copy that also needs to be protected and erased), and values that are cheaper to compute than to fetch over the network.

See it in the playground

Try it: the scenario below puts a cache-aside Redis in front of the catalog database. Start the run with the default settings and watch the hit ratio, the database QPS and the p99 latency. Then (1) set the cache TTL to a very short value and watch the hit ratio and the database load move together; (2) flush the cache in the middle of the run (cold cache) and see how long p99 takes to recover; (3) raise the load until the database saturates on misses alone.

Open the full playground to add an in-process cache or a CDN in front of the same design. The cache component is described in the component catalog.

The caching concept map

A cache is not one component but a layer at every hop: each palette component on the read path can answer from a copy before the request reaches the source of truth. The map links each palette component (green) to the caching concepts it uses (blue):

flowchart LR
    client["client · mobile"]:::comp -->|"miss"| cdn["cdn"]:::comp
    cdn -->|"miss"| apigw["apigw"]:::comp
    apigw -->|"miss"| app["app"]:::comp
    app -->|"miss"| cache["cache"]:::comp
    cache -->|"miss"| sql["sql"]:::comp
    sql -->|"CDC"| stream["stream · pubsub"]:::comp
    stream -->|"invalidate"| cache
    client -.- c1["Cache-Control, ETag,<br/>versioned URLs"]:::tech
    cdn -.- c2["purge by tag,<br/>short edge TTL"]:::tech
    app -.- c3["in-process L1,<br/>two-level cache"]:::tech
    cache -.- c4["cache-aside, TTL, eviction,<br/>hit ratio, sizing"]:::tech
    cache -.- c5["stampede, hot keys,<br/>the cache goes down"]:::tech
    sql -.- c6["buffer pool,<br/>never cache decisions"]:::tech
    classDef comp fill:#1e9e5a22,stroke:#1e9e5a
    classDef tech fill:#2f6fed22,stroke:#2f6fed

Connections

• Databases — the source of truth the cache protects; read replicas are the other way to scale reads.
• Traffic & Edge — DNS and CDN — HTTP caching and CDNs, the outermost cache layers.
• Hands-on 5 — Caching — Redis cache-aside on the catalog service; measure the hit ratio and p99; reproduce and fix a stampede.
• Exercise — cache + replica — before/after measurements of a cache and a read replica.
• Case studies: distributed cache, URL shortener, search autocomplete.
• Playground — scenario read-heavy-cache.

Key takeaways

What breaks next

The read path is now cheap: at a 95% hit ratio the database sees one read in twenty, and lab 5 measures it. The write path did not change. POST /orders still holds the seat, calls the payment provider and sends the confirmation e-mail inside the same request, so the fan waits for the slowest of them, and an e-mail outage becomes an ordering outage. And no cache helps when 2 million fans want to buy, not read: the database takes about 1,100 holds/s and 33,000 arrive. Taking slow work out of the request, and putting the crowd in a fair line, is class 8 — Messaging.