Scenario guide

Every scenario of the System Design Playground is a small lab: a realistic situation, an initial design that fails its goals for a reason, and one possible fix. Each section below says what the scenario teaches, what to try, what to watch, the goals, and questions to discuss in class. The embedded simulators are compact (run controls, metrics and parameters); use Open in the playground (↗) to edit the structure.

How to work on a scenario

1. Predict before you run. Estimate utilization (ρ = λ / μ) and the queue behaviour from the numbers in the brief.
2. Run the initial design and read the Report: which goal fails, and what is the bottleneck?
3. Change one thing at a time and run again with the same seed — differences are then caused by your change, not by luck.
4. Change the seed at the end: a good design passes with any seed.
5. Explain the numbers with the formulas of the manual.

All scenarios

Monolith: find the breaking point

Version 1 of the app: one app server and one PostgreSQL. The breakpoint profile ramps the load until errors pass 5% or p99 explodes. Where does it break, and which component breaks first? Make it survive 2 000 req/s without spending more than 2 000 USD/month.

• Lesson: Every system has a breaking point, and the first component to saturate decides it. Scaling one tier moves the bottleneck to the next one.
• Setup: monolith · profile breakpoint · target 600 req/s · 90% reads · seed 7 · chaos off
• Goals: breaking point ≥ 2,000 req/s · cost/month < US$ 2,000
• Try: Press ▶ and read the Report: breaking point and bottleneck. Add a Load Balancer in front of the app and raise its replicas; run again.
• Observe: The knee (p99 doubles) comes well before the break. After scaling the app, the database becomes the bottleneck.
• Related: What is system design · Estimation

Scale out behind a load balancer

The catalog gets 1 500 req/s. Two replicas are not enough, and at t = 80 s one replica crashes. The load balancer keeps sending it traffic until it notices (30 s without health checks). Meet the goals with headroom for one failure (N+1).

• Lesson: Horizontal scaling needs headroom for failures (N+1), and a load balancer only routes around a dead replica after it detects it.
• Setup: scale-out · profile load · target 1,500 req/s · 90% reads · seed 11 · chaos off · scripted events: t = 80 s — one catalog replica crashes
• Goals: error rate < 1% · p99 < 250 ms · cost/month < US$ 1,800
• Try: Raise the app replicas until ρ ≈ 0.6. Then add a Health Check connected to the Load Balancer.
• Observe: At t = 80 s one replica dies: errors last ~30 s without health checks and ~3 s with them. The survivors must absorb its share.
• Related: Traffic & edge · Reliability

Read-heavy catalog: add a cache

Users browse the catalog: 3 000 req/s, 95% reads. The app tier is fine; the database is not. At t = 100 s the cache (once you have one) is flushed: will the database survive the stampede of misses?

• Lesson: A cache in front of the database cuts its read load to reads × (1 − hit ratio); a cold cache sends everything back to the database at once (stampede).
• Setup: read-heavy-cache · profile load · target 3,000 req/s · 95% reads · seed 3 · chaos off · scripted events: t = 100 s — cache flushed (deploy / restart)
• Goals: p99 < 100 ms · error rate < 0.5% · cost/month < US$ 3,000
• Try: Insert a Cache between app and database. Run; then turn request coalescing on and compare what happens at t = 100 s.
• Observe: The database utilization drops with the hit ratio. At the flush, without coalescing, it saturates for several seconds.
• Related: Caching

Offload static content to a CDN

Product pages are full of images, styles and scripts: 3 000 req/s, 98% reads, served by the app from object storage. Users far from the region wait, and the app is saturated. Serve them from the edge.

• Lesson: Static and cacheable content belongs at the edge: the origin only sees misses and writes, and users get a nearby answer.
• Setup: cdn-static · profile load · target 3,000 req/s · 98% reads · seed 5 · chaos off
• Goals: p99 < 150 ms · error rate < 0.5%
• Try: Put a CDN after the DNS (dns → cdn → lb). Lower the app replicas and compare latency and cost.
• Observe: The app tier falls from saturated to ~35% with fewer replicas. The CDN egress becomes the largest cost line.
• Related: Traffic & edge

Read replicas and replication lag

4 000 req/s with 10% writes overload the single PostgreSQL primary. Read replicas take reads but never writes, and they lag behind the primary. Keep the lag under 1 s.

• Lesson: Read replicas multiply read capacity but not write capacity, and they serve slightly old data (replication lag).
• Setup: db-replicas · profile load · target 4,000 req/s · 90% reads · seed 13 · chaos off
• Goals: p99 < 150 ms · error rate < 0.5% · replication lag < 1 s
• Try: Add 2 read replicas to the database. Then raise the write share in the toolbar to 30% and watch the primary.
• Observe: Reads spread over 1 + replicas; the primary keeps every write and its lag grows with the write load.
• Related: Databases

Partitions and the hot key

Stock lookups and updates (50% writes) at 8 000 req/s on a key-value store with 2 partitions. At t = 60 s a celebrity posts a link to one product: one key gets a huge share of the traffic for a minute. More partitions fix the average — do they fix the hot key?

• Lesson: Partitioning scales with the number of partitions only while keys are spread evenly; a hot key concentrates the load on one partition.
• Setup: sharding · profile load · target 8,000 req/s · 50% reads · seed 17 · chaos off · scripted events: t = 60 s — celebrity effect: one hot key
• Goals: error rate < 0.5% · p99 < 100 ms
• Try: Raise the partitions to 8 and run. Then add a Cache in front of the NoSQL store for the hot reads.
• Observe: At t = 60 s the celebrity effect starts: with 8 partitions the busiest one still saturates; the cache absorbs the hot reads.
• Related: Databases · Caching

A queue absorbs the spike

Each order generates a PDF receipt (50 ms of work). Today the order service waits for it. The spike profile multiplies the load by 5 for 30 s: the workers cannot keep up and orders fail. Let the user go and do the work later.

• Lesson: A queue decouples the user from slow work: the producer waits only for the broker; the backlog absorbs the spike and drains afterwards.
• Setup: async-queue · profile spike · target 150 req/s · 0% reads · seed 19 · chaos off
• Goals: error rate < 0.1% · p99 < 100 ms · consumer lag < 60 s
• Try: Replace the call app → worker by app → Message Queue → worker. Run the spike profile again.
• Observe: Errors disappear; the backlog grows during the 30 s spike and drains after it. Consumer lag = backlog ÷ drain rate.
• Related: Messaging · Compute & services

Publish an event, let consumers react

When an order is placed, the app notifies the buyer, updates analytics and re-indexes availability — today with three synchronous calls in a row. At t = 60 s analytics becomes 20× slower and orders slow down with it. Decouple the order service from its consumers.

• Lesson: With an event log each consumer group reads at its own pace: a slow consumer lags behind without slowing the producer or the others.
• Setup: event-fanout · profile load · target 400 req/s · 0% reads · seed 23 · chaos off · scripted events: t = 60 s — analytics is 20× slower
• Goals: p99 < 120 ms · error rate < 0.1% · consumer lag < 60 s
• Try: Publish to Kafka (app → kafka) and let notification, analytics and the indexer consume (kafka → each).
• Observe: When analytics slows down at t = 60 s, only its lag grows; order latency stays flat.
• Related: Messaging

Cascading failure: a slow dependency

Half of the catalog pages call an external recommendations API. At t = 60 s it becomes 20× slower (3 s). Without timeouts every request holds a thread while waiting; the thread pools run out and the whole catalog fails — although the app's CPU is almost idle (Little's Law: threads = λ × W).

• Lesson: A slow dependency without timeouts exhausts the caller's threads (Little's Law) and takes down requests that never touch it.
• Setup: cascading-failure · profile load · target 800 req/s · 90% reads · seed 29 · chaos off · scripted events: t = 60 s — recommendations API 20× slower
• Goals: error rate < 1% · p99 < 500 ms
• Try: Run as is and read the bottleneck in the Report. Then set a 300 ms timeout on app → recommendations, then add a fallback and a circuit breaker.
• Observe: Without a timeout the app shows threads exhausted with idle CPU. With the timeout only the recommendation calls fail; with the fallback nobody does.
• Related: Reliability

Retry storm: when retries make it worse

The inventory service retries every failed query 3 times, immediately, with a 100 ms timeout. At t = 60 s the database restarts (8 s). When it comes back, it receives up to 4× the normal load, its queue fills, answers take longer than the timeout… and it never recovers, although the trigger is long gone (a metastable failure).

• Lesson: Immediate retries multiply the load on a struggling service; when answers exceed the timeout, every attempt fails and the system stays down after the trigger is gone (metastable failure).
• Setup: retry-storm · profile load · target 2,000 req/s · 90% reads · seed 31 · chaos off · scripted events: t = 60 s — database restarts
• Goals: error rate < 15% · p99 < 300 ms
• Try: Run and watch the edge app → db after the 8 s restart. Then enable the circuit breaker; try also 1 retry with exponential backoff + jitter.
• Observe: The edge rate jumps to ~4× and errors never recover. With the breaker the database drains its queue and the system recovers.
• Related: Reliability

Rate limiting protects the backend

1 000 req/s of real users — plus scraper bots sending twice that. The catalog melts down for everybody. Reject the excess fast at the gateway (429) instead of letting it queue behind the real users.

• Lesson: Rejecting excess traffic fast (429) protects the backend and the real users; letting it queue degrades everyone.
• Setup: rate-limiter · profile load · target 1,000 req/s · 90% reads · seed 37 · chaos off
• Goals: error rate < 1% · p99 < 200 ms
• Try: Set a rate limit on the API Gateway below the app capacity (e.g. 1 500 req/s). Alternatively, add a WAF in front.
• Observe: Bots are cut first (per-client token buckets), real users keep low latency and the app stays below saturation.
• Related: Traffic & edge

Surviving a region outage

The app runs in one cloud region. At t = 60 s the whole region goes down. Build a second region and let DNS move users there. Failover time = detection + DNS TTL: check both.

• Lesson: Surviving a region needs a full second stack and a way to move users: DNS failover takes detection plus TTL.
• Setup: multi-region · profile load · target 1,000 req/s · 90% reads · seed 41 · chaos off · scripted events: t = 60 s — region sa-east-1 is down
• Goals: availability ≥ 95% · p99 < 80 ms
• Try: Draw a second VPC (another region) with LB, app and database; connect DNS to it, attach a Health Check to the DNS and lower its TTL.
• Observe: At t = 60 s the first region disappears. Errors last about (3 s + TTL) for the users routed there.
• Related: Reliability · Traffic & edge

Serverless burst: cold starts and limits

A discount-coupon check runs as a function. The spike profile jumps from 200 to 2 000 req/s. New instances start cold (600 ms) and the account has a concurrency limit of 50. Instances needed = λ × W (Little's Law).

• Lesson: Functions scale per request: instances = λ × W, cold starts hit exactly when traffic jumps, and the concurrency limit throttles the rest.
• Setup: serverless-burst · profile spike · target 200 req/s · 90% reads · seed 43 · chaos off
• Goals: error rate < 0.5% · p99 < 300 ms · cost/month < US$ 5,000
• Try: Raise the concurrency limit, run the spike; then add provisioned instances and compare p99 and cost.
• Observe: Throttling (429) disappears with a higher limit; cold starts show up in p99 at the start of the spike until warm instances exist.
• Related: Compute & services

Flash sale: a limited-stock launch

A limited-stock product goes on sale at t = 60 s: traffic jumps 20× (to ~6 000 req/s) for 90 s. 80% of requests read the product page, 20% try to buy. The database holds the stock (no overselling!) and the payment provider accepts 500 req/s. Today everything collapses. Design for the peak: serve reads from the edge and caches, and put buyers in a waiting room (queue) drained at the speed the inventory and the payment provider can handle.

• Lesson: A flash sale is designed for the peak: reads are absorbed by the edge and caches, and buyers wait in a queue drained at the pace of the inventory and the payment provider.
• Setup: flash-sale · profile load · target 300 req/s · 80% reads · seed 2027 · chaos off · scripted events: t = 60 s — the sale opens
• Goals: p99 < 300 ms · error rate < 1% · consumer lag < 300 s
• Try: Follow the hints one at a time: CDN, catalog cache, then API Gateway routing writes to a waiting-room queue consumed by order workers.
• Observe: With the waiting room, errors vanish and p99 stays low; the consumer lag is how long a buyer waits in line.
• Related: Case studies · Messaging · Caching

URL shortener (sho.rt/xyz)

Every order e-mail and shared post carries a short link. 5 000 redirects/s, 99% reads, a few writes (new links). The classic interview problem: what is the read path, and what store fits a key → URL lookup?

• Lesson: A read-heavy key lookup: cache + key-value store, sized with simple arithmetic.
• Setup: url-shortener · profile load · target 5,000 req/s · 99% reads · seed 47 · chaos off
• Goals: p99 < 50 ms · error rate < 0.1% · cost/month < US$ 3,500
• Try: Replace the SQL database by Cache → NoSQL and size the app tier.
• Observe: p99 drops to a few tens of ms; most redirects never reach the store.
• Related: Case studies

News feed: fan-out on write vs on read

The feed shows posts from the accounts a user follows. Today the feed is assembled when it is read: a heavy query (40 ms) that merges hundreds of sources. 2 000 req/s, 95% reads. Precompute timelines when something is posted instead.

• Lesson: Fan-out on write moves work from the read path (frequent) to the write path (rare): a feed read becomes one key lookup.
• Setup: news-feed · profile load · target 2,000 req/s · 95% reads · seed 53 · chaos off
• Goals: p99 < 100 ms · error rate < 0.5% · consumer lag < 10 s
• Try: Route reads to a timeline cache over a NoSQL store, and writes to the posts database plus Kafka → fan-out workers → timelines.
• Observe: Reads become cheap; the fan-out lag shows how long a post takes to reach the timelines.
• Related: Case studies · Messaging

Chat for a live event

During a live event, users chat in its room: 2 000 req/s, half are new messages. Each message is stored in PostgreSQL and delivered synchronously before the sender gets an answer. Writes overload the database and the sender waits for every delivery.

• Lesson: Chat separates storing messages (partitioned, append-heavy) from delivering them (pub/sub to the connected servers).
• Setup: chat · profile load · target 2,000 req/s · 50% reads · seed 59 · chaos off
• Goals: p99 < 80 ms · error rate < 0.1% · consumer lag < 2 s
• Try: Store messages in a NoSQL store and publish them to Pub/Sub consumed by the delivery service.
• Observe: Write load spreads over partitions; senders no longer wait for delivery.
• Related: Case studies · Traffic & edge

Ride-sharing: a write-heavy geo index

Drivers send their position every 4 s: 20 000 req/s, 90% writes, while riders query nearby drivers. A relational database cannot absorb that write rate. Keep only the latest position per driver in a partitioned store and stream the history.

• Lesson: Location updates are a write-heavy stream: keep only the latest position in a partitioned store and send the history asynchronously.
• Setup: ride-sharing · profile load · target 20,000 req/s · 10% reads · seed 61 · chaos off
• Goals: p99 < 100 ms · error rate < 0.5% · cost/month < US$ 6,000
• Try: Replace SQL by a partitioned NoSQL geo index; sample positions into Kafka for analytics.
• Observe: The relational primary was saturated by writes; the partitioned store absorbs them with headroom.
• Related: Case studies · Databases

Video streaming: CDN + object store + transcoding

Creators upload videos; users watch them. 2 000 segment requests/s from viewers go through the app, and each upload is transcoded synchronously (20 s per video) while the uploader waits. Serve segments from the edge and transcode in the background.

• Lesson: Video is bytes at the edge plus heavy batch work in the background: CDN + object store for viewers, a queue + workers for transcoding.
• Setup: video-streaming · profile load · target 2,000 req/s · 99% reads · seed 67 · chaos off
• Goals: p99 < 150 ms · error rate < 0.5% · consumer lag < 120 s
• Try: Viewers → CDN → Object Store; uploads → API Gateway → app → queue → transcoders.
• Observe: The app tier almost disappears from the viewing path; the transcoding backlog drains at replicas × 1/20 s.
• Related: Case studies · Traffic & edge

Notification system

Every purchase sends a confirmation e-mail (50 req/s), and a scheduler sends 6 000 digest e-mails every 2 minutes — all at the same instant. The e-mail provider accepts 200 sends/s. Today buyers wait for the provider and fail whenever the digest burst arrives.

• Lesson: Notifications are asynchronous by nature: queue them, pace them to the provider quota, and never fire scheduled jobs all at once.
• Setup: notification-system · profile load · target 50 req/s · 0% reads · seed 71 · chaos off
• Goals: error rate < 0.5% · p99 < 150 ms · consumer lag < 120 s
• Try: Put a queue in front of the notification service and spread the scheduler's jobs over 60 s.
• Observe: Buyers no longer feel the digest burst; the provider quota sets the drain rate.
• Related: Case studies · Compute & services

Payments: retries need idempotency

Orders call the payment provider synchronously with a 1 s timeout and 2 immediate retries. At t = 60 s the provider becomes 4× slower: calls time out, and each retry after a timeout may charge the card AGAIN — the provider did receive the first attempt. Make payments safe and keep buyers fast.

• Lesson: A retry after a timeout may repeat an operation that already happened: payments need idempotency keys, and the buyer should not wait for the provider.
• Setup: payment-system · profile load · target 300 req/s · 50% reads · seed 73 · chaos off · scripted events: t = 60 s — payment provider 4× slower
• Goals: duplicate charges < 1 · error rate < 1% · p99 < 1,500 ms
• Try: Turn on the idempotency key on the payment edge; then move the call behind a queue and a payment worker.
• Observe: Duplicate charges drop to zero with the key; with the queue, buyer latency no longer depends on the provider.
• Related: Case studies · Compute & services

Search autocomplete

Every keystroke in the search box asks for suggestions: 10 000 req/s, all reads, and they must feel instant (< 50 ms). The search cluster cannot take it. Most prefixes are popular and repeat a lot.

• Lesson: Autocomplete is a latency problem: popular prefixes repeat, so a prefix cache answers most keystrokes.
• Setup: search-autocomplete · profile load · target 10,000 req/s · 100% reads · seed 79 · chaos off
• Goals: p99 < 50 ms · error rate < 0.1% · cost/month < US$ 5,000
• Try: Insert a Cache between the app and the search service; size the app tier. Lower the target RPS to simulate client debouncing.
• Observe: The search cluster load drops by the hit ratio and p99 meets the 50 ms budget.
• Related: Case studies · Caching

AI support assistant (RAG + tools)

The support assistant answers questions about products and orders. Its agent loop takes 6 steps (6 LLM calls + tool calls) per question, the provider quota is 10 calls/s, and nothing checks for prompt injection. 2 questions/s already exceed the quota, take seconds, and cost a fortune.

• Lesson: In an LLM application latency and cost scale with the number of model calls: fewer agent steps, a semantic cache and quota-aware routing matter more than raw capacity; guardrails belong in the path.
• Setup: ai-assistant · profile load · target 2 req/s · 100% reads · seed 83 · chaos off
• Goals: error rate < 1% · p99 < 10,000 ms · cost/month < US$ 60,000 · unsafe answers < 0.1%
• Try: Reduce the agent steps, turn on the semantic cache and a fallback model, and put a Safety Mesh between the gateway and the assistant.
• Observe: Errors (429) vanish under the quota, cost falls with the calls per question and the unsafe-answer rate drops.
• Related: AI systems

You can't fix what you don't measure

Utilization is hidden ("?") until a service is connected to Metrics. At t = 50 s one catalog replica dies; without health checks the load balancer notices only after 30 s, and without alerting nobody repairs it quickly. Wire the observability stack: metrics on every service, health checks, alerting, logs and tracing.

• Lesson: Observability changes outcomes: health checks shorten detection, alerting shortens repair, and metrics make utilization visible.
• Setup: observability-stack · profile load · target 600 req/s · 90% reads · seed 89 · chaos off · scripted events: t = 50 s — a catalog replica dies
• Goals: availability ≥ 99% · p99 < 200 ms · services measured ≥ 100%
• Try: Connect Metrics to the LB, app and database; add a Health Check on the LB, Alerting on Metrics, plus Logs and Tracing on the app.
• Observe: The "?" become numbers; the crash at t = 50 s costs seconds instead of half a minute; the Trace tab shows where a slow request spends its time.
• Related: Observability

Microservices on Kubernetes: pods vs nodes

The app was split into catalog, inventory, order and notify microservices, each with its own database, behind an API gateway, on a Kubernetes cluster of 2 nodes × 6 pods. At 2 000 req/s the catalog's HPA asks for more pods than the cluster can schedule: they stay Pending, and the catalog drowns. Buying is a synchronous chain (order → inventory, catalog, notify, its DB): every hop adds its latency to the buyer's wait.

• Lesson: The HPA adds pods, but only nodes add room: pods beyond nodes × pods per node stay Pending and serve nothing. In a chain of synchronous microservices every hop adds its latency to the user's wait, so work that the user does not need to wait for belongs behind a queue.
• Setup: microservices · profile load · target 2,000 req/s · 80% reads · seed 2031 · chaos off
• Goals: error rate < 1% · p99 < 200 ms · cost/month < US$ 8,500
• Try: Select the cluster frame: pods = nodes × pods per node. Add nodes or turn on the cluster autoscaler (a node takes ~90 s, so keep some headroom), raise the catalog's minimum replicas, and put a queue between order and notify.
• Observe: As drawn, errors ≈ 25% and p99 ≈ 3.8 s: the catalog's HPA wants ~14 pods, but the 2 × 6 cluster is full (6 catalog + 2 order + 2 inventory + 2 notify), so 9 pods stay Pending; and the buyer also waits for notify (120 ms median, σ 0.6). The possible solution — 3 nodes with the cluster autoscaler (max 6; it adds a 4th node at ~45 s, ready at ~135 s), catalog replicas and minimum 10, and order → queue confirmations → notify — passes with errors ≈ 0%, p99 ≈ 60 ms and ≈ 8 000 USD/month. Read the cost lines: the API gateway's per-request price, not the nodes, dominates the bill.
• Related: Compute & services · Cloud-native platforms

Kubernetes: losing a node

Catalog and pricing run 6 pods each on a cluster of 3 nodes × 4 pods: completely full. At t = 60 s a worker node dies and takes a third of the pods with it. Callers keep sending requests to the dead pods until they notice (30 s behind the load balancer, 40 s for the direct catalog → pricing call), and the lost pods have nowhere to be rescheduled. Survive the loss of any one node.

• Lesson: N+1 applies to nodes, not only to replicas: a lost node takes several pods at once, callers keep calling them until they notice, and Kubernetes can only reschedule them where there is room. A direct service-to-service call needs service discovery and a timeout to stop waiting for dead pods.
• Setup: k8s-node-failure · profile load · target 1,700 req/s · 90% reads · seed 2033 · chaos off · scripted events: t = 60 s — a worker node dies
• Goals: error rate < 1% · p99 < 150 ms · cost/month < US$ 2,000
• Try: Leave room in the cluster (more pods per node or one more node) and size the replicas so the survivors carry the load (ρ after losing a third below ~0.85). Put a timeout with retries on catalog → pricing, and connect a Service Registry to catalog and pricing.
• Observe: As drawn, the node takes 2 catalog + 2 pricing pods. Catalog → pricing calls keep going to the dead pricing pods for 40 s; each one hangs 10 s, the catalog's threads run out and the failure cascades (~90% errors in that window); after 40 s the lost pods are recreated but stay Pending (no room): ≈ 27% errors and p99 > 4 s over the run. The possible solution — 8 pods per node (3 × 8 = 24 slots: after losing a node, 16 slots for the 16 pods — N+1 for nodes), 8 replicas each, a Service Registry connected to catalog and pricing, and a 300 ms timeout + 2 retries on catalog → pricing — passes with errors ≈ 0.8% and p99 ≈ 43 ms: dead pods are dropped in 3 s, a call that lands on one gives up after 300 ms and is retried on a live pod, the surviving pods carry the load, and the 6 lost pods are all rescheduled after 40 s.
• Related: Cloud-native platforms · Reliability

Database failover: RTO and RPO

Flash-sale checkout: 1 500 req/s, half of them writes (orders), on a DB Cluster with one primary and one follower replicated asynchronously. At t = 90 s the primary dies. Until a follower is promoted (30 s) every write fails, and the writes still inside the replication lag are lost for good: orders paid and forgotten. Make the failover lose nothing and hurt little.

• Lesson: A failover has two costs: the time without a primary (RTO) and the writes not yet replicated (RPO). Synchronous or quorum replication makes the RPO zero at the price of a round trip per write; a fast failover and more shards shrink the RTO and the share of writes it hits — and every copy is paid.
• Setup: db-failover · profile load · target 1,500 req/s · 50% reads · seed 2035 · chaos off · scripted events: t = 90 s — the primary of orders-db dies
• Goals: lost writes < 1 · availability ≥ 99% · p99 (writes) < 100 ms
• Try: Switch the replication to sync or quorum, shorten the failover time and split the keys into shards. Then inject a latency degradation on the cluster (it slows one follower) and compare sync and quorum on the write p99.
• Observe: As drawn, ≈ 1 570 writes are lost when the primary dies, writes fail for 30 s (availability ≈ 90.6%), and after the promotion a single node carries reads and writes, so the write p99 reaches ≈ 375 ms. The possible solution — 2 shards, 2 followers each, quorum, failover 5 s — passes with 0 lost writes, availability ≈ 99.2% and a write p99 ≈ 34 ms; but the database bill goes from 900 to 2 700 USD/month (2 shards × 3 nodes × 450).
• Related: Databases · Reliability