Disaster Recovery

Two bad nights, two very different disasters.

On the first, the cloud region where BoxOffice runs has a major outage: networking inside the region is degraded for hours. Every replica of every service is in that region — three availability zones, all affected. Redundancy inside the region does not help.

On the second, everything is up. At 22:00 an engineer runs a cleanup script against what they believe is the staging database. It is production: DELETE FROM orders WHERE created_at < now() — every order is gone. The PostgreSQL standby in another zone faithfully replicates the delete within milliseconds. Redundancy does not help here either; it spread the damage.

Disaster recovery (DR) is the set of strategies, tools and procedures to restore service and data after events like these — events larger than what the resilience patterns of the previous page absorb. It starts with two numbers.

RPO and RTO

RPO — Recovery Point Objective : The maximum amount of data, measured in time, you accept to lose. RPO = 5 minutes means that after recovery, at most the last 5 minutes of writes may be missing.

RTO — Recovery Time Objective : The maximum time from the disaster until service is restored. RTO = 1 hour means fans can buy tickets again at most one hour after the failure.

flowchart LR
    lb["last good<br/>recovery point"] -->|"RPO: data lost"| d(("disaster<br/>22:00"))
    d -->|"RTO: service down"| r["service<br/>restored"]

They are business decisions expressed in minutes, and they map to money: every minute of RTO during an on-sale is lost sales; every minute of RPO is orders (and payments!) that must be reconstructed by hand or refunded. Lower numbers cost more, roughly exponentially.

Backups

A backup is a copy of data, kept independently from the original, from which you can restore. The kinds:

Point-in-time recovery (PITR) is what saves the second night. PostgreSQL continuously archives its WAL³ to object storage (with tools such as pgBackRest or WAL-G, or automatically in managed databases like Amazon RDS and Cloud SQL). To undo the 22:00 delete, restore the latest base backup and replay the WAL up to just before it:

# restore the order database to 21:59:30, just before the accidental DELETE at 22:00
pgbackrest --stanza=order --type=time "--target=2027-03-14 21:59:30-03" \
  --target-action=promote restore

RPO is then the WAL archiving delay (seconds to a minute), and RTO is the restore time, which you must estimate — and measure:

Rules for backups that actually work

• 3-2-1: at least 3 copies of the data, on 2 different kinds of storage, 1 of them off-site (another region or account). Modern practice adds one immutable copy (object lock / WORM — write once, read many) that even an attacker with admin credentials cannot delete, against ransomware.
• Separate the blast radius. Backups in the same account, with the same credentials, as production can be deleted by the same mistake or the same attacker.
• Encrypt backups and manage the keys separately; a backup is a full copy of your customers' data (LGPD/GDPR apply).
• Test restores regularly. A backup that has never been restored is a hope. Automate a periodic restore into a scratch environment and check the data (row counts, checksums, a smoke test of the application).
• Monitor backup jobs and WAL archiving lag; alert when they fail or fall behind.

Multi-AZ and multi-region

Synchronous replication across regions puts a cross-region round trip in every write: \(\approx 30\)–\(100\) ms per commit is often unacceptable for the seat-hold path. So multi-region DR usually means asynchronous replication, and an RPO equal to the replication lag at the moment of the disaster. That is why "RPO = 0 across regions" is either very expensive (synchronous replication, or a globally consistent database such as Spanner or CockroachDB paying the latency) or a misunderstanding.

Multi-region is also a legal question: personal data may be subject to residency rules (LGPD in Brazil, GDPR in the EU), which constrain which regions can hold the copy.

The four DR strategies

The AWS disaster-recovery whitepaper describes a spectrum of four strategies, from cheap and slow to expensive and instant¹. The same ideas apply on any cloud².

flowchart LR
    subgraph A ["Region A (primary)"]
        appA["services"] --> dbA[("databases")]
    end
    subgraph B ["Region B (recovery)"]
        bk[("backups + WAL<br/>(object storage)")]
        iac["infrastructure as code<br/>(nothing running)"]
    end
    dbA -->|"backups, WAL archive"| bk

flowchart LR
    subgraph A ["Region A (primary)"]
        appA["services"] --> dbA[("databases")]
    end
    subgraph B ["Region B (recovery)"]
        dbB[("database replicas<br/>(running)")]
        appB["services<br/>(off: 0 replicas)"]
    end
    dbA -->|"async replication"| dbB

flowchart LR
    subgraph A ["Region A (primary, 100%)"]
        appA["services × 12"] --> dbA[("databases")]
    end
    subgraph B ["Region B (standby, ~10%)"]
        appB["services × 2<br/>(running, idle)"] --> dbB[("replicas")]
    end
    dbA -->|"async replication"| dbB

flowchart LR
    dns["GeoDNS / global LB"] --> A
    dns --> B
    subgraph A ["Region A (serving)"]
        appA["services"] --> dbA[("data")]
    end
    subgraph B ["Region B (serving)"]
        appB["services"] --> dbB[("data")]
    end
    dbA <-->|"replication both ways"| dbB

Active-active and the seat inventory

Active-active is easy for read-mostly data: the catalog can be read in any region from local replicas and the CDN. It is hard for data with strong invariants: if São Paulo and Virginia both accept holds for seat B-12 and replicate asynchronously, two fans can buy the same seat — exactly what BoxOffice must never do (see Consistency). The options:

• Home region per entity: each event (show) is owned by one region; all holds and sales for it go there; the other region is its warm standby. Fans in other regions pay a little latency for that show. Ownership moves only during a failover.
• A globally consistent database (Spanner, CockroachDB, YugabyteDB): consensus across regions on every write, at the cost of cross-region latency on the seat-hold path.
• Multi-leader with conflict resolution (last-writer-wins, CRDTs): fine for a shopping cart or a "favorites" list, unacceptable for seats.

Failover between regions

A regional failover is a sequence of steps, each with a duration that adds to the RTO:

sequenceDiagram
    participant M as external monitors
    participant OC as on-call
    participant DNS as DNS (health-checked)
    participant A as region A
    participant B as region B (warm standby)
    Note over A: region degraded at 22:00
    M->>OC: purchase probes failing from 3 locations (2 min)
    OC->>OC: confirm it is regional, declare disaster (5 min)
    OC->>A: fence: stop writes in A (if reachable)
    OC->>B: promote database replicas (1 min)
    OC->>B: scale services from 2 to 12 replicas (5 min)
    OC->>DNS: point boxoffice.example to region B
    Note over DNS: clients follow after the TTL (60 s)
    DNS-->>B: traffic arrives, RTO ≈ 14 min

Points that decide whether it works:

• Detection from outside. Probes that run inside the failing region may fail with it. Use external synthetic checks of the purchase journey.
• Automatic or human decision? Automatic failover is fast but risky: a network blip that triggers a regional failover (and then a failback) can cause more damage than it prevents ("flapping"). Many teams automate within a region and keep a human decision, with a rehearsed runbook, across regions.
• Fencing. Before the standby accepts writes, make sure the old primary cannot (revoke its credentials, stop it, or use a lease). Two primaries accepting holds is split brain: double-sold seats.
• DNS TTL. Clients and resolvers cache DNS answers for the TTL; a 1-hour TTL means a 1-hour tail of traffic going to the dead region. Health-checked DNS records (e.g. Amazon Route 53 failover records) with short TTLs on the entry point (see DNS and CDN).
• Static stability. The recovery region must not depend on the failed one: container images, secrets, configuration and capacity quotas must already be in region B.
• Failback. Coming back is another migration: resynchronize data from B to A, then switch at a quiet time. Plan and rehearse it too.

The BoxOffice DR plan

Why a warm standby for tier 0, and not something cheaper or something stronger: the 15-minute RTO rules out backup and restore (restoring the 2 TB order database alone takes about 2.8 hours) and makes pilot light risky (promoting and scaling from zero takes tens of minutes); active-active would let two regions sell seat B-12 at once. The price is a second region running at about a tenth of the scale all year, plus the drills that prove it can take over.

And the practices that keep the plan honest:

• A runbook per scenario (region loss, database corruption, compromised account) with the exact commands, owners and the decision criteria.
• DR drills twice a year: actually fail over to region B for an hour in a quiet period, and restore a production backup into a scratch environment every month. The chaos engineering page covers how to run them as game days.
• Review RPO/RTO with the business yearly: the stadium contract may demand more, or the analytics tier may accept less.

How it connects

flowchart LR
    BIZ["RPO and RTO<br/>per data class"] --> STRAT["backup and restore · pilot light ·<br/>warm standby · active-active"]
    INFRA["infrastructure loss<br/>(zone, region)"] --> REPL["replication, multi-AZ<br/>→ Databases — replication"]
    LOGIC["logical corruption<br/>(the bad DELETE)"] --> PITR["PITR backups, 3-2-1,<br/>immutable copy"]
    REPL --> STRAT
    PITR --> STRAT
    STRAT --> AA["active-active vs the seat invariant<br/>→ Distributed — consistency"]
    STRAT --> FO["regional failover:<br/>detect, fence, promote, scale"]
    FO --> FENCE["fencing, no split brain<br/>→ Distributed — consensus"]
    FO --> DNS["health-checked DNS, short TTL<br/>→ Edge — DNS"]
    FO --> DRILL["DR drills, restore tests<br/>→ Chaos engineering"]

Palette components: sql (standby, replicas, PITR) · objstore (base backups and the WAL archive) · dns (failover records and TTL) · cdn (serves the catalog stale while the origin recovers) · kafka (replays unsent notifications) · app (scaled from 2 to 12 replicas in the standby region). Scenarios: db-failover, multi-region, db-replicas. Continues in: Databases — replication, Distributed — consistency, Distributed — consensus, Edge — DNS & CDN, Cloud-native — delivery (infrastructure as code), Project — game day.