Disaster Recovery

Disaster recovery is the plan for when the thing you were relying on is gone: a deleted table, a corrupted backup, a compromised account, a lost region. Nobody gets credit for it until the day it's the only thing that matters, so it has to be planned, priced and practiced before that day.

Set Targets From Impact

Every system gets two numbers, agreed with the people who own the business impact, not picked by engineering:

Target	Question	Example
RTO (recovery time objective)	How long can it be down?	4 hours
RPO (recovery point objective)	How much data can we lose?	5 minutes

• Set them per system. The ledger and the marketing site don't deserve the same budget.
• Tighter targets cost more, in money and in complexity. Make that trade explicitly.
• Write them down next to the system's SLOs, and check the DR design actually meets them.

Backups

• 3-2-1: three copies of the data, on two different kinds of storage, one of them off-site. Add one copy that's immutable: write-once storage or a separate account the production credentials can't touch, so ransomware or a compromised key can't delete your way out of recovery.
• Point-in-time recovery for databases: continuous log archiving, so you can restore to the minute before the bad migration, not last night. See PostgreSQL continuous archiving.
• Back up everything needed to rebuild, not just data: configuration, secrets (in their own secure store), DNS, and the code that defines the infrastructure.
• Replication isn't a backup. A replica faithfully copies the DELETE too. You need history, not just copies.

Restores Are What Matter

As the Google SRE book puts it: "No one really wants to make backups; what people really want are restores."

• Restore on a schedule, automatically, into a scratch environment, and verify the result (row counts, checksums, an application smoke test).
• Time it. The restore time is your real RTO, whatever the document says.
• Alert when a restore drill fails like any other production failure.

A backup that has never been restored is a hope, not a backup.

Strategies

There are four broad strategies, in rising cost and falling recovery time. The names are AWS's, but the tradeoff is the same on any provider: you pay continuously for capacity in the recovery location, to buy back time on the day you need it.

Strategy	What's running in the recovery region	Recovery
Backup and restore	Nothing but backups	Hours: rebuild infrastructure, restore data
Pilot light	Data replicated; core infrastructure provisioned but switched off	Tens of minutes: switch on and scale up
Warm standby	A scaled-down, fully working copy	Minutes: scale up
Multi-site active/active	Full production, serving traffic	Near zero for infrastructure loss

Choose from the RTO and RPO, not from ambition. Most systems need backup and restore done properly. Even active/active needs backups: a data corruption replicates to every region, and the recovery point is always before it was discovered.

Rebuild From Code

If the infrastructure only exists as clicks in a console, recovering it means remembering those clicks under pressure.

• Infrastructure as code for everything: networks, databases, queues, DNS, permissions. Recovery is running the same code in a new account or region.
• Immutable artifacts. Applications deploy as versioned images or builds, so rebuilding is redeploying, not reinstalling. See Containers & Immutable Infrastructure.
• Know your dependencies' DR. Your RTO can't beat your payment processor's, your identity provider's or your DNS host's. List them, read their commitments, and decide what you do while each one is down.

Runbooks and Game Days

• A runbook per scenario: region loss, database corruption, compromised credentials, a dependency outage. Each one says who decides, the exact steps, and how to verify recovery. See On-Call & Runbooks.
• Game days rehearse them for real: fail over a region, restore the database, rotate every secret, on a scheduled day with everyone watching. The gaps you find on a game day are gaps you didn't find in an incident.
• Every real incident updates the plan. See Incident Postmortems.

This Site

This site holds no user data, and everything it serves is rebuilt from the repository on every deploy, so its recovery plan is redeploying from Git. The only datastore, Upstash Redis, holds rate-limit counters that expire within an hour; losing it costs nothing, and the API falls back to in-memory limits while it's gone. That's the cheapest DR there is: make most of the system disposable.

References

Reference

• AWS: Disaster recovery options in the cloud, backup and restore, pilot light, warm standby, multi-site active/active
• AWS Well-Architected: Plan for disaster recovery, defining RTO and RPO
• PostgreSQL: Continuous archiving and point-in-time recovery, restoring to a moment, not a nightly snapshot

In practice

• Google SRE Book: Data Integrity, restores over backups
• Veeam: The 3-2-1 backup rule, three copies, two media, one off-site, plus an immutable copy
• Principles of Chaos Engineering, testing recovery on purpose