Stage 7 · Master
Phase 20 — Production Readiness
Disaster Recovery
Prove the platform can be restored into infrastructure that shares nothing with the primary, on a measured schedule, with named RPO and RTO numbers backed by evidence instead of assumption.
The Backup lesson made off-cluster, point-in-time-capable backups exist. This lesson answers the harder question leadership actually asks during an incident review: if the entire primary cluster and region disappeared right now, how long would it take to have a working platform again, and how much data would be gone forever. Those two numbers — recovery time objective and recovery point objective — are meaningless until they have been measured by an independent restore drill, not estimated from how backups are configured.
Restore Into Infrastructure That Shares Nothing With the Primary
Spinning up a second Postgres instance on the same Kubernetes cluster, same node pool, and same cloud account as the primary and restoring repo2 into it will succeed — and will prove only that pgBackRest works, not that the platform survives losing that cluster or account. A genuine restore drill provisions a database in infrastructure the primary's failure cannot touch: a different cloud account, a different region, created by automation that does not depend on the primary cluster being reachable at all.
#!/usr/bin/env bash
set -euo pipefail
# Runs against a freshly provisioned DR-account Postgres instance —
# never the primary, never a scratch database on the primary's cluster.
: "${DR_DATABASE_HOST:?set to a host in the disaster-recovery account}"
: "${DR_TARGET_TIME:?e.g. 2024-03-14 09:14:52+00, or 'now' for latest}"
drill_start_epoch=$(date +%s)
echo "==> restoring repo2 into $DR_DATABASE_HOST (target: $DR_TARGET_TIME)"
pgbackrest \
--stanza=hoa \
--repo=2 \
--pg1-path=/var/lib/postgresql/data \
--type=time \
--target="$DR_TARGET_TIME" \
--target-action=promote \
restore
echo "==> waiting for restored instance to accept connections"
until pg_isready --host="$DR_DATABASE_HOST" --port=5432 --timeout=5; do
sleep 5
done
echo "==> measuring recovery point: newest committed row per service"
psql "postgres://hoa_maintenance_app_1@$DR_DATABASE_HOST:5432/hoa" \
-tAc "SELECT max(created_at) FROM maintenance.charges;" \
> /tmp/dr-maintenance-newest-row.txt
echo "==> running the read-path smoke suite against the restored instance"
DATABASE_URL="postgres://hoa_maintenance_app_1@$DR_DATABASE_HOST:5432/hoa" \
go test -C tests/disaster-recovery ./... -run TestRestoredInstance -v
drill_end_epoch=$(date +%s)
rto_seconds=$(( drill_end_epoch - drill_start_epoch ))
echo "==> recovery time objective measured this drill: ${rto_seconds}s"
echo "==> newest recovered row: $(cat /tmp/dr-maintenance-newest-row.txt)"
echo "==> requested target time: $DR_TARGET_TIME"
python3 deploy/disaster-recovery/record_drill_result.py \
--rto-seconds="$rto_seconds" \
--target-time="$DR_TARGET_TIME" \
--newest-row-file=/tmp/dr-maintenance-newest-row.txtgo test is scoped with -C tests/disaster-recovery, exactly like every module-scoped test invocation elsewhere in this platform — a bare go test ./... at the repository root would try to compile every service's module at once and is never how tests run here.
Turn RPO and RTO From Targets Into Measured Facts
A recovery point objective is a promise about how much committed data a restore may lose; a recovery time objective is a promise about how long a restore may take. Both numbers are worthless as targets alone — they only mean something once a real drill measures the actual gap and the actual duration, on infrastructure that was not warmed up in advance for the test.
#!/usr/bin/env python3
"""Append one disaster-recovery drill result to the append-only ledger.
Run at the end of every restore-drill.sh execution. Never edits or
deletes a prior row — the point of this ledger is a trend the team
cannot quietly rewrite after a bad drill.
"""
import argparse
import datetime
import json
import pathlib
LEDGER_PATH = pathlib.Path("deploy/disaster-recovery/drill-history.jsonl")
RPO_TARGET_SECONDS = 15 * 60
RTO_TARGET_SECONDS = 60 * 60
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--rto-seconds", type=int, required=True)
parser.add_argument("--target-time", required=True)
parser.add_argument("--newest-row-file", required=True)
args = parser.parse_args()
newest_row_raw = pathlib.Path(args.newest_row_file).read_text().strip()
target_time = datetime.datetime.fromisoformat(args.target_time.replace(" ", "T"))
newest_row = datetime.datetime.fromisoformat(newest_row_raw.replace(" ", "T"))
rpo_seconds = abs((target_time - newest_row).total_seconds())
result = {
"drill_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"target_time": args.target_time,
"newest_recovered_row": newest_row_raw,
"measured_rpo_seconds": rpo_seconds,
"measured_rto_seconds": args.rto_seconds,
"rpo_within_target": rpo_seconds <= RPO_TARGET_SECONDS,
"rto_within_target": args.rto_seconds <= RTO_TARGET_SECONDS,
}
with LEDGER_PATH.open("a", encoding="utf-8") as ledger:
ledger.write(json.dumps(result) + "\n")
print(json.dumps(result, indent=2))
if not (result["rpo_within_target"] and result["rto_within_target"]):
raise SystemExit(
f"drill missed target: rpo_ok={result['rpo_within_target']} "
f"rto_ok={result['rto_within_target']}"
)
if __name__ == "__main__":
main()Appending to drill-history.jsonl rather than overwriting a single 'last result' file means a slipping RTO trend across ten drills is visible in the ledger even if any single drill still happened to pass — a pattern a one-off pass/fail check would hide.
| Question | Backup lesson answers | This lesson answers |
|---|---|---|
| Does an off-cluster copy of the data exist? | Yes — repo2, continuously extended by WAL archiving | N/A |
| Can that copy actually be restored? | pgbackrest check confirms connectivity and integrity | restore-drill.sh proves an end-to-end restore into independent infrastructure actually completes |
| How much data would a real incident lose? | Not answered — retention policy is a ceiling, not a measurement | measured_rpo_seconds, computed from the drill's actual target time vs. actual newest recovered row |
| How long would recovery take? | Not answered | measured_rto_seconds, timed from drill start to a passing smoke suite against the restored instance |
Schedule the Drill and Alert When It Would Fail an Audit
A restore drill run once, by hand, the week the disaster-recovery plan was written, decays the same way an unmonitored backup does — nobody notices when the DR-account credentials expire, the restore script drifts out of sync with a new migration, or the smoke suite stops covering a service added six months later. Running the drill on a schedule, with its own alerting, keeps the RPO and RTO numbers current instead of historical.
name: disaster-recovery-drill
on:
schedule:
- cron: "0 6 * * 1"
workflow_dispatch: {}
jobs:
restore-drill:
runs-on: ubuntu-latest
environment: disaster-recovery
steps:
- uses: actions/checkout@v4
- name: Provision scratch instance in the DR account
run: deploy/disaster-recovery/provision-dr-instance.sh
env:
DR_CLOUD_ACCOUNT: ${{ secrets.DR_CLOUD_ACCOUNT_ID }}
DR_REGION: dr-region
- name: Run independent restore drill
run: deploy/disaster-recovery/restore-drill.sh
env:
DR_DATABASE_HOST: ${{ steps.provision.outputs.host }}
DR_TARGET_TIME: "now"
- name: Tear down scratch DR instance
if: always()
run: deploy/disaster-recovery/teardown-dr-instance.sh
- name: Page on-call if the drill missed RPO or RTO
if: failure()
run: |
curl -fsS -X POST "${{ secrets.PAGERDUTY_WEBHOOK_URL }}" \
-H 'Content-Type: application/json' \
-d '{"summary":"weekly disaster-recovery drill failed to meet RPO/RTO","severity":"critical"}'The DR_CLOUD_ACCOUNT_ID secret and the disaster-recovery GitHub Environment are deliberately distinct from the primary deployment's credentials — the workflow itself never depends on the primary cluster being healthy, which is the property being tested in the first place.
provision-dr-instance.sh and teardown-dr-instance.sh only ever talk to the DR cloud account, and restore-drill.sh only ever talks to $DR_DATABASE_HOST — none of these scripts accept the primary cluster's kubeconfig or database URL as a fallback. A drill that could silently succeed by restoring onto the primary's own infrastructure would pass every week without ever proving the thing it exists to prove.
Applied exercise
Diagnose a drill that passes but should not be trusted
This week's disaster-recovery-drill.yml run went green: measured_rto_seconds was 1,140 (under the 3,600 target) and measured_rpo_seconds was 200 (under the 900 target). A teammate wants to close the DR ticket. You notice provision-dr-instance.sh was changed two weeks ago to reuse a warm, already-running DR instance instead of provisioning a fresh one each time, to save cost.
- Explain specifically what the reused warm instance invalidates about this week's RTO measurement, even though the recorded number looks good.
- Explain whether the RPO measurement is affected by the same change, and why or why not.
- Rewrite provision-dr-instance.sh's contract (in words, not code) so cost savings do not compromise what the drill is measuring.
- State what you would tell the teammate about closing the ticket, and why.
Deliverable
A short written diagnosis distinguishing which of RTO and RPO the warm-instance shortcut invalidates, plus a corrected provisioning contract and a recommendation on the ticket.
Completion checks
- The answer identifies that RTO no longer includes the time to provision infrastructure from zero, which is exactly the scenario a real disaster requires.
- The answer correctly reasons about whether RPO — a measure of data loss, not provisioning time — is or is not affected.
- The recommendation does not close the ticket until the drill is re-run against freshly provisioned infrastructure.
Why must restore-drill.sh run against a freshly provisioned instance in a separate cloud account rather than a long-running scratch database on the primary cluster?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.