Backups & Point-in-Time Recovery

A Backup You Never Restored Is a Guess

Teams love saying they have backups. What they often have is a scheduled job that writes files somewhere and a shared hope that those files are useful. We rejected that level of confidence for Fieldwork because the system is explicitly multi-tenant: tenants, workspaces, projects, tasks, memberships, and users all live in shared tables with foreign-key relationships that matter during restore. If you discover under incident pressure that your restore procedure skips WAL files, brings the cluster up with the wrong target time, or passes health checks while half the data relationships are broken, you do not have a recovery plan. You have storage bills.

That framing changed the whole topic. We did not start from "which backup product should we use?" We started from "what failure are we promising we can recover from, and how would we prove it?" Once you ask that question honestly, a nightly logical dump stops looking sufficient for a transactional system with user-facing writes all day long. Nightly dumps are too coarse if a bad migration or operator error happens at 16:42 and the last full snapshot is from 02:00. That potential data-loss window was too large for the kind of system Fieldwork is pretending to be.

Why We Chose WAL-Based Continuous Backup

Fieldwork chose WAL-based continuous backup because it matches the system's actual risk profile. We wanted the ability to restore to a timestamp just before a destructive migration, a runaway delete, or a bad bulk update against one tenant's workspace tree. Base backups give you a starting snapshot; WAL segments give you the history required to replay from that snapshot to a precise recovery target. That is a better fit than pretending a single nightly export can describe a day of transactional writes accurately enough.

We still kept logical exports for narrower jobs like selective inspection or moving a small dataset into analysis tooling, but we rejected them as the primary disaster-recovery mechanism. The real decision was about recovery guarantees, not tool popularity. WAL archiving costs more storage and requires more operational care, yet the trade-off was worth it because the course system's task and membership data changes too often to accept a wide restore gap.

archive_mode = on
archive_command = 'wal-g wal-push %p'
archive_timeout = 60
max_wal_senders = 10
wal_level = replica
restore_command = 'wal-g wal-fetch %f %p'

Restore Drills Need a Script, Not Heroics

The restore drill mattered more than the configuration. We rejected documentation that only said "restore from backup" because that is not a procedure; it is a wish. Fieldwork's recovery rehearsal needed a target timestamp, a clean restore destination, a database boot sequence, and then application-level validation that the recovered data still made sense. That last part is easy to skip and dangerous to skip. A database can start successfully while still missing the tenant, workspace, project, and task relationships your API relies on.

So the drill became scripted and repeatable. Restore to a chosen time, wait for the instance to accept queries, run a smoke test that checks row counts and a few critical tenant-scoped invariants, then record elapsed time and findings. The goal was not to impress anyone with backup tooling. It was to reduce incident-time improvisation. If recovery depends on whoever remembers the right commands from a wiki page, you are outsourcing resilience to adrenaline.

What Recovery Guarantees We Actually Had

By the end of the decision, Fieldwork had a concrete recovery story instead of backup vibes. Base backups and WAL archives defined the raw materials. Rehearsed restore drills defined the confidence. And application-level verification defined what success meant. We were careful not to overclaim. PITR reduces data loss and speeds recovery, but it does not eliminate downtime, and it does not magically reconstruct side effects outside Postgres unless those systems have their own recovery model. That honesty is part of the design.

So the lasting decision was simple: continuous backup with WAL, regular restore rehearsals, and success criteria expressed in Fieldwork's domain, not only in database process health. That is the difference between saying "we back up Postgres" and being able to say "we can restore tenant-scoped task data to just before the damaging event and prove the recovered graph is usable."

• Choose backup strategy based on recovery objectives, not on which tool is easiest to demo.
• Use base backups plus WAL archiving when you need point-in-time recovery for active transactional data.
• Treat replicas as availability tools, not as sufficient backup strategy on their own.
• Rehearse restores with scripted verification against real domain invariants like tenant, workspace, project, and task relationships.
• Record restore duration and failure points so recovery claims stay honest over time.