Database Backup in Kubernetes

Velero volume snapshots, pg_basebackup CronJobs, and offsite backup patterns.

7 min readDatabases for SREBuild

Backup Challenges in Kubernetes

Backing up databases in Kubernetes is harder than on VMs because: pods are ephemeral, storage is dynamic, and database processes run inside containers with limited access. You need to combine Kubernetes-native tools (Velero) with database-native tools (pg_dump, redis-cli).

Tool	What It Backs Up	Best For
Velero	Kubernetes resources + PV snapshots	Full cluster recovery
pg_dump/pg_basebackup	Database contents only	Database-specific recovery
CronJob + script	Custom backup logic	Flexible, database-aware
CSI snapshot	Volume snapshot only	Volume-level recovery

Velero Volume Snapshots

Velero backup configuration


# Install Velero
# velero install --provider aws --bucket my-backups \
#   --backup-location-config region=us-east-1 \
#   --snapshot-location-config region=us-east-1

# Backup all persistent volumes
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: database-daily
spec:
  schedule: "0 2 * * *"  # 2 AM daily
  template:
    includedNamespaces:
      - databases
    includedResources:
      - persistentvolumeclaims
      - statefulsets
      - services
      - configmaps
      - secrets
    snapshotVolumes: true
    storageLocation: default
    ttl: 720h  # 30 days retention

# Manual backup before migration
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: pre-migration-backup
spec:
  includedNamespaces:
    - databases
  snapshotVolumes: true
  storageLocation: default
  ttl: 168h  # 7 days

Velero snapshots are volume-level — they capture the raw storage state. This is fast but not database-consistent. For database-consistent backups, quiesce the database first or use database-native tools.

Velero snapshots are not application-consistent

Velero snapshots the volume at the block level. If the database is mid-write, the snapshot may be inconsistent. For databases, use pg_dump in a CronJob or use the operator's backup feature instead.

CronJob-Based Backups

PostgreSQL backup CronJob


apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: postgres:16
              command:
                - /bin/bash
                - -c
                - |
                  set -euo pipefail
                  TIMESTAMP=$(date +%Y%m%d_%H%M%S)
                  BACKUP_FILE="/backup/postgres_${TIMESTAMP}.sql.gz"

                  echo "Starting backup: ${TIMESTAMP}"

                  # Run pg_dump and compress
                  pg_dump -h postgres-headless -U postgres mydb \
                    | gzip > "${BACKUP_FILE}"

                  # Verify backup is not empty
                  FILESIZE=$(stat -c%s "${BACKUP_FILE}")
                  if [ "$FILESIZE" -lt 1000 ]; then
                    echo "ERROR: Backup file too small (${FILESIZE} bytes)"
                    exit 1
                  fi

                  echo "Backup complete: ${BACKUP_FILE} (${FILESIZE} bytes)"

                  # Cleanup old backups (keep 7 days)
                  find /backup -name "*.sql.gz" -mtime +7 -delete
              env:
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: postgres-secret
                      key: password
              volumeMounts:
                - name: backup-volume
                  mountPath: /backup
          restartPolicy: OnFailure
          volumes:
            - name: backup-volume
              persistentVolumeClaim:
                claimName: backup-pvc

This CronJob runs pg_dump daily, compresses the output, verifies the backup file size, and cleans up old backups. The backup volume is a separate PVC that persists across job runs.

Offsite Backup Patterns

S3 sync sidecar pattern


apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-offsite
spec:
  schedule: "0 3 * * *"  # Run after database backup
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: sync
              image: amazon/aws-cli
              command:
                - /bin/sh
                - -c
                - |
                  # Sync local backups to S3
                  aws s3 sync /backup s3://offsite-backups/postgres/ \
                    --storage-class STANDARD_IA \
                    --sse AES256

                  # Verify upload
                  aws s3 ls s3://offsite-backups/postgres/ --summarize

                  # Cleanup local backups older than 3 days
                  find /backup -name "*.sql.gz" -mtime +3 -delete
              env:
                - name: AWS_ACCESS_KEY_ID
                  valueFrom:
                    secretKeyRef:
                      name: aws-credentials
                      key: access-key
                - name: AWS_SECRET_ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      name: aws-credentials
                      key: secret-key
              volumeMounts:
                - name: backup-volume
                  mountPath: /backup
                  readOnly: true
          restartPolicy: OnFailure
          volumes:
            - name: backup-volume
              persistentVolumeClaim:
                claimName: backup-pvc

This pattern syncs local backups to S3 for offsite storage. STANDARD_IA (Infrequent Access) reduces storage costs for backups accessed rarely. AES256 encryption at rest protects the data.

Restore Procedures

Restore from backup


# Restore PostgreSQL from backup
apiVersion: batch/v1
kind: Job
metadata:
  name: postgres-restore
spec:
  template:
    spec:
      containers:
        - name: restore
          image: postgres:16
          command:
            - /bin/bash
            - -c
            - |
              set -euo pipefail

              # Find the latest backup
              LATEST=$(ls -t /backup/*.sql.gz | head -1)
              echo "Restoring from: ${LATEST}"

              # Drop and recreate database
              dropdb -h postgres-headless -U postgres mydb
              createdb -h postgres-headless -U postgres mydb

              # Restore from backup
              gunzip -c "${LATEST}" | psql -h postgres-headless -U postgres mydb

              echo "Restore complete"
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
          volumeMounts:
            - name: backup-volume
              mountPath: /backup
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: backup-volume
          persistentVolumeClaim:
            claimName: backup-pvc

Restore is the inverse of backup. The restore job mounts the backup PVC, finds the latest backup, drops and recreates the database, and restores from the backup file. This is a manual operation — do not automate restore.

Testing Backups in Kubernetes

Weekly: restore backup to a temporary namespace and verify data
Monthly: test full disaster recovery (restore to new cluster)
Validate: row counts, table sizes, and query results match production
Automate: run restore tests as CronJobs with alerting on failure
Document: restore procedures and estimated recovery time

3-2-1 backup rule in Kubernetes

3 copies of data: live database + local backup (CronJob PVC) + offsite (S3). 2 storage types: block storage (EBS) + object storage (S3). 1 offsite: different region or cloud provider.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 3 · Build

Databases in Kubernetes