Course outline

01 · Kubernetes Architecture Deep Dive

02 · Workloads & Scheduling

03 · Networking, Storage & Service Mesh

04 · Platform Engineering & Multi-Tenancy

05 · Networking & Service Mesh

06 · Observability & Autoscaling

07 · Configuration & Secrets Management

08 · Stateful Storage Operations

09 · Security, Identity & Admission

10 · Release Packaging with Helm & Kustomize

11 · Operators, CRDs & Controllers

12 · Production Troubleshooting & Operations

Stage 5 · Platform

Stateful Storage Operations

Backup & Restore Drills

Velero, application quiescing, restore namespaces, and recovery point verification.

8 min readKubernetes & Cloud NativePlatform

Backup Strategy

A comprehensive backup strategy covers: cluster state (Kubernetes resources), persistent data (PV snapshots), and configuration (Git repos). Use multiple backup tools and test restores regularly. RPO (Recovery Point Objective) and RTO (Recovery Time Objective) guide backup frequency.

Backup Type	Tool	Frequency	RPO
Cluster resources	Velero	Daily	24 hours
PV snapshots	CSI snapshots	Hourly	1 hour
etcd	etcdctl snapshot	Hourly	1 hour
Git repos	Git remote	Real-time	0 (continuous)

Velero Operations

Velero backs up Kubernetes resources and PV data. It creates tar archives of resources and triggers CSI snapshots for PVs. Velero supports scheduled backups, incremental backups, and cross-cluster migration.

Velero backup schedule


apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: full-cluster-daily
  namespace: velero
spec:
  schedule: "0 2 * * *"
  template:
    includedNamespaces:
    - "*"
    excludedNamespaces:
    - kube-system
    - kube-public
    excludedResources:
    - events
    - events.events.k8s.io
    storageLocation: default
    volumeSnapshotLocations:
    - default
    ttl: 720h
    includedClusterResources: true
---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: production-hourly
  namespace: velero
spec:
  schedule: "0 * * * *"
  template:
    includedNamespaces:
    - production
    storageLocation: default
    volumeSnapshotLocations:
    - default
    ttl: 168h
    labelSelector:
      matchLabels:
        backup: hourly

Full cluster backup runs daily with 30-day retention. Production backup runs hourly with 7-day retention. excludedResources reduces backup size. labelSelector lets you tag resources for hourly backup.

Application Quiescing

Application quiescing ensures data consistency during backup. It flushes buffers, pauses writes, and creates a consistent snapshot. For databases, use Velero hooks to run pre-backup commands (e.g., pg_dump, FLUSH TABLES WITH READ LOCK).

Velero backup hooks for databases


apiVersion: velero.io/v1
kind: Backup
metadata:
  name: db-backup-with-quiesce
spec:
  includedNamespaces:
  - production
  hooks:
    resources:
    - name: postgres-quiesce
      includedNamespaces:
      - production
      labelSelector:
        matchLabels:
          app: postgres
      pre:
      - exec:
          container: postgres
          command:
          - /bin/sh
          - -c
          - "pg_dumpall -U postgres > /tmp/backup.sql"
          onError: Fail
          timeout: 300s
      post:
      - exec:
          container: postgres
          command:
          - /bin/sh
          - -c
          - "rm -f /tmp/backup.sql"
          onError: Continue
          timeout: 60s

pre hooks run before the backup starts. pg_dumpall creates a logical backup of all databases. onError: Fail stops the backup if quiescing fails. post hooks run after the backup completes — cleanup temporary files.

Restore Process

Velero restores resources and PV data from backups. You can restore to the same cluster, a different namespace, or a different cluster. Restore is selective — choose specific resources, namespaces, or labels.

Velero restore operations


# List available backups
velero backup get

# Restore specific backup
velero restore create --from-backup full-cluster-daily-20240115

# Restore to different namespace
velero restore create --from-backup full-cluster-daily-20240115 \
  --namespace-mappings production:production-restored

# Restore specific resources
velero restore create --from-backup full-cluster-daily-20240115 \
  --include-resources deployments,services,configmaps

# Restore with label selector
velero restore create --from-backup full-cluster-daily-20240115 \
  --selector "app=api"

# Check restore status
velero restore get
velero restore describe <restore-name>

namespace-mappings redirects resources to a different namespace. include-resources limits restore to specific resource types. --selector restores only resources matching the label. Always verify restore status before declaring success.

DR Drills

Disaster recovery drills test your backup and restore procedures. Practice quarterly: restore a full cluster, verify data integrity, measure RTO, and document issues. A backup that hasn't been tested is not a backup.

DR drill schedule


apiVersion: batch/v1
kind: CronJob
metadata:
  name: dr-drill-trigger
  namespace: velero
spec:
  schedule: "0 0 1 */3 *"  # Quarterly
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: velero-server
          containers:
          - name: drill
            image: velero/velero:v1.13.0
            command:
            - /bin/sh
            - -c
            - |
              # Trigger restore to drill namespace
              velero restore create dr-drill-$TIMESTAMP \
                --from-backup full-cluster-daily \
                --namespace-mappings production:dr-drill

              # Wait for restore
              velero restore wait dr-drill-$TIMESTAMP

              # Verify pods are running
              kubectl get pods -n dr-drill

              # Run verification tests
              kubectl run verify --image=curlimages/curl --rm -it \
                -- curl http://api.dr-drill:8080/healthz

              # Cleanup
              velero restore delete dr-drill-$TIMESTAMP
          restartPolicy: Never

The DR drill restores production to a drill namespace. It verifies pods are running and services are healthy. This tests the entire restore process without affecting production. Document RTO and any issues.

Recovery Verification

After restore, verify: pods are running, services are reachable, data is intact, and configurations are correct. Use health checks, data validation scripts, and smoke tests.

Restore verification checklist


# 1. Check pod status
kubectl get pods -n production-restored

# 2. Check service endpoints
kubectl get endpoints -n production-restored

# 3. Verify data integrity
kubectl exec -it postgres-0 -n production-restored -- \
  psql -U postgres -c "SELECT count(*) FROM important_table"

# 4. Run smoke tests
kubectl run smoke-test --image=curlimages/curl --rm -it -- \
  curl -s http://api-service:8080/healthz

# 5. Check PVC binding
kubectl get pvc -n production-restored

# 6. Verify ConfigMaps and Secrets
kubectl get configmap -n production-restored
kubectl get secret -n production-restored

Systematic verification ensures the restore is complete and functional. Check each layer: pods, services, data, configuration, storage. Document any discrepancies and fix them before declaring DR success.

Automate DR verification

Create a verification script that runs after every restore. It should check pod health, service endpoints, data integrity, and configuration. Automate this in your CI/CD pipeline to ensure DR procedures are always tested.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.