Stage 3 · Build

Databases in Kubernetes

StatefulSets & Persistent Volumes

StatefulSet semantics, PVC provisioning, storage classes, and pod disruption.

7 min readDatabases for SREBuild

Why StatefulSets for Databases?

Databases are stateful — they maintain data across restarts. Deployments are stateless and disposable. StatefulSets provide the ordered, persistent identity that databases require: stable network names, persistent storage, and ordered deployment and scaling.

Feature	Deployment	StatefulSet
Pod naming	Random (myapp-abc123)	Ordered (myapp-0, myapp-1)
Network identity	Ephemeral	Stable DNS names
Storage	Shared or ephemeral	Per-pod persistent volumes
Scaling	Any order	Ordered (0, 1, 2...)
Rolling updates	Any order	Ordered (one at a time)
Deleting	Any order	Reverse order (2, 1, 0)

StatefulSet Semantics

PostgreSQL StatefulSet


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi

The volumeClaimTemplate creates a separate PersistentVolumeClaim for each pod. Pod postgres-0 gets data-postgres-0, pod postgres-1 gets data-postgres-1. These volumes persist across pod restarts.

Persistent Volumes and Claims

PersistentVolumeClaim


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-postgres-0
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi

# Check PVC status
# kubectl get pvc
# NAME                  STATUS   VOLUME                                     CAPACITY
# data-postgres-0       Bound    pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890  100Gi
# data-postgres-1       Bound    pvc-b2c3d4e5-f6a7-8901-bcde-f12345678901  100Gi
# data-postgres-2       Bound    pvc-c3d4e5f6-a7b8-9012-cdef-123456789012  100Gi

PVCs are requests for storage. The PersistentVolume (PV) is the actual storage provisioned by the cluster. The PVC binds to a PV. With StatefulSets, PVCs are automatically created for each pod.

Storage Classes

Storage class definitions


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  fsType: ext4
  encrypted: "true"
reclaimPolicy: Retain  # Keep data when PVC is deleted
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Delete  # Delete data when PVC is deleted
allowVolumeExpansion: true

reclaimPolicy: Retain preserves data when the PVC is deleted — critical for databases. The default Delete policy destroys data permanently. Always set Retain for database storage classes.

Never use reclaimPolicy: Delete for databases

The default reclaimPolicy is Delete, which destroys the underlying storage when the PVC is deleted. For databases, always use Retain to prevent accidental data loss during cluster operations.

Pod Disruption Budgets

PodDisruptionBudget


apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: postgres-pdb
spec:
  minAvailable: 2  # At least 2 pods must be available
  selector:
    matchLabels:
      app: postgres

# Or use maxUnavailable
# maxUnavailable: 1  # At most 1 pod can be unavailable

# PDB protects against voluntary disruptions:
# - kubectl drain for node maintenance
# - Cluster autoscaler removing nodes
# - Pod eviction for resource pressure

# PDB does NOT protect against involuntary disruptions:
# - Node hardware failure
# - OOM kill
# - Network partition

A PDB ensures that cluster operations (node drains, upgrades) do not take down too many database pods simultaneously. minAvailable: 2 means at least 2 of 3 replicas must stay running during voluntary disruptions.

Headless Services

Headless service for StatefulSet


apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None  # Headless — no cluster IP
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432

# DNS records created:
# postgres-headless.default.svc.cluster.local  (headless, returns all pod IPs)
# postgres-0.postgres-headless.default.svc.cluster.local  (individual pod)
# postgres-1.postgres-headless.default.svc.cluster.local
# postgres-2.postgres-headless.default.svc.cluster.local

# Application connects to specific pods:
# postgresql://postgres-0.postgres-headless:5432/mydb
# postgresql://postgres-1.postgres-headless:5432/mydb

A headless service (clusterIP: None) creates individual DNS entries for each pod. This allows clients to connect to specific pods, which is essential for database replication where you need to distinguish primary from replicas.

StatefulSets are necessary but not sufficient

StatefulSets provide stable identity and storage, but they do not handle replication setup, failover, or backup. For production databases, use a database operator that builds on StatefulSets to provide these features.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.