Stage 5 · Platform
Kubernetes Architecture Deep Dive
etcd Deep Dive
Raft, leases, compaction, defragmentation, and disaster recovery.
etcd in Kubernetes
etcd is the single source of truth for all cluster state. Every Kubernetes object — from Pods to CRDs — is stored as a key-value pair. The API server is the only component that directly accesses etcd. All other components (scheduler, controllers, kubelet) go through the API server.
Keys follow the pattern /registry/<resource-type>/<namespace>/<name>. For example, a Pod named nginx in namespace default is stored at /registry/pods/default/nginx. This flat key space enables efficient prefix watches.
Raft Consensus
etcd uses the Raft consensus protocol to replicate data across cluster members. One node is elected leader; all writes go through the leader. The leader replicates entries to followers and waits for a quorum (majority) before acknowledging a write. This ensures consistency even if some nodes fail.
# Check member list
etcdctl member list --write-out=table --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
# Check endpoint health
etcdctl endpoint health --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.keyThe etcdctl tool communicates with etcd using gRPC. All commands require TLS credentials. The --write-out=table flag formats output as a readable table.
Leases and TTLs
etcd supports leases — objects can be associated with a lease that has a TTL (time-to-live). When the lease expires, all associated keys are automatically deleted. Kubernetes uses leases for ephemeral resources like endpoints and leases for leader election.
# Grant a 30-second lease
etcdctl lease grant 30 --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
# Put a key with a lease
etcdctl put /test/key "value" --lease=<lease-id> --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
# Keep the lease alive (renew TTL)
etcdctl lease keep-alive <lease-id> --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.keyLeases are essential for Kubernetes leader election. The controller-manager and scheduler each acquire a lease; only the holder becomes the leader. If the leader dies, its lease expires and a new election occurs.
Compaction and Defragmentation
etcd keeps all historical revisions of keys. Over time, this history grows unbounded. Compaction removes old revisions. After compaction, defragmentation reclaims disk space. Kubernetes runs automatic compaction via the API server's --etcd-compaction-revision-per-hour flag.
# Get current revision
etcdctl endpoint status --write-out=json --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
# Compact to a specific revision
etcdctl compact <revision> --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
# Defragment (reclaims disk space)
etcdctl defrag --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.keyDefragmentation is a blocking operation — the etcd member is unavailable during the process. In HA clusters, defrag one member at a time to avoid quorum loss.
During defragmentation, the etcd member does not respond to requests. Never defrag all members simultaneously — this will cause quorum loss and cluster unavailability. Schedule defrags during maintenance windows.
Disaster Recovery
etcd backup and restore is the last line of defense against data loss. A snapshot captures the entire etcd state. To restore, you stop etcd, restore from snapshot, and restart. This is a critical procedure that every cluster operator must practice.
# Create a snapshot
etcdctl snapshot save /backup/etcd-snapshot.db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
# Verify snapshot
etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table
# Restore (stop etcd first, then restore)
etcdctl snapshot restore /backup/etcd-snapshot.db --data-dir=/var/lib/etcd-restored --name=<member-name> --initial-cluster=<member1>=https://ip1:2380 --initial-advertise-peer-urls=https://<ip1>:2380The restore command writes to a new data directory. Point the etcd static pod manifest to the restored directory. After verification, you can remove the old data directory.
Performance Tuning
etcd performance depends on disk I/O, network latency, and the number of keys. Large clusters (thousands of nodes) need dedicated SSDs for etcd, low-latency networking, and regular compaction to prevent slow reads.
- Use SSDs for etcd data directory — disk I/O is the #1 bottleneck
- Keep etcd members in the same availability zone for low latency
- Limit etcd to 8GB storage with compaction + defrag
- Monitor etcd_server_has_leader and etcd_disk_wal_fsync_duration
- Avoid large objects (pods with hundreds of annotations or labels)
Key metrics: etcd_server_has_leader (1=healthy), etcd_disk_wal_fsync_duration (p99 < 10ms), etcd_network_peer_round_trip_time_seconds (p99 < 50ms), and etcd_debugging_mvcc_db_total_size_in_bytes (keep < 8GB).
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.