Stage 7 · Master
Advanced Topics & Future of Platform Engineering
Multi-Cluster & Federation
Cluster API, Karmada, Fleet. Workload placement, failover, geographic distribution. Shared control plane vs replicated.
8 min readPlatform Engineering & Developer ExperienceMaster
Why Multi-Cluster?
- Blast radius isolation: Failure in one cluster doesn't affect others
- Geographic distribution: Low latency for global users, data residency
- Scalability: Single cluster has limits (~5000 nodes, 150k pods)
- Upgrade safety: Canary cluster upgrades, rollback fast
- Compliance: Data sovereignty, regulatory boundaries
- Cloud provider diversity: Avoid vendor lock-in, optimize costs
Architecture Patterns
| Pattern | Description | Pros | Cons | Use Case |
|---|---|---|---|---|
| Independent Clusters | Each cluster fully autonomous | Simple, isolated | No shared services, duplication | Small orgs, distinct environments |
| Shared Control Plane | One API server manages multiple clusters (Karmada, Fleet) | Centralized policy, single API | Control plane = SPOF, scaling limits | Org wants unified management |
| Replicated Control Plane | Each cluster has full control plane, sync via git/CRDs | Resilient, autonomous | Eventual consistency, conflict resolution | High availability, multi-region |
| Hub & Spoke | Hub cluster manages spoke clusters via agents | Balance of control/autonomy | Hub = SPOF (mitigate with HA) | Most enterprise platforms |
Cluster API (CAPI)
CAPI brings Kubernetes-style declarative API to cluster lifecycle: create, scale, upgrade, delete clusters. Providers: AWS (CAPA), GCP (CAPG), Azure (CAPZ), vSphere, Docker, Metal3.
CAPI Cluster Definition
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: production-east
namespace: clusters
spec:
clusterNetwork:
services:
cidrBlocks: ["10.96.0.0/12"]
pods:
cidrBlocks: ["192.168.0.0/16"]
serviceDomain: "cluster.local"
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
name: production-east-control-plane
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster
name: production-east
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: production-east-control-plane
namespace: clusters
spec:
replicas: 3
version: v1.28.4
machineTemplate:
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
name: production-east-control-plane
kubeadmConfigSpec:
clusterConfiguration:
apiServer:
extraArgs:
audit-policy-file: /etc/kubernetes/audit-policy.yaml
audit-log-path: /var/log/kubernetes/audit.log
initConfiguration:
nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
joinConfiguration:
nodeRegistration:
kubeletExtraArgs:
cloud-provider: external
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: production-east-workers
namespace: clusters
spec:
clusterName: production-east
replicas: 5
selector:
matchLabels:
cluster.x-k8s.io/cluster-name: production-east
template:
spec:
clusterName: production-east
version: v1.28.4
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: production-east-workers
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
name: production-east-workers
CAPI resources: Cluster, KubeadmControlPlane, MachineDeployment. Declarative cluster lifecycle.
Federation: Karmada, Fleet
| Feature | Karmada | Fleet (Rancher) | Cluster API + ClusterSet |
|---|---|---|---|
| Architecture | Central control plane + agent | Hub + agent per cluster | CAPI + ClusterSet (experimental) |
| Scheduling | Custom scheduler (spread, affinity) | GitOps-based (Fleet) | Native K8s scheduling |
| Multi-Cluster Service | Built-in (ServiceExport/Import) | Via Rancher Service Discovery | Via MCS (Multi-Cluster Services) |
| Policy | PropagationPolicy, OverridePolicy | GitOps (bundles, targets) | ClusterPolicy (Kyverno/OPA) |
| Maturity | CNCF Graduated | CNCF Sandbox | Alpha/Beta |
| Best For | Complex scheduling, multi-cloud | Rancher shops, GitOps-native | CAPI-native orgs |
Karmada PropagationPolicy
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: payment-service-propagation
namespace: payments
spec:
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
name: payment-service
placement:
clusterAffinity:
clusterNames:
- production-east
- production-west
replicaScheduling:
replicaSchedulingType: Divided
replicaDivisionPreference: Weighted
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- production-east
weight: 3
- targetCluster:
clusterNames:
- production-west
weight: 1
# Or dynamic: based on cluster capacity
# replicaSchedulingType: Dynamic
# dynamicWeightPreference:
# resource: cpu
Karmada PropagationPolicy: defines which clusters, how replicas divided (weighted, dynamic).
Workload Placement
- Static: Explicit cluster list or label selector
- Weighted: Distribute replicas by weight (3:1 east:west)
- Dynamic: Schedule based on real-time capacity (CPU, memory, GPU, cost)
- Affinity/Anti-affinity: Spread across zones, avoid same rack
- Custom: Custom scheduler plugin (GPU, cost, latency, compliance)
- Platform API integration:
platform service create --region=auto→ platform chooses optimal cluster
Failover & Disaster Recovery
| Strategy | RTO | RPO | Complexity | Cost |
|---|---|---|---|---|
| Active-Active | Seconds | Near-zero | High | 2x |
| Active-Passive (Warm) | Minutes | Seconds | Medium | 1.5x |
| Active-Passive (Cold) | Hours | Minutes-Hours | Low | 1x |
| Backup/Restore | Hours-Days | Hours | Lowest | Minimal |
Active-Passive with ArgoCD + Karmada
# Primary cluster: active
# Secondary cluster: passive (ArgoCD apps synced but scaled to 0)
# Karmada OverridePolicy for failover
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
name: payment-service-failover
namespace: payments
spec:
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
name: payment-service
overriders:
plaintext:
- path: "/spec/replicas"
operator: "replace"
value: "0" # Scale to 0 in passive cluster
# On failover: update to replicas: 3
---
# Failover runbook (automated via platform API)
# 1. Detect primary cluster unhealthy (health check, ArgoCD app degraded)
# 2. Platform API: POST /failover { service: payment-service, from: prod-east, to: prod-west }
# 3. Karmada: Update OverridePolicy replicas to 3 in prod-west
# 4. ArgoCD: Sync prod-west ArgoCD app (if not auto-sync)
# 5. DNS: Update Global Load Balancer (AWS Global Accelerator, Cloudflare)
# 6. Verify: Health checks pass, traffic flowing
# 6. Notify: Slack, PagerDuty, status page
Active-passive failover: passive cluster scaled to 0. On failover, scale up, sync ArgoCD, update DNS.
Multi-Cluster Networking
| Solution | Approach | Pros | Cons |
|---|---|---|---|
| Submariner | VXLAN/IPSec tunnels between clusters | K8s-native, MCS support | Overlay overhead, MTU issues |
| Istio Multi-Cluster | Shared control plane or replicated | Full mesh, mTLS, traffic management | Complex, resource heavy |
| Cilium Cluster Mesh | eBPF-based, WireGuard/IPSec | Performant, L7 policy, no overlay | Requires Cilium CNI |
| Karmada Service Export/Import | MCS API (ServiceExport/Import) | Standard API, works with any CNI | Limited to L4, no mesh features |
| Cloud Provider (AWS VPC Lattice, GCP PSC) | Cloud-native private connectivity | Managed, scalable, secure | Vendor lock-in, cost |
Start Simple
Don't federate on day one. Start with independent clusters + GitOps. Add federation when you have: multi-region requirements, blast radius concerns, or compliance needs. The platform API should abstract cluster topology from developers.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.