Stage 7 · Master
Observability & Developer Experience Metrics
Cost Visibility & Optimization
Showback/chargeback per team. Kubecost, OpenCost. Right-sizing recommendations. Auto-scaling policies. Budget alerts in portal.
Why Cost Visibility?
Platform teams often own the infrastructure bill but lack visibility into which teams/services drive costs. Without cost visibility: teams over-provision, zombie resources accumulate, no incentive to optimize, budget surprises. With cost visibility: teams self-optimize, platform team identifies waste, finance gets accurate showback.
| Without Cost Visibility | With Cost Visibility |
|---|---|
| Teams request max resources 'just in case' | Teams right-size based on actual usage + cost |
| Zombie resources (unused DBs, PVCs, LBs) accumulate | Automated cleanup + owner alerts |
| Finance gets single cloud bill | Finance gets per-team/service breakdown |
| Platform team blamed for cost overruns | Teams own their cost, platform enables optimization |
| No incentive to optimize | Cost dashboards + budgets create accountability |
Showback vs Chargeback
| Aspect | Showback | Chargeback |
|---|---|---|
| Definition | Report costs to teams, no money moves | Teams pay from their budget |
| Complexity | Low: reporting only | High: billing integration, budgets, disputes |
| Adoption | Easier to start | Requires finance partnership |
| Behavior Change | Awareness only | Direct financial incentive |
| Best For | Starting out, culture building | Mature orgs, strict budgets |
| Implementation | Dashboard + monthly email | Billing system integration, cost centers |
Begin with showback: build dashboards, send monthly reports, establish cost culture. Migrate to chargeback when: teams trust data, finance is partner, budgets are established, disputes process exists.
Tools: Kubecost, OpenCost, Cloud Provider
| Tool | Type | Strengths | Limitations |
|---|---|---|---|
| Kubecost | Commercial (free tier) | K8s-native, allocation, optimization, UI, alerts | Cost at scale, limited cloud integration |
| OpenCost | Open Source (CNCF) | Free, vendor-neutral, Prometheus integration | Less polished UI, community support |
| AWS Cost Explorer + CUR | Cloud Native | Native, detailed, integrates with AWS billing | AWS only, complex setup |
| Azure Cost Management | Cloud Native | Native, EA/CSP support, budgets/alerts | Azure only |
| GCP Cloud Billing | Cloud Native | Native, BigQuery export, labels | GCP only |
| Finout / CloudZero / Vantage | SaaS | Multi-cloud, business context, anomaly detection | Cost, vendor lock-in |
# OpenCost deployment (Helm)
# helm repo add opencost https://opencost.github.io/opencost-helm-chart
# helm install opencost opencost/opencost -n opencost --create-namespace
# Prometheus scrape config for OpenCost
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: opencost
namespace: monitoring
spec:
selector:
matchLabels:
app: opencost
endpoints:
- port: metrics
interval: 30s
path: /metrics
# PrometheusRule for cost alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: opencost-alerts
namespace: monitoring
spec:
groups:
- name: opencost
rules:
- alert: TeamCostBudgetExceeded
expr: |
sum by (team) (opencost_allocation_total_cost{team!=""})
>
sum by (team) (opencost_budget_monthly{team!=""}) * 0.8
for: 1h
labels:
severity: warning
team: platform
annotations:
summary: "Team {{ $labels.team }} exceeded 80% of monthly budget"
description: "Current spend: {{ $value | humanizeCurrency }}. Budget: {{ $labels.budget | humanizeCurrency }}."
runbook_url: "https://runbooks.platform.example.com/cost-budget-exceeded"
OpenCost exposes allocation metrics. PrometheusRule alerts when team spend exceeds 80% of budget.
Cost Allocation Model
Allocate shared cluster costs (control plane, ingress, monitoring, logging) to teams. Methods: proportional (by CPU/memory request), equal, or custom weights.
# opencost-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: opencost-config
namespace: opencost
data:
opencost.cfg: |
# Allocation configuration
allocation:
# Shared cost allocation strategy
sharedCostAllocation:
# Proportional to resource requests
cpu: "request"
memory: "request"
# Or equal split
# cpu: "equal"
# memory: "equal"
# Label to use for team allocation
teamLabel: "platform.example.com/team"
# Namespace to team mapping (for namespaces without team label)
namespaceLabels:
platform-system: "platform"
monitoring: "platform"
ingress-nginx: "platform"
argocd: "platform"
# Custom allocation weights (optional)
customAllocations:
- name: "ingress-shared"
label: "platform.example.com/ingress-class"
weight: 0.5
- name: "monitoring-shared"
label: "platform.example.com/monitoring-enabled"
weight: 0.3
# Pricing configuration
pricing:
# Use cloud provider pricing API
provider: "aws" # or gcp, azure, custom
# Or custom pricing sheet
# customPricing:
# cpuHourly: 0.03
# memoryGiBHourly: 0.004
# storageGiBMonthly: 0.10
# loadBalancerHourly: 0.025
# pvHourly: 0.0001
# Budget configuration
budgets:
- team: "payments"
monthlyLimit: 5000
currency: "USD"
alertThresholds: [0.5, 0.75, 0.9, 1.0]
- team: "identity"
monthlyLimit: 3000
currency: "USD"
alertThresholds: [0.5, 0.75, 0.9, 1.0]
- team: "platform"
monthlyLimit: 10000
currency: "USD"
alertThresholds: [0.5, 0.75, 0.9, 1.0]
OpenCost config: team label for allocation, shared cost strategy, custom budgets per team with alert thresholds.
Right-Sizing & Automation
- Vertical Pod Autoscaler (VPA): Recommends/updates resource requests/limits based on actual usage
- Goldilocks: Dashboard for VPA recommendations, visualizes over/under-provisioned
- Kubecost/Goldilocks integration: Cost-aware right-sizing recommendations
- Automated PRs: Bot creates PRs with VPA recommendations for team approval
- Cluster Autoscaler / Karpenter: Node-level right-sizing, spot instance optimization
- KEDA: Event-driven scaling (Kafka, RabbitMQ, Prometheus metrics) for workloads
# VPA for a deployment
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: payment-api-vpa
namespace: payments-payment-api
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: payment-api
updatePolicy:
updateMode: "Auto" # Or "Initial" for recommendations only
resourcePolicy:
containerPolicies:
- containerName: payment-api
minAllowed:
cpu: "50m"
memory: "64Mi"
maxAllowed:
cpu: "2000m"
memory: "4Gi"
controlledResources: ["cpu", "memory"]
controlledValues: RequestsAndLimits
---
# Goldilocks deployment (visualizes VPA recommendations)
apiVersion: apps/v1
kind: Deployment
metadata:
name: goldilocks
namespace: goldilocks
spec:
replicas: 1
selector:
matchLabels:
app: goldilocks
template:
metadata:
labels:
app: goldilocks
spec:
containers:
- name: goldilocks
image: fairwinds/goldilocks:v5.0.0
ports:
- containerPort: 8080
env:
- name: LABEL_SELECTOR
value: "platform.example.com/managed-by=platform-controller"
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- "--disable-vpa-creation"
- "--vpa-namespace=all"
resources:
requests:
cpu: "50m"
memory: "64Mi"
limits:
cpu: "200m"
memory: "256Mi"
VPA automatically adjusts resource requests/limits. Goldilocks provides dashboard to visualize recommendations before applying.
Budget Alerts in Portal
Cost visibility in developer portal: per-service cost, team budget burn rate, optimization recommendations. Alerts: budget thresholds, anomaly detection, zombie resource detection.
// components/CostWidget.tsx
interface CostWidgetProps {
entityRef: string; // e.g., "component:payments/payment-api"
}
export const CostWidget: React.FC<CostWidgetProps> = ({ entityRef }) => {
const { data: costData } = useQuery({
queryKey: ['cost', entityRef],
queryFn: () => fetchCostData(entityRef),
refetchInterval: 5 * 60 * 1000, // 5 min
});
const { data: budgetData } = useQuery({
queryKey: ['budget', entityRef],
queryFn: () => fetchBudgetData(entityRef),
});
if (!costData) return <Skeleton />;
const burnRate = budgetData ? costData.monthly / budgetData.monthlyLimit : 0;
const isOverBudget = burnRate > 1;
const isWarning = burnRate > 0.8;
return (
<Card className={isOverBudget ? 'border-red-500' : isWarning ? 'border-yellow-500' : ''}>
<CardHeader>
<div className="flex items-center justify-between">
<CardTitle>Monthly Cost</CardTitle>
<Badge variant={isOverBudget ? 'destructive' : isWarning ? 'warning' : 'default'}>
{burnRate > 1 ? 'OVER BUDGET' : burnRate > 0.8 ? 'WARNING' : 'ON TRACK'}
</Badge>
</div>
</CardHeader>
<CardContent>
<div className="text-3xl font-bold text-primary">
${costData.monthly.toLocaleString()}
</div>
<div className="text-sm text-muted-foreground">
${costData.daily.toFixed(2)}/day • ${costData.monthly.toLocaleString()}/month projected
</div>
<Progress value={Math.min(burnRate * 100, 100)} className="mt-4 h-2"
className={isOverBudget ? 'bg-red-500' : isWarning ? 'bg-yellow-500' : 'bg-green-500'} />
<div className="flex justify-between text-xs text-muted-foreground mt-1">
<span>$0</span>
<span>${budgetData?.monthlyLimit.toLocaleString() || 'N/A'}/mo</span>
</div>
<div className="mt-4 grid grid-cols-3 gap-4 text-sm">
<div>
<div className="text-muted-foreground">Compute</div>
<div className="font-medium">${costData.breakdown.compute.toLocaleString()}</div>
</div>
<div>
<div className="text-muted-foreground">Storage</div>
<div className="font-medium">${costData.breakdown.storage.toLocaleString()}</div>
</div>
<div>
<div className="text-muted-foreground">Network</div>
<div className="font-medium">${costData.breakdown.network.toLocaleString()}</div>
</div>
</div>
{costData.recommendations.length > 0 && (
<div className="mt-4 p-3 bg-yellow-50 rounded-lg">
<div className="font-medium text-yellow-800 mb-2">💡 Optimization Opportunities</div>
<ul className="text-sm text-yellow-700 space-y-1">
{costData.recommendations.map((rec, i) => (
<li key={i} className="flex items-center gap-2">
<span className="text-yellow-600">{rec.icon}</span>
<span>{rec.description}</span>
<span className="ml-auto text-xs text-yellow-600">
Save ~${rec.monthlySavings}/mo
</span>
</li>
))}
</ul>
</div>
)}
</CardContent>
</Card>
);
};
Portal cost widget shows monthly cost, budget burn rate, breakdown, and optimization recommendations per service.
Optimization Strategies
| Strategy | Description | Savings Potential | Effort |
|---|---|---|---|
| Right-sizing | VPA/Goldilocks recommendations applied | 20-40% | Low (automated PRs) |
| Spot/Preemptible | Batch/workload-tolerant on spot instances | 60-90% | Medium (tolerations, fallback) |
| Cluster Autoscaler/Karpenter | Scale nodes to demand, consolidate | 15-30% | Low (managed) |
| Zombie Resource Cleanup | Auto-delete unused PVCs, LBs, IPs, DBs | 5-15% | Low (automation) |
| Storage Tiering | Move old data to cheaper storage class | 20-50% storage | Medium (policy + migration) |
| Idle Resource Detection | Auto-scale to zero (KEDA) or delete | 10-25% | Medium (KEDA setup) |
| Commitment Discounts | Reserved Instances, Savings Plans, CUDs | 30-60% | High (finance + planning) |
| Multi-AZ Optimization | Reduce cross-AZ traffic, right-size NAT | 5-15% | Medium (architecture) |
Cost optimization isn't a one-time project — it's a platform capability. Build: cost allocation, showback dashboards, budget alerts, right-sizing automation, zombie cleanup, anomaly detection. Teams should see their cost in the same portal where they deploy.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.