Monitoring

Starting From an SLO, Not From 'Alert on Everything'

This service's organization_http_requests_total and organization_http_request_duration_seconds metrics, built in the Metrics lesson, could support dozens of plausible alert rules. Writing all of them would guarantee alert fatigue — a 2am page for a 30-second latency blip nobody needed to know about trains the on-call engineer to stop trusting pages at all. This lesson derives exactly two alert rules from one stated SLO: 99.5% of requests succeed (non-5xx) over a rolling 30-day window, and the p99 latency for GET /api/v1/organizations/:id stays under 300ms.

Deriving the Alert Threshold Instead of Guessing a Round Number

A 99.5% success SLO over 30 days permits an error budget of 0.5% of requests — burning that budget at a constant rate would exhaust it exactly at 30 days. Burning it 14.4 times faster than that constant rate exhausts the entire monthly budget in about 1 hour, which is the standard 'fast burn' multiplier used to justify paging immediately rather than waiting for a slow, unremarkable drift. The rule below alerts when the 5-minute error rate implies that fast-burn condition, not on an arbitrarily chosen percentage.

groups:
  - name: organization-service
    rules:
      - alert: OrganizationErrorBudgetFastBurn
        expr: |
          sum(rate(organization_http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(organization_http_requests_total[5m]))
          > (14.4 * 0.005)
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Organization service is burning its 30-day error budget at 14.4x the sustainable rate"
          description: "5xx rate over the last 5 minutes exceeds the fast-burn threshold derived from the 99.5% availability SLO. At this rate, the entire monthly error budget is exhausted in roughly 1 hour."

      - alert: OrganizationReadLatencyHigh
        expr: |
          histogram_quantile(0.99,
            sum(rate(organization_http_request_duration_seconds_bucket{route="/api/v1/organizations/:id",method="GET"}[5m])) by (le)
          ) > 0.3
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "GET /api/v1/organizations/:id p99 latency exceeds 300ms"
          description: "Sustained for 10 minutes — long enough to rule out a brief GC pause or a single slow query, short enough to catch a real regression before the next deploy."

Why Two Different severity Labels, Not a Uniform Page-Everyone Policy

A 300ms p99 latency regression is worth investigating but is rarely an emergency — it does not mean requests are failing, only that they are somewhat slower than target. Routing it as a page would train whoever is on call to treat pages as routine and safely ignorable, which is precisely how a genuine fast-burn error-budget page eventually gets missed or dismissed alongside the noise.

A Deliberately Minimal Dashboard: Four Panels, Not Forty

deploy/observability/organization-dashboard.json defines exactly four panels: request rate by status code, p50/p99 latency by route, the current error-budget burn rate, and pod count from the Kubernetes Deployment lesson's replica count. A dashboard with forty panels covering every metric this service could possibly emit is not more informative during an incident — it is a wall of noise an on-call engineer has to visually search through under time pressure, when they need the answer to one specific question in the first five seconds.

{
  "title": "Organization Service",
  "panels": [
    {
      "title": "Request rate by status",
      "targets": [
        { "expr": "sum(rate(organization_http_requests_total[5m])) by (status)" }
      ]
    },
    {
      "title": "Latency (p50 / p99) by route",
      "targets": [
        { "expr": "histogram_quantile(0.50, sum(rate(organization_http_request_duration_seconds_bucket[5m])) by (le, route))" },
        { "expr": "histogram_quantile(0.99, sum(rate(organization_http_request_duration_seconds_bucket[5m])) by (le, route))" }
      ]
    },
    {
      "title": "Error budget burn rate (5m)",
      "targets": [
        { "expr": "sum(rate(organization_http_requests_total{status=~\"5..\"}[5m])) / sum(rate(organization_http_requests_total[5m]))" }
      ]
    },
    {
      "title": "Ready replica count",
      "targets": [
        { "expr": "kube_deployment_status_replicas_ready{deployment=\"organization\"}" }
      ]
    }
  ]
}

Validating Both Files Before They Ever Reach a Real Prometheus or Grafana

Checking deploy/observability/organization-alerts.yml
  SUCCESS: 2 rules found