SLO Tooling

Prometheus rules, Grafana dashboards, and SLO management platforms — the tools that make it real.

6 min readSRE PlaybookOperate

SLO Tooling Landscape

SLO management ranges from pure Prometheus recording rules to dedicated platforms like Pyrra, Sloth, and OpenSLO. The right choice depends on your scale, team expertise, and how much abstraction you want from raw PromQL.

Start with Prometheus

Before investing in SLO platforms, build your SLOs with Prometheus recording rules. This gives you full control, deep understanding, and no vendor lock-in. Add platforms later if you need abstraction.

Prometheus-Native SLOs

Prometheus recording rules are the foundation of SLO tooling. They pre-compute SLI ratios, burn rates, and error budget values. The result is a set of metrics that your dashboards, alerts, and reports consume.

Prometheus recording rules for SLOs


groups:
  - name: slo_recording
    interval: 30s
    rules:
      # SLI: Availability ratio (non-5xx / total)
      - record: slo:service:availability:ratio
        expr: |
          sum(rate(http_request_duration_seconds_count{code!~"5.."}[{{WINDOW}}]))
          /
          clamp_min(sum(rate(http_request_duration_seconds_count[{{WINDOW}}])), 1)

      # SLI: Latency p99
      - record: slo:service:latency:p99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{code!~"5.."}[{{WINDOW}}])) by (le)
          )

      # Error budget remaining
      - record: slo:service:budget:remaining
        expr: |
          clamp(
            ({{SLO_TARGET}} - slo:service:availability:ratio)
            / (1 - {{SLO_TARGET}}) * -1,
            0, 1
          )

      # Burn rate (1h window)
      - record: slo:service:burn_rate:1h
        expr: |
          (1 - slo:service:availability:ratio)
          / (1 - {{SLO_TARGET}})

Grafana Dashboards

Grafana is the standard visualization layer for Prometheus SLOs. A well-designed SLO dashboard shows current status, historical trends, and error budget consumption at a glance. Use variables for service selection and time range.

Grafana SLO dashboard panel


{
  "title": "Error Budget Remaining",
  "type": "gauge",
  "targets": [
    {
      "expr": "slo:service:budget:remaining",
      "legendFormat": "{{service}}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "thresholds": {
        "steps": [
          { "color": "red", "value": 0 },
          { "color": "yellow", "value": 0.25 },
          { "color": "green", "value": 0.5 }
        ]
      },
      "unit": "percentunit",
      "min": 0,
      "max": 1
    }
  }
}

SLO Platforms

As SLO adoption grows, dedicated platforms provide abstraction, validation, and multi-window burn rate generation out of the box. They simplify SLO management but add a layer between you and your metrics.

Platform	Approach	Best For	Tradeoffs
Pyrra	Kubernetes operator, generates Prometheus rules	K8s-native teams	Requires K8s, less flexible
Sloth	SLO definition files, generates Prometheus rules	Teams wanting abstraction	Extra tool to maintain
OpenSLO	Vendor-neutral SLO specification	Multi-platform teams	Specification only, no engine
Grafana SLO	Grafana-integrated SLO management	Grafana shops	Vendor lock-in to Grafana Cloud

Alertmanager Integration

SLO alerts need proper routing in Alertmanager. Critical burn rate alerts (page) go to PagerDuty. Slow burn alerts (ticket) go to Slack or Jira. Label your alerts with action type so Alertmanager can route them correctly.

Alertmanager routing for SLO alerts


route:
  receiver: default
  routes:
    # SLO critical burn - page immediately
    - match:
        alertname: ErrorBudgetBurnFastPage
      receiver: pagerduty-critical
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h

    # SLO slow burn - create ticket
    - match:
        alertname: ErrorBudgetBurnSlowTicket
      receiver: slack-reliability
      group_wait: 5m
      group_interval: 30m
      repeat_interval: 24h

receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: <pagerduty-key>
        severity: critical

  - name: slack-reliability
    slack_configs:
      - channel: "#reliability"
        title: "SLO Slow Burn Detected"
        text: "Error budget burning at 3x rate"

Label alerts by action type

Add a label like action: page or action: ticket to your SLO alerts. This makes Alertmanager routing explicit and avoids complex regex matching. It also makes it easy to filter alerts in dashboards.

Choosing Your Tools

Small team (under 10 services): Prometheus recording rules + Grafana dashboards.
Medium team (10-50 services): Add Sloth or Pyrra for SLO definition abstraction.
Large team (over 50 services): Consider Grafana Cloud SLO or a custom SLO platform.
Multi-cloud: Use OpenSLO as a vendor-neutral specification layer.
Regulated industry: Use a platform with audit logs and compliance reporting.

Tooling is secondary to practice

The best SLO platform is useless without the organizational practice of reviewing SLOs, acting on error budgets, and adjusting targets. Start with the practice, then invest in tooling to make the practice easier.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

SLOs & Error Budgets