Recording Rules

Precomputing expensive PromQL expressions with rule groups, evaluation intervals, and promtool tests.

7 min readObservability & Monitoring StackOperate

Why Recording Rules?

Recording rules evaluate PromQL expressions at regular intervals and write the results as new time series. They shift computation from query time to write time. This is essential for dashboards with complex queries that would otherwise be slow.

Use recording rules for dashboards

If a dashboard query takes more than a few hundred milliseconds, move it to a recording rule. Dashboard panels should read precomputed values. Alerts should also use recording rules for expensive expressions.

Rule File Syntax

Recording rules live in rule files loaded by Prometheus. Each file contains one or more groups. Each group contains one or more rules. Groups are evaluated independently.

Recording rules file


groups:
  - name: http_metrics
    interval: 15s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by(job) (rate(http_requests_total[5m]))

      - record: job:http_errors:ratio5m
        expr: |
          sum by(job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by(job) (rate(http_requests_total[5m]))

  - name: node_metrics
    interval: 30s
    rules:
      - record: instance:node_cpu_utilization
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

      - record: instance:node_memory_utilization
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

Evaluation Intervals

Each group has an evaluation interval that controls how often its rules run. The interval can differ from the global evaluation_interval. Record the same interval as your scrape interval to keep metrics aligned.

Global and group intervals


global:
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

groups:
  - name: fast_metrics
    interval: 15s    # Match scrape interval
    rules: [...]

  - name: slow_metrics
    interval: 60s    # Computationally expensive queries
    rules: [...]

Naming Recorded Metrics

Recorded metric names must follow Prometheus naming conventions. The name should describe what was computed and which labels were aggregated. Use colons to separate the aggregation level from the metric.

Naming patterns for recorded metrics


# Format: level:metric:operations
job:http_requests:rate5m
job:http_errors:ratio5m
instance:node_cpu_utilization
instance:node_memory_utilization
namespace:container_memory_usage_bytes:sum

Name reflects the aggregation

The recorded metric name encodes what labels were aggregated away. If you sum by job, name it job:http_requests:rate5m. This tells future readers exactly what the metric represents without reading the expression.

Testing with promtool

promtool validates rule files before loading them into Prometheus. It checks syntax, evaluates test cases, and catches common errors. Always test rules before deploying.

Testing recording rules


# Validate syntax
promtool check rules /etc/prometheus/rules/*.yml

# Run unit tests
promtool test rules /etc/prometheus/tests/*.yml

Unit test file


evaluation_interval: 15s

tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", status="200"}'
        values: "0+100x10"
      - series: 'http_requests_total{job="api", status="500"}'
        values: "0+5x10"
    alert_rule_test:
      - eval_time: 5m
        alertname: HighErrorRate
        exp_alerts:
          - labels:
              severity: critical

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

PromQL in Depth