Stage 6 · Operate

Capacity & Performance Engineering

USE & RED Methods

Utilization, saturation, errors, rate, duration, and dashboards that expose bottlenecks.

6 min readSRE PlaybookOperate

Why Structured Methods?

Without a systematic approach, you miss important metrics. The USE and RED methods provide frameworks for ensuring you measure everything that matters. USE is for infrastructure. RED is for services.

USE for infrastructure, RED for services

USE measures infrastructure health: Utilization, Saturation, Errors. RED measures service health: Rate, Errors, Duration. Use both to get complete visibility.

USE Method

The USE method (Utilization, Saturation, Errors) is for every resource in your infrastructure. For each resource, measure these three signals. If any signal is high, you have a bottleneck.

USE method framework


use_method:
  for_each_resource:
    cpu:
      utilization: "Percentage of CPU time used"
      saturation: "Run queue length, load average"
      errors: "Hardware errors, corrected errors"

    memory:
      utilization: "Percentage of memory used"
      saturation: "Swap usage, OOM kills"
      errors: "Correctable memory errors"

    disk:
      utilization: "Percentage of disk bandwidth used"
      saturation: "I/O queue length"
      errors: "I/O errors, bad sectors"

    network:
      utilization: "Percentage of network bandwidth used"
      saturation: "Packet drop rate, buffer overflow"
      errors: "Packet errors, CRC errors"

    container:
      utilization: "CPU/memory usage vs limits"
      saturation: "Throttling, OOM kills"
      errors: "Container restarts, crashes"

RED Method

The RED method (Rate, Errors, Duration) is for services. For each service, measure these three signals. They map directly to user experience: how many requests, how many fail, and how long they take.

RED method framework


red_method:
  for_each_service:
    rate:
      description: "Requests per second"
      metric: "rate(http_requests_total[5m])"
      use: "Traffic volume, scaling decisions"

    errors:
      description: "Errors per second"
      metric: "rate(http_requests_total{code=~'5..'}[5m])"
      use: "Error rate, SLO monitoring"

    duration:
      description: "Request latency distribution"
      metric: "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))"
      use: "User experience, SLO monitoring"

  derived_metrics:
    error_ratio: "errors / rate"
    latency_p99: "histogram_quantile(0.99, duration)"
    latency_p95: "histogram_quantile(0.95, duration)"
    throughput: "rate"

USE Examples

USE metrics for Kubernetes nodes


use_metrics:
  cpu:
    utilization: |
      1 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))
    saturation: |
      avg(node_load15) by (instance)
    errors: |
      sum(rate(node_cpu_errors_total[5m])) by (instance)

  memory:
    utilization: |
      1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
    saturation: |
      node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes
    errors: |
      sum(rate(node_edac_correctable_errors_total[5m])) by (instance)

  disk:
    utilization: |
      rate(node_disk_io_time_seconds_total[5m])
    saturation: |
      avg(node_disk_io_time_weighted_seconds_total) by (instance)
    errors: |
      sum(rate(node_disk_errors_total[5m])) by (instance)

  network:
    utilization: |
      rate(node_network_receive_bytes_total[5m]) * 8 / node_network_speed_bytes
    saturation: |
      rate(node_network_receive_drop_total[5m])
    errors: |
      rate(node_network_receive_errs_total[5m])

RED Examples

RED metrics for microservices


red_metrics:
  rate: |
    sum(rate(http_requests_total[5m])) by (service)

  errors: |
    sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)

  duration_p99: |
    histogram_quantile(0.99,
      sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
    )

  duration_p95: |
    histogram_quantile(0.95,
      sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
    )

  error_ratio: |
    sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)
    /
    sum(rate(http_requests_total[5m])) by (service)

  slo_availability: |
    1 - (
      sum(rate(http_requests_total{code=~"5.."}[30d])) by (service)
      /
      clamp_min(sum(rate(http_requests_total[30d])) by (service), 1)
    )

Building Dashboards

Build dashboards using USE and RED methods. Each service gets a RED dashboard. Each resource gets a USE dashboard. This systematic approach ensures complete coverage without gaps.

Dashboard structure using USE and RED


dashboard_structure:
  infrastructure_dashboard:
    title: "Infrastructure Health — USE Method"
    sections:
      - title: "CPU"
        panels:
          - "CPU Utilization by Node"
          - "CPU Saturation (Load Average)"
          - "CPU Errors"

      - title: "Memory"
        panels:
          - "Memory Utilization by Node"
          - "Memory Saturation (Swap Usage)"
          - "OOM Kill Count"

      - title: "Disk"
        panels:
          - "Disk Utilization by Node"
          - "Disk Saturation (I/O Queue)"
          - "Disk Errors"

      - title: "Network"
        panels:
          - "Network Utilization by Node"
          - "Network Saturation (Drops)"
          - "Network Errors"

  service_dashboard:
    title: "Service Health — RED Method"
    sections:
      - title: "Request Rate"
        panels:
          - "Requests per Second by Service"
          - "Request Rate Trend (7 days)"

      - title: "Errors"
        panels:
          - "Error Rate by Service"
          - "Errors per Second"
          - "Error Rate Trend (7 days)"

      - title: "Duration"
        panels:
          - "P99 Latency by Service"
          - "P95 Latency by Service"
          - "Latency Distribution"

Start with RED, add USE later

If you are building dashboards from scratch, start with RED for your services. It directly measures user experience. Add USE dashboards for infrastructure later. RED gives you the most value fastest.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.