Histograms and Quantiles

Histogram Structure

A Prometheus histogram counts observations into predefined buckets. Each bucket is a counter that tracks how many observations fell below a threshold. The le label defines the upper bound of each bucket.

http_request_duration_seconds_bucket{le="0.01"}     1234
http_request_duration_seconds_bucket{le="0.05"}     5678
http_request_duration_seconds_bucket{le="0.1"}      8901
http_request_duration_seconds_bucket{le="0.5"}      9876
http_request_duration_seconds_bucket{le="1.0"}      9990
http_request_duration_seconds_bucket{le="+Inf"}     10000
http_request_duration_seconds_sum                   4567.89
http_request_duration_seconds_count                 10000

histogram_quantile()

histogram_quantile() estimates a percentile from histogram buckets using linear interpolation. It requires the le label and operates on the _bucket time series. The result is an approximation that depends on bucket granularity.

# 95th percentile latency across all instances
histogram_quantile(
  0.95,
  sum by(le) (rate(http_request_duration_seconds_bucket[5m]))
)

# 99th percentile by endpoint
histogram_quantile(
  0.99,
  sum by(endpoint, le) (rate(http_request_duration_seconds_bucket[5m]))
)

Choosing Bucket Boundaries

Bucket boundaries determine accuracy. If most of your observations fall between two bucket boundaries, the quantile estimate will be imprecise. Choose boundaries that match the expected distribution of your data.

from prometheus_client import Histogram

# Latency histogram with custom buckets
REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration",
    ["endpoint"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0)
)

durationHistogram := promauto.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request duration",
        Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0},
    },
    []string{"endpoint"},
)

Sum and Count Metrics

Every histogram also emits _sum and _count metrics. These give you the total value and total count of observations. Use them to compute averages and ratios without dealing with buckets.

# Average latency
rate(http_request_duration_seconds_sum[5m])
  / rate(http_request_duration_seconds_count[5m])

# Request rate (same as count per second)
rate(http_request_duration_seconds_count[5m])

Native Histograms

Native histograms (introduced in Prometheus 2.53) store bucket counts in a single time series using a logarithmic schema instead of the traditional _bucket suffix. They provide automatic bucket selection and better accuracy for sparse distributions.

Accuracy Considerations

• histogram_quantile returns an approximation, not an exact value.
• Accuracy improves with more buckets and more observations.
• Between-bucket values are linearly interpolated.
• Aggregating quantiles across instances is not meaningful for p95/p99.
• Use sum by(le) aggregation before calling histogram_quantile.
• Prefer server-side histograms over client-side summaries for aggregation.