Query Debugging

Missing Series

When a query returns no results or fewer series than expected, the most common causes are label mismatches, stale data, or the target not being scraped. Start by verifying the metric exists and check label values.

# Check if the metric exists at all
{__name__="http_requests_total"}

# List all label values for a metric
label_values(http_requests_total, method)

# Check if the target is up
up{job="my-service"}

Many-to-Many Errors

Binary operations between two vectors fail when one series on the left matches multiple series on the right. This is a many-to-many error. It happens when both sides have matching labels that are not unique.

# This fails if method has multiple status values
rate(http_requests_total{method="GET"}[5m])
  / rate(http_request_duration_seconds_count{method="GET"}[5m])

# Fix: use on() to restrict matching labels
rate(http_requests_total{method="GET"}[5m])
  / on(endpoint) rate(http_request_duration_seconds_count{method="GET"}[5m])

NaN Values

NaN values in query results typically come from division by zero or histogram_quantile with insufficient data. They appear as gaps in graphs or broken alert conditions.

# Division by zero produces NaN
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# Fix: filter out zero denominators
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) > 0 or vector(0)

High Cardinality Queries

Queries that return thousands or millions of series are slow and can cause Prometheus to run out of memory. Common causes include high-cardinality labels like user_id, request_id, or trace_id in the query filters.

# Count series per metric
count by(__name__) ({__name__=~".+"})

# Find the most expensive metric
topk(10, count by(__name__) ({__name__=~".+"}))

# Check label cardinality
count by(method) (http_requests_total)
count by(endpoint) (http_requests_total)

Debugging Tools

• Prometheus UI — Use the graph and table views to inspect raw and computed values.
• promtool query — Run queries from the CLI against a running Prometheus instance.
• promtool test rules — Validate alert and recording rule expressions.
• Grafana Explore — Use the Prometheus datasource to build and test queries interactively.
• TSDB status page — Check the highest cardinality metrics at /api/v1/status/tsdb.

# Query from CLI
promtool query instant http://localhost:9090 'rate(http_requests_total[5m])'

# Check TSDB status
curl -s http://localhost:9090/api/v1/status/tsdb | jq .data.highestCardinalityMetricNames