Prometheus Python Client

Instrument applications, expose metrics, and build custom exporters.

7 min readPython for DevOpsMaster

Prometheus Data Model

Prometheus stores time series data as metric names with label key-value pairs. Each unique combination of name and labels is a separate time series. Understanding this model is essential for writing useful metrics.

Prometheus metric naming


# Metric format: name{label1="value1", label2="value2"} value timestamp
# Example time series:
# http_requests_total{method="GET", status="200"} 1234 1705312000
# http_requests_total{method="POST", status="500"} 12 1705312000
# http_request_duration_seconds{method="GET", quantile="0.99"} 0.25 1705312000

# Naming conventions:
# _total suffix for counters
# _seconds suffix for durations
# _bytes suffix for sizes
# _info suffix for info metrics
# Use snake_case, not camelCase

Prometheus metrics are time series. Labels create separate series for each unique value. High-cardinality labels (user IDs, request IDs) create millions of series and crash Prometheus.

Metric Types

Type	Description	Use Case	Example
Counter	Monotonically increasing value	Request count, error count	http_requests_total
Gauge	Value that can go up or down	Temperature, queue depth, connections	active_connections
Histogram	Value distribution with buckets	Request duration, response size	http_request_duration_seconds
Summary	Quantiles calculated on client	Request latency percentiles	http_request_duration_seconds

Instrumenting Applications

Instrument a FastAPI application


from prometheus_client import Counter, Histogram, Gauge, Info
from fastapi import FastAPI, Request
import time

# Define metrics
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "endpoint", "status"],
)

REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration in seconds",
    ["method", "endpoint"],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
)

ACTIVE_CONNECTIONS = Gauge(
    "active_connections",
    "Number of active connections",
)

APP_INFO = Info("app", "Application information")
APP_INFO.info({"version": "1.2.3", "environment": "production"})

app = FastAPI()

@app.middleware("http")
async def metrics_middleware(request: Request, call_next):
    ACTIVE_CONNECTIONS.inc()
    start = time.perf_counter()

    response = await call_next(request)

    duration = time.perf_counter() - start
    REQUEST_COUNT.labels(
        method=request.method,
        endpoint=request.url.path,
        status=response.status_code,
    ).inc()
    REQUEST_DURATION.labels(
        method=request.method,
        endpoint=request.url.path,
    ).observe(duration)
    ACTIVE_CONNECTIONS.dec()

    return response

Instrument at the middleware level to capture all requests automatically. Use labels for method, endpoint, and status code. Histograms let you calculate percentiles in PromQL.

Custom Exporters

Build a custom Prometheus exporter


from prometheus_client import Gauge, start_http_server
import subprocess
import json
import time

# Define metrics for a custom system
DISK_USAGE = Gauge(
    "disk_usage_percent",
    "Disk usage percentage",
    ["mountpoint"],
)

CONTAINER_RESTARTS = Gauge(
    "container_restart_count",
    "Number of container restarts",
    ["container", "namespace"],
)

def collect_disk_metrics():
    """Collect disk usage metrics."""
    result = subprocess.run(
        ["df", "-h", "--output=target,pcent"],
        capture_output=True, text=True,
    )
    for line in result.stdout.strip().split("\n")[1:]:
        parts = line.split()
        if len(parts) == 2 and parts[1].endswith("%"):
            mountpoint = parts[0]
            usage = int(parts[1].rstrip("%"))
            DISK_USAGE.labels(mountpoint=mountpoint).set(usage)

def collect_container_metrics():
    """Collect container restart metrics from Docker."""
    result = subprocess.run(
        ["docker", "ps", "--format", "{{.Names}}|{{.Status}}"],
        capture_output=True, text=True,
    )
    for line in result.stdout.strip().split("\n"):
        if "|" in line:
            name, status = line.split("|", 1)
            # Parse restart count from status
            CONTAINER_RESTARTS.labels(container=name, namespace="docker").set(0)

# Start Prometheus metrics server
start_http_server(8000)
print("Metrics server started on :8000")

# Collect metrics every 30 seconds
while True:
    collect_disk_metrics()
    collect_container_metrics()
    time.sleep(30)

Custom exporters collect metrics from systems that do not natively support Prometheus. Start a metrics HTTP server and scrape it with Prometheus. Collect metrics periodically in a loop.

Exposing Metrics

Expose metrics from FastAPI


from fastapi import FastAPI, Response
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST

app = FastAPI()

@app.get("/metrics")
def metrics():
    """Expose Prometheus metrics."""
    return Response(
        content=generate_latest(),
        media_type=CONTENT_TYPE_LATEST,
    )

# Or use prometheus_fastapi_instrumentator for automatic instrumentation
# pip install prometheus-fastapi-instrumentator
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()
Instrumentator().instrument(app).expose(app, endpoint="/metrics")

The /metrics endpoint returns metrics in Prometheus format. prometheus_fastapi_instrumentator handles instrumentation automatically — just install and configure it.

Best Practices

Use Counter for things that only increase (requests, errors)
Use Gauge for things that fluctuate (connections, queue depth)
Use Histogram for distributions (latency, response size)
Avoid high-cardinality labels (user IDs, request IDs, UUIDs)
Use _seconds suffix for duration metrics
Keep metric names short but descriptive
Use labels sparingly — each label combination is a separate time series
Initialize counters to 0 if you need zero values in graphs

High cardinality kills Prometheus

If you label metrics with user IDs, request IDs, or timestamps, you create millions of time series. Prometheus will run out of memory. Keep label values to a small, bounded set.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 7 · Master

Observability & Incident Tools