Scrape Configuration

Scrape Config Structure

The scrape_configs section in prometheus.yml defines how Prometheus discovers and scrapes targets. Each entry is a job that groups targets sharing the same collection configuration.

scrape_configs:
  - job_name: "node-exporter"
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets:
          - "10.0.0.1:9100"
          - "10.0.0.2:9100"
        labels:
          environment: "production"

Intervals and Timeouts

Scrape interval controls how often Prometheus pulls metrics from a target. Timeout controls how long Prometheus waits for a response. Both have global defaults that can be overridden per job.

Static Configs

Static configs list targets directly in the configuration file. This is useful for infrastructure that does not change frequently, such as databases, message queues, or dedicated monitoring endpoints.

static_configs:
  - targets:
      - "postgres-primary:5432"
      - "postgres-replica:5432"
    labels:
      team: "database"
      environment: "production"

Relabel Configs

Relabel configs modify labels before a target is scraped. They run on the __address__ label and can add, remove, or transform labels. This is the primary mechanism for enriching targets with metadata.

relabel_configs:
  - source_labels: [__address__]
    regex: "(.+):\d+"
    target_label: instance
    replacement: "$1"

  - source_labels: [__meta_kubernetes_pod_label_team]
    target_label: team

Metric Relabeling

Metric relabel configs run after metrics are scraped but before they are stored. They can drop metrics, rename labels, or modify label values. This is useful for filtering noisy metrics or fixing label inconsistencies.

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "go_.*"
    action: drop

  - source_labels: [le]
    target_label: bucket
    replacement: "$1"

Best Practices

• Use consistent scrape intervals across jobs to simplify PromQL queries.
• Set scrape_timeout to 80% of scrape_interval to leave headroom.
• Add job and environment labels to every target for consistent filtering.
• Use file_sd_config for dynamic environments instead of static_configs.
• Monitor Prometheus self-monitoring metrics like up and scrape_duration_seconds.