Stage 6 · Operate

Logs & Traces

promtail Pipelines

Parsing container logs with promtail stages, relabeling, JSON extraction, and tenant labels.

7 min readObservability & Monitoring StackOperate

promtail Architecture

promtail is a log shipping agent designed for Loki. It runs on every node, tails log files, attaches labels, and pushes logs to Loki. It supports multiple discovery mechanisms and processing pipelines for parsing and enriching logs.

promtail data flow


Log Files -> Discovery -> Pipelines -> Labels -> Push -> Loki

Scrape Configs

promtail uses scrape configs similar to Prometheus. It supports static configs, Kubernetes pod discovery, and file-based discovery. Each config defines how to find logs, which labels to apply, and which pipeline to use.

promtail scrape config


scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
    pipeline_stages:
      - cri: {}
      - json:
          expressions:
            level: level
            msg: message

Pipeline Stages

Pipeline stages process log lines sequentially. Each stage transforms the log entry, extracts labels, or modifies timestamps. Stages include regex, json, logfmt, template, and output stages.

Multi-stage pipeline


pipeline_stages:
  - cri: {}
  - regex:
      expression: '^(?P<level>\w+) (?P<ts>.+) (?P<msg>.+)$'
  - labels:
      level:
  - timestamp:
      source: ts
      format: RFC3339Nano
  - output:
      source: msg

JSON Parsing

JSON extraction pulls fields from structured log lines. The json stage parses the entire line as JSON and extracts values into the log entry. Extracted values can become labels or be used in subsequent stages.

JSON extraction pipeline


pipeline_stages:
  - json:
      expressions:
        level: level
        msg: message
        trace_id: trace_id
        user_id: user_id
  - labels:
      level:
  - output:
      source: msg
  - drop:
      expression: ".*"
      older_than: 24h

Do not extract high-cardinality fields as labels

Fields like user_id, request_id, or trace_id should be extracted as output or used in processing, but never as labels. High-cardinality labels create too many streams and degrade Loki performance.

Relabeling

promtail uses relabel_configs to transform labels before shipping. This is similar to Prometheus relabeling. Use it to add, remove, or modify labels based on file paths, Kubernetes metadata, or other source information.

Relabeling examples


relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_app]
    target_label: app

  - source_labels: [__meta_kubernetes_namespace]
    regex: "(.*)"
    target_label: namespace

  - target_label: environment
    replacement: "production"

Multi-Tenancy

Loki uses tenant IDs to isolate data between teams or environments. promtail sets the X-Scope-OrgID header on push requests. This ensures each tenant can only query their own logs.

Multi-tenant config


clients:
  - url: http://loki:3100/loki/api/v1/push
    tenant_id: "team-platform"

# Or use a label as tenant ID
pipeline_stages:
  - labels:
      team:
  - tenant:
      source: team

Use Kubernetes labels for tenant assignment

In Kubernetes, assign tenant IDs using pod annotations or namespace labels. This automates multi-tenancy without changing application code. Each team's logs stay isolated automatically.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.