Log Aggregation Patterns

Shipping logs with Fluent Bit, Vector, Logstash, promtail, and Kubernetes sidecars.

7 min readObservability & Monitoring StackOperate

Aggregation Architectures

Log aggregation follows a pipeline pattern: logs are collected at the source, processed (parsed, enriched, filtered), and shipped to a central store. The agent runs on every node or as a sidecar. The aggregator runs as a dedicated service.

Log pipeline flow


Applications -> Agent -> Aggregator -> Backend
   (stdout)    (node)   (cluster)    (Loki/ES)

Fluent Bit

Fluent Bit is a lightweight log processor and shipper. It runs on every node with minimal resource usage. It supports parsing, filtering, and multiple output destinations including Loki, Elasticsearch, and S3.

Fluent Bit config for Loki


[SERVICE]
    Flush        5
    Log_Level    info

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Tag               kube.*

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443

[OUTPUT]
    Name                loki
    Match               *
    Url                 http://loki:3100/loki/api/v1/push
    Labels              job=fluent-bit
    Label_Keys          $kubernetes['pod_name'],$kubernetes['namespace']

Vector

Vector is a high-performance observability data pipeline. It handles logs, metrics, and traces. Vector uses a source-transform-sink model and supports advanced processing with VRL (Vector Remap Language).

Vector config


[sources.logs]
  type = "file"
  include = ["/var/log/containers/*.log"]

[transforms.parse]
  type = "remap"
  inputs = ["logs"]
  source = '''
    . = parse_json!(.message)
    .timestamp = now()
  '''

[sinks.loki]
  type = "loki"
  inputs = ["parse"]
  endpoint = "http://loki:3100"
  labels.job = "vector"
  labels.service = .service

Kubernetes Patterns

In Kubernetes, the most common pattern is a DaemonSet running one log agent per node. The agent reads container logs from /var/log/containers/ and enriches them with Kubernetes metadata. Sidecar patterns are used for special cases.

DaemonSet log agent


apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
spec:
  selector:
    matchLabels:
      app: log-agent
  template:
    spec:
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:latest
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: containers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: containers
          hostPath:
            path: /var/lib/docker/containers

Tool Comparison

Tool	Language	Performance	Best For
Fluent Bit	C	Very fast	Kubernetes, resource-constrained
Vector	Rust	Fast	Advanced processing, multi-signal
promtail	Go	Fast	Loki-native integration
Logstash	JVM	Moderate	ELK stack, complex parsing

Use Fluent Bit for Kubernetes

Fluent Bit is the standard choice for Kubernetes log aggregation. It is lightweight, fast, and has excellent Kubernetes integration. Use Vector if you need advanced processing or multi-signal pipelines.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

Logging Operations