Retention and Redaction

Retention Strategies

Log retention determines how long logs are kept before deletion. Different log types require different retention periods. Application logs might be kept for 7 days, security logs for 90 days, and audit logs for a year.

Sampling Logs

Sampling keeps a percentage of logs for high-volume streams. This reduces storage costs while maintaining visibility. Sample error logs at 100% and debug logs at 10% to balance cost with diagnostic value.

pipeline_stages:
  - match:
      selector: '{level="debug"}'
      stages:
        - sample:
            value: 0.1
            drop: true
  - match:
      selector: '{level="error"}'
      stages:
        - sample:
            value: 1.0
            drop: false

PII Redaction

PII (Personally Identifiable Information) must be removed from logs before storage. Emails, phone numbers, credit card numbers, and social security numbers should be redacted at the log shipping layer.

pipeline_stages:
  - regex:
      expression: '(?P<email>[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})'
  - labels:
      email:
  - output:
      source: output
      template: '{{ .Value | replaceRE "[a-zA-Z0-9._%+-]+@" "***@" }}'

[FILTER]
    Name    parser
    Match   *
    Key_Name log
    Parser  json

[FILTER]
    Name    modify
    Match   *
    Remove  credit_card
    Remove  ssn

Drop Filters

Drop filters completely remove log entries before they reach storage. Use them to eliminate noisy health check logs, debug messages in production, or any logs that provide no value.

pipeline_stages:
  - match:
      selector: '{path="/health"}'
      stages:
        - drop:
            expression: ".*"
            older_than: 0s

  - match:
      selector: '{level="debug"}'
      stages:
        - drop:
            expression: ".*"
            older_than: 0s

Storage Tiering

Move older logs to cheaper storage tiers over time. Hot storage (SSD) for recent logs, warm storage (HDD) for older logs, and cold storage (object storage) for archived logs. This reduces costs while maintaining access.