Stage 7 · Master
AIOps: Detection & Triage
Log Clustering & Summarization
Collapsing millions of log lines into the handful that matter.
The Log Volume Problem
During an incident, log volume explodes. A service restart produces thousands of error messages. A database failure cascades through every service that depends on it. An engineer cannot read 50,000 log lines. Log clustering groups similar lines together and summarizes the clusters.
Log Template Extraction
Most log lines follow templates. A line like Connection to Redis at 10.0.1.5:6379 failed is a template with a variable IP address. Extracting the template lets you group millions of lines into hundreds of unique patterns.
import re
def extract_template(log_line):
"""Replace variable parts of a log line with placeholders."""
# Replace IP addresses
line = re.sub(r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', '<IP>', log_line)
# Replace UUIDs
line = re.sub(r'[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '<UUID>', line)
# Replace timestamps
line = re.sub(r'\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}', '<TIMESTAMP>', line)
# Replace port numbers
line = re.sub(r':\d{4,5}', ':<PORT>', line)
# Replace numbers
line = re.sub(r'\b\d+\b', '<N>', line)
return line
# Example
line = "Connection to Redis at 10.0.1.5:6379 failed after 3000ms"
template = extract_template(line)
print(template) # "Connection to Redis at <IP>:<PORT> failed after <N>ms"This simple approach works for many log formats. For more complex logs, use specialized tools like Drain or LenMa.
Clustering Log Lines
from collections import Counter
def cluster_logs(log_lines):
"""Group log lines by their template."""
templates = Counter()
template_examples = {}
for line in log_lines:
template = extract_template(line)
templates[template] += 1
if template not in template_examples:
template_examples[template] = line
# Sort by frequency
ranked = sorted(templates.items(), key=lambda x: x[1], reverse=True)
return [
{
"template": template,
"count": count,
"example": template_examples[template],
"percentage": count / len(log_lines) * 100
}
for template, count in ranked
]
# Usage
log_lines = read_log_file("/var/log/app.log")
clusters = cluster_logs(log_lines)
for c in clusters[:10]:
print(f"[{c['count']:>6}x {c['percentage']:5.1f}%] {c['template']}")This reveals the dominant log patterns. If 40% of your logs are the same template, that template is likely related to the incident.
LLM Summarization
def summarize_log_clusters(clusters, context=""):
"""Use an LLM to summarize the most important log patterns."""
cluster_text = "\n".join([
f"- [{c['count']}x] {c['template']} (example: {c['example']})"
for c in clusters[:20]
])
response = openai.ChatCompletion.create(
model="gpt-4",
temperature=0.0,
messages=[
{
"role": "system",
"content": "Summarize these log patterns. Identify the most likely root cause and which patterns are symptoms vs causes."
},
{
"role": "user",
"content": f"Context: {context}\n\nLog patterns:\n{cluster_text}"
}
]
)
return response["choices"][0]["message"]["content"]The LLM can distinguish root cause patterns from symptom patterns based on frequency, timing, and service context.
Processing Live Logs
from collections import deque
import time
class LiveLogAggregator:
def __init__(self, window_seconds=60):
self.window = window_seconds
self.buffer = deque()
self.templates = Counter()
def ingest(self, log_line):
"""Add a log line and update template counts."""
now = time.time()
template = extract_template(log_line)
self.buffer.append((now, template))
self.templates[template] += 1
self._evict_old(now)
def _evict_old(self, now):
"""Remove entries outside the window."""
while self.buffer and self.buffer[0][0] < now - self.window:
_, old_template = self.buffer.popleft()
self.templates[old_template] -= 1
if self.templates[old_template] == 0:
del self.templates[old_template]
def get_top_patterns(self, n=10):
"""Get the most frequent patterns in the current window."""
return self.templates.most_common(n)A sliding window aggregator gives you real-time visibility into which log patterns are most frequent right now.
The most useful signal is not the absolute count but the rate of change. A template going from 10 to 1000 per minute is an anomaly even if other templates have higher absolute counts.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.