Indexing Past Incidents

Turning years of postmortems into an answerable knowledge base.

7 min readAI for SRE WorkflowsMaster

Incident Data Sources

Your incident history is one of the most valuable knowledge bases for an AI assistant. It contains real failures, real root causes, and real remediation steps. The challenge is turning unstructured incident data into searchable, structured knowledge.

Postmortem documents — structured reports with timeline, root cause, and action items.
Incident channel transcripts — raw chat messages from Slack or Teams during an incident.
PagerDuty incident metadata — severity, timestamps, who was paged, resolution time.
Change logs — deployments, config changes, and feature flags around the time of incidents.

Extracting Structured Info

Extract incident metadata with an LLM


import openai
import json

def extract_incident_metadata(postmortem_text):
    """Use an LLM to extract structured metadata from a postmortem."""
    response = openai.ChatCompletion.create(
        model="gpt-4",
        temperature=0.0,
        messages=[
            {
                "role": "system",
                "content": """Extract structured metadata from this postmortem.
Return JSON with:
- services_affected: list of service names
- root_cause_category: one of [code_bug, config_change, capacity, dependency, human_error, unknown]
- severity: P1/P2/P3/P4
- duration_minutes: estimated duration
- key_findings: list of 2-3 key findings
- action_items: list of action items"""
            },
            {"role": "user", "content": postmortem_text}
        ]
    )
    return json.loads(response["choices"][0]["message"]["content"])

Using an LLM to extract structured metadata from postmortems creates filterable fields for your RAG system.

Chunking Incident Reports

Postmortems have a natural structure — timeline, root cause, impact, action items. Chunk along these boundaries so each chunk is self-contained and useful.

Chunk a postmortem by section


def chunk_postmortem(postmortem_text, metadata):
    """Chunk a postmortem by its natural sections."""
    sections = {
        "summary": extract_section(postmortem_text, "Summary"),
        "timeline": extract_section(postmortem_text, "Timeline"),
        "root_cause": extract_section(postmortem_text, "Root Cause"),
        "resolution": extract_section(postmortem_text, "Resolution"),
        "action_items": extract_section(postmortem_text, "Action Items"),
    }

    chunks = []
    for section_name, content in sections.items():
        if content:
            chunks.append({
                "text": content,
                "metadata": {
                    **metadata,
                    "section": section_name,
                    "doc_type": "postmortem",
                }
            })
    return chunks

# Example
chunks = chunk_postmortem(
    open("postmortem-2024-01-15.md").read(),
    metadata={
        "incident_id": "INC-2024-0015",
        "services": ["redis", "checkout"],
        "severity": "P1",
        "date": "2024-01-15",
    }
)

Each section becomes a separate chunk with shared metadata. This allows targeted retrieval — a query about resolution steps only retrieves resolution chunks.

The Indexing Pipeline

Automated incident indexing


def index_incidents(incident_dir):
    """Index all postmortems in a directory."""
    for filename in os.listdir(incident_dir):
        if not filename.endswith(".md"):
            continue

        filepath = os.path.join(incident_dir, filename)
        text = open(filepath).read()

        # Extract metadata
        metadata = extract_incident_metadata(text)

        # Chunk by section
        chunks = chunk_postmortem(text, metadata)

        # Embed and store
        for chunk in chunks:
            embedding = get_embedding(chunk["text"])
            vector_db.insert(
                text=chunk["text"],
                embedding=embedding,
                metadata=chunk["metadata"]
            )

        print(f"Indexed {filename}: {len(chunks)} chunks")

# Run monthly or on-demand
index_incidents("/data/postmortems/")

Run this pipeline periodically to keep your incident knowledge base up to date. New postmortems are indexed automatically.

Keeping the Index Fresh

Trigger re-indexing when postmortems are updated with new action items.
Remove chunks from incidents that have been fully remediated and archived.
Version your index schema so you can migrate when you change chunking strategies.
Monitor retrieval quality over time as new incidents are added.

Start with recent incidents

Begin indexing with your most recent 50 to 100 postmortems. They are most relevant and highest quality. Expand historically as you validate retrieval quality.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 7 · Master

RAG Over Your Ops Knowledge