Stage 7 · Master

RAG Over Your Ops Knowledge

Evaluating Retrieval Quality

Recall, groundedness, and catching confidently-wrong answers.

7 min readAI for SRE WorkflowsMaster

Why Evaluate RAG?

A RAG system can fail in two ways: bad retrieval (the right documents were not found) or bad generation (the model ignored or misinterpreted the retrieved documents). You need to measure both independently.

Separate retrieval quality from generation quality

If the answer is wrong, is it because retrieval failed or because the model failed? Measuring these independently tells you where to focus your improvement efforts.

Retrieval Metrics

Retrieval metrics measure whether the right documents were returned. These are objective and can be computed automatically against a labeled dataset.

Retrieval evaluation metrics


def evaluate_retrieval(eval_dataset):
    """Evaluate retrieval quality against labeled data."""
    results = {"recall@5": [], "precision@3": [], "mrr": []}

    for query, relevant_doc_ids in eval_dataset:
        retrieved = retriever.retrieve(query, top_k=5)
        retrieved_ids = [r.id for r in retrieved]

        # Recall@5: Did we find all relevant docs in top 5?
        relevant_found = len(set(retrieved_ids) & set(relevant_doc_ids))
        recall = relevant_found / len(relevant_doc_ids)
        results["recall@5"].append(recall)

        # Precision@3: Of top 3, how many were relevant?
        top3_ids = retrieved_ids[:3]
        precision = len(set(top3_ids) & set(relevant_doc_ids)) / 3
        results["precision@3"].append(precision)

        # MRR: Rank of first relevant result
        for i, doc_id in enumerate(retrieved_ids):
            if doc_id in relevant_doc_ids:
                results["mrr"].append(1 / (i + 1))
                break

    return {k: sum(v)/len(v) for k, v in results.items()}

# Example output: {"recall@5": 0.87, "precision@3": 0.73, "mrr": 0.82}

Recall measures coverage. Precision measures relevance. MRR measures how quickly the user finds a relevant result.

Generation Metrics

Generation metrics measure whether the model produced a good answer given the retrieved context. These often require LLM-as-judge evaluation.

LLM-as-judge for groundedness


def evaluate_groundedness(question, answer, context_docs):
    """Check if the answer is grounded in the retrieved documents."""
    context = "\n---\n".join([doc.text for doc in context_docs])

    response = openai.ChatCompletion.create(
        model="gpt-4",
        temperature=0.0,
        messages=[
            {
                "role": "system",
                "content": """Rate the groundedness of the answer on a scale of 1-5:
1 = Answer is completely made up, not in context
2 = Answer has some context support but mostly guesses
3 = Answer is partially grounded in context
4 = Answer is mostly grounded with minor unsupported claims
5 = Answer is entirely supported by the context

Return JSON: {"score": int, "reasoning": "string"}"""
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {question}\n\nAnswer: {answer}"
            }
        ]
    )
    return json.loads(response["choices"][0]["message"]["content"])

Groundedness scoring checks whether every claim in the answer can be traced to the retrieved context. Scores below 3 indicate hallucination.

Building Eval Datasets

An eval dataset is a set of question-document-answer triples. Each entry has a query, the expected documents that should be retrieved, and the expected answer. This is your regression test suite.

Eval dataset format


eval_dataset = [
    {
        "query": "How do I failover Redis?",
        "expected_docs": ["runbooks/redis-failover.md"],
        "expected_answer": "Run redis-cli cluster failover on the replica",
        "category": "runbook_lookup"
    },
    {
        "query": "What caused the checkout outage on Jan 15?",
        "expected_docs": ["postmortems/2024-01-15-checkout.md"],
        "expected_answer": "OOMKills due to memory limit of 256Mi being too low",
        "category": "incident_history"
    },
    {
        "query": "How do I scale pods horizontally?",
        "expected_docs": ["docs/kubernetes-scaling.md"],
        "expected_answer": "kubectl autoscale deployment NAME --min=N --max=M",
        "category": "how_to"
    },
]

Start with 20 to 50 high-quality eval entries. Cover each category of query your system handles. Add new entries as you discover failure modes.

Automated Evaluation Pipelines

CI/CD eval pipeline


import pytest

@pytest.mark.parametrize("eval_entry", eval_dataset)
def test_rag_quality(eval_entry):
    """Run RAG evaluation in CI."""
    results = rag_answer(eval_entry["query"])

    # Check retrieval
    retrieved_sources = [r.source for r in results.retrieved_docs]
    assert any(
        expected in retrieved_sources
        for expected in eval_entry["expected_docs"]
    ), f"Retrieval failed for: {eval_entry['query']}"

    # Check groundedness
    groundedness = evaluate_groundedness(
        eval_entry["query"],
        results.answer,
        results.retrieved_docs
    )
    assert groundedness["score"] >= 3, (
        f"Low groundedness ({groundedness['score']}) for: {eval_entry['query']}"
    )

Run these tests in CI on every prompt or retrieval change. A regression in eval scores blocks deployment.

LLM-as-judge is not perfect

The judge model can make mistakes. Use human review for borderline cases and periodically validate judge scores against human ratings.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.