LLM Tracing

Trace prompts, retrieved chunks, tool calls, model parameters, latency, and errors with OpenTelemetry.

8 min readAI for SRE WorkflowsMaster

Why Trace LLM Calls?

LLM calls are expensive and non-deterministic. You need to know which prompt produced which response, how long it took, what it cost, and what context was used. Tracing gives you this visibility.

OpenTelemetry Integration

Instrument LLM calls with OpenTelemetry


from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanExporter

# Setup
provider = TracerProvider()
exporter = BatchSpanExporter(endpoint="http://otel-collector:4318")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm-service")

def traced_llm_call(messages, model, feature):
    """Execute an LLM call with full tracing."""
    with tracer.start_as_current_span("llm_completion") as span:
        # Set attributes
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.feature", feature)
        span.set_attribute("llm.input_tokens", estimate_tokens(messages))

        # Trace retrieval if RAG
        with tracer.start_as_current_span("retrieval") as retrieval_span:
            context_chunks = retrieve_context(messages[-1]["content"])
            retrieval_span.set_attribute("retrieval.chunk_count", len(context_chunks))
            retrieval_span.set_attribute("retrieval.sources", [c["source"] for c in context_chunks])

        # Call the model
        start = time.time()
        response = openai.ChatCompletion.create(
            model=model,
            messages=messages
        )
        latency = time.time() - start

        # Set output attributes
        span.set_attribute("llm.output_tokens", response["usage"]["completion_tokens"])
        span.set_attribute("llm.total_tokens", response["usage"]["total_tokens"])
        span.set_attribute("llm.latency_ms", latency * 1000)
        span.set_attribute("llm.cost_usd", calculate_cost(model, response["usage"]))

        return response["choices"][0]["message"]["content"]

Each span captures one unit of work. The trace connects retrieval, model call, and tool execution into a single view.

Trace Spans

Span	What It Captures	Key Attributes
llm_completion	Full LLM call	model, tokens, latency, cost
retrieval	Document retrieval	chunk_count, sources, query
tool_call	Tool execution	tool_name, arguments, result
embedding	Embedding generation	model, dimensions, input_tokens
postprocessing	Output parsing	format, validation_result

Custom Attributes

Rich custom attributes


def trace_rag_request(query, user_id, channel):
    """Trace a full RAG request with rich attributes."""
    with tracer.start_as_current_span("rag_request") as span:
        span.set_attribute("user.id", user_id)
        span.set_attribute("channel.id", channel)
        span.set_attribute("query.length", len(query))

        # Retrieval span
        with tracer.start_as_current_span("retrieve") as ret_span:
            chunks = retrieve(query)
            ret_span.set_attribute("retrieval.count", len(chunks))
            ret_span.set_attribute("retrieval.top_score", chunks[0]["score"] if chunks else 0)
            ret_span.set_attribute("retrieval.sources", list(set(c["source"] for c in chunks)))

        # LLM span
        with tracer.start_as_current_span("generate") as gen_span:
            response = generate_answer(query, chunks)
            gen_span.set_attribute("llm.model", response["model"])
            gen_span.set_attribute("llm.tokens.input", response["usage"]["input_tokens"])
            gen_span.set_attribute("llm.tokens.output", response["usage"]["output_tokens"])
            gen_span.set_attribute("llm.latency_ms", response["latency_ms"])
            gen_span.set_attribute("llm.cost_usd", response["cost"])

        # Quality span
        with tracer.start_as_current_span("quality_check") as q_span:
            groundedness = check_groundedness(response["answer"], chunks)
            q_span.set_attribute("quality.groundedness", groundedness)
            q_span.set_attribute("quality.score", groundedness)

Rich attributes let you filter and analyze traces by user, channel, model, and quality metrics.

Trace Analysis

Analyze traces for optimization


def analyze_traces(traces):
    """Analyze traces to find optimization opportunities."""
    analysis = {
        "total_requests": len(traces),
        "avg_latency_ms": np.mean([t["latency_ms"] for t in traces]),
        "p99_latency_ms": np.percentile([t["latency_ms"] for t in traces], 99),
        "total_cost_usd": sum(t["cost_usd"] for t in traces),
        "avg_tokens_per_request": np.mean([t["total_tokens"] for t in traces]),
        "cache_hit_rate": sum(1 for t in traces if t.get("cached")) / len(traces),
        "error_rate": sum(1 for t in traces if t.get("error")) / len(traces),
    }

    # Find slow requests
    slow_requests = [t for t in traces if t["latency_ms"] > 10000]
    analysis["slow_requests"] = len(slow_requests)

    # Find expensive requests
    expensive = [t for t in traces if t["cost_usd"] > 0.10]
    analysis["expensive_requests"] = len(expensive)

    return analysis

Trace analysis reveals which features are slow, expensive, or error-prone. Use this to prioritize optimizations.

Export traces to your existing backend

Export LLM traces to Jaeger, Zipkin, or your cloud tracing service. View them alongside infrastructure traces for full-stack visibility.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 7 · Master

LLM Cost, Latency & Observability