Auditing AI Actions

Logging every prompt and action so a copilot is debuggable and safe.

6 min readAI for SRE WorkflowsMaster

Why Audit Everything?

When an AI copilot takes or suggests an action, you need a complete record. If something goes wrong, you need to know what the model was asked, what context it had, what it suggested, and what was executed. Without this, the AI is a black box.

What to Log

User input — the exact prompt or command the user provided.
Retrieved context — which documents or data the model was given.
Model parameters — model name, temperature, max tokens.
Raw model output — the full response before any parsing.
Parsed output — extracted commands, classifications, or actions.
Tool calls — which tools were called with what arguments.
Tool results — what the tools returned.
Action taken — what was actually executed.
User feedback — reactions, corrections, or approvals.

Log Format

Structured audit log


import json
import time
import uuid

class AuditLogger:
    def __init__(self, log_file="audit.jsonl"):
        self.log_file = log_file

    def log_event(self, event_type, **kwargs):
        event = {
            "id": str(uuid.uuid4()),
            "timestamp": time.time(),
            "event_type": event_type,
            **kwargs
        }
        with open(self.log_file, "a") as f:
            f.write(json.dumps(event) + "\n")
        return event["id"]

    def log_prompt(self, user_id, channel, query, context_chunks):
        return self.log_event("prompt",
            user_id=user_id,
            channel=channel,
            query=query,
            context_count=len(context_chunks),
            context_sources=[c["source"] for c in context_chunks]
        )

    def log_response(self, audit_id, response, model, tokens_used):
        self.log_event("response",
            audit_id=audit_id,
            response_preview=response[:200],
            model=model,
            tokens_input=tokens_used["input"],
            tokens_output=tokens_used["output"]
        )

    def log_action(self, audit_id, action, result, approved_by=None):
        self.log_event("action",
            audit_id=audit_id,
            action=action,
            result=result,
            approved_by=approved_by
        )

JSONL format is easy to query with tools like jq, DuckDB, or Elasticsearch. Each line is a complete event record.

Compliance & Retention

Data Type	Retention	Storage
Audit logs	90 days	S3 or cold storage
Prompt content	30 days	Encrypted storage
Tool execution results	90 days	Same as audit logs
User identifiers	90 days	Encrypted storage

Redact sensitive data from logs

Never log API keys, passwords, tokens, or PII. Use a redaction layer before writing audit logs. If the prompt contains sensitive data, redact it before storage.

Debugging with Audit Logs

Querying audit logs


# Find all actions taken by the bot today
cat audit.jsonl | jq -r 'select(.event_type == "action") | select(.timestamp > 1705276800)'

# Find prompts that led to high-risk actions
cat audit.jsonl | jq -r 'select(.event_type == "action" and .risk_level == "high")'

# Count tokens used per user
cat audit.jsonl | jq -r 'select(.event_type == "response") | .tokens_input' | paste -sd+ | bc

Audit logs are your debugging tool. When the AI gives a bad answer, you can trace exactly what context it had and why it made that choice.

Build a dashboard for audit data

Create a simple dashboard showing daily token usage, action counts, approval rates, and error rates. This gives you visibility into the AI's operational health.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 7 · Master

ChatOps & On-Call Copilots