Contributing Factors

## Five Whys: Database connection pool exhaustion

### Why 1: Why did the API return 503 errors?
The application ran out of database connections.
→ Connection pool exhaustion caused request failures.

### Why 2: Why did the connection pool exhaust?
The pool was configured for 50 connections but needed 80.
→ Configuration did not match actual demand.

### Why 3: Why was the pool configured for 50?
The original configuration was set during initial deployment
and never updated as traffic grew.
→ No process for reviewing and updating configurations.

### Why 4: Why was there no review process?
There is no standard set of application metrics that all
services must expose, including connection pool utilization.
→ No standard observability requirements.

### Why 5: Why are there no standard requirements?
We do not have a service onboarding checklist that defines
minimum observability, alerting, and documentation.
→ No standard service maturity framework.

### Root Cause
Lack of a service maturity framework that requires connection
pool monitoring, capacity review processes, and configuration
documentation.

### Action Items
1. Define standard application metrics including connection pools
2. Create service onboarding checklist
3. Implement quarterly configuration review process

human_factors:
  fatigue:
    description: "Tired engineers make more mistakes"
    indicators:
      - "On-call engineer was paged at 3 AM"
      - "Engineer had been on-call for > 5 days"
      - "Multiple incidents in short succession"
    systemic_fixes:
      - "Fair on-call rotation"
      - "Post-incident recovery time"
      - "Adequate staffing"

  time_pressure:
    description: " rush leads to shortcuts"
    indicators:
      - "Incident during high-traffic period"
      - "Pressure to resolve quickly"
      - "Competing priorities"
    systemic_fixes:
      - "Clear escalation criteria"
      - "Pre-approved remediation paths"
      - "足够 staffing for peak times"

  unclear_procedures:
    description: "Ambiguity leads to mistakes"
    indicators:
      - "Runbook is outdated or missing"
      - "Multiple valid approaches with different risks"
      - "No clear ownership"
    systemic_fixes:
      - "Regular runbook testing"
      - "Clear decision frameworks"
      - "Defined ownership"

  inadequate_training:
    description: "Unknown procedures lead to errors"
    indicators:
      - "New team member involved"
      - "Unfamiliar system or failure mode"
      - "No game day practice"
    systemic_fixes:
      - "Shadow on-call rotation"
      - "Regular game days"
      - "Knowledge sharing sessions"

latent_conditions:
  untested_procedures:
    description: "Procedures that have never been tested"
    examples:
      - "Database failover procedure"
      - "Disaster recovery plan"
      - "Certificate renewal automation"
    detection: "Game days and regular testing"
    remediation: "Test and update procedures quarterly"

  missing_monitoring:
    description: "Metrics not collected or alerts not configured"
    examples:
      - "Connection pool utilization"
      - "Certificate expiry"
      - "Disk space on specific nodes"
    detection: "USE/RED method audits"
    remediation: "Standard observability requirements"

  single_points_of_failure:
    description: "Components that can cause full outage"
    examples:
      - "Single database instance"
      - "Single DNS provider"
      - "Single certificate authority"
    detection: "Architecture reviews"
    remediation: "Redundancy and failover"

  untested_backups:
    description: "Backups that have never been restored"
    examples:
      - "Database backups"
      - "Configuration backups"
      - "Certificate backups"
    detection: "Regular restore tests"
    remediation: "Monthly restore tests"

safeguards_analysis:
  prevention_safeguards:
    description: "Controls that prevent incidents from occurring"
    examples:
      - "Pre-deploy validation"
      - "Capacity limits"
      - "Configuration review"
    missing: "List safeguards that were absent"

  detection_safeguards:
    description: "Controls that detect incidents early"
    examples:
      - "Monitoring and alerting"
      - "SLO tracking"
      - "Health checks"
    missing: "List safeguards that were absent"

  mitigation_safeguards:
    description: "Controls that limit incident impact"
    examples:
      - "Circuit breakers"
      - "Rate limiting"
      - "Feature flags"
    missing: "List safeguards that were absent"

  recovery_safeguards:
    description: "Controls that enable recovery"
    examples:
      - "Automated rollback"
      - "Backup restoration"
      - "Failover procedures"
    missing: "List safeguards that were absent"

systems_thinking_questions:
  interactions:
    - "How did this component interact with others?"
    - "What dependencies were involved?"
    - "How did the failure cascade?"

  organizational:
    - "What team structures influenced this incident?"
    - "What incentives led to this situation?"
    - "What communication gaps existed?"

  temporal:
    - "How did this situation develop over time?"
    - "What past decisions contributed?"
    - "Why was this not caught earlier?"

  environmental:
    - "What external factors contributed?"
    - "How did traffic patterns affect this?"
    - "What timing factors were involved?"

  feedback_loops:
    - "What feedback loops existed?"
    - "Were there delays in feedback?"
    - "How could feedback be improved?"