Load Testing and the First On-Call Runbook

Why Performance Needs a Plan

Load testing is useful only when it tests a system behavior that matters. A single endpoint hammered without realism often measures the test harness more than the backend. A production-grade load-test plan therefore starts from workload shape: which routes dominate, which requests are latency-sensitive, which flows are read-heavy, which flows trigger writes, and which dependencies are expected to saturate first. The goal is not to discover a vanity requests-per-second number. The goal is to learn where the system bends and what an operator should do when it bends.

Current Limitations and Baselines

Because the current Meridian scaffold exposes only /healthz, the only honest performance baseline available today is process-level and routing-level behavior: startup time, health-check latency, concurrent connection handling, and the gateway's ability to fan out to downstream health endpoints once proxy routes exist. That limited baseline is still useful. It teaches an important engineering habit: performance work should grow with actual functionality, not with invented benchmark theater.

Designing the Load Test

A sensible Meridian load-test plan should evolve in phases. Phase one measures health and routing overhead while the repository is still a scaffold. Phase two introduces authenticated gateway-to-service CRUD traffic once business handlers exist. Phase three adds mixed workloads: tenant lookup, resident reads, invoice writes, and downstream event publication. Phase four becomes soak and failure-mode testing, such as degraded Redis, slower Postgres, or a restarting downstream service. That progression generalizes to almost any backend: baseline first, business mix second, resilience third.

What a Runbook Must Contain

A runbook is a decision aid for a stressed engineer, not a prose essay. It should name the symptom, the alert threshold, the first dashboards or commands to open, the safe mitigations to try, the dangerous actions to avoid, and the escalation criteria. A vague runbook that says 'investigate logs' is operationally close to having no runbook at all. The first five minutes of an incident should become more deterministic, not more improvisational.

Planned Assets

Reference Scenarios

import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  vus: 25,
  duration: "2m",
  thresholds: {
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(95)<100"],
  },
};

export default function () {
  const res = http.get("http://localhost:8080/healthz");
  check(res, { "gateway healthz is 200": (r) => r.status === 200 });
  sleep(1);
}

# Alert: Meridian gateway latency or upstream failures

## Trigger
- P95 gateway latency exceeds the agreed SLO threshold
- or gateway 5xx rate rises above the paging threshold

## First look
1. Confirm whether /healthz is failing only on gateway or also on downstream services.
2. Check the most recent rollout history for gateway and the affected downstream Deployment.
3. Compare request volume with pod restarts, readiness failures, and downstream error rates.

## Safe mitigations
- Roll back the newest Deployment revision if the issue correlates with a recent release.
- Scale the affected request-path Deployment only if the bottleneck is capacity, not a failing downstream dependency.
- Restore a broken ConfigMap or Secret version if the symptom started immediately after configuration change.

## Avoid first
- Do not restart every service at once.
- Do not bypass gateway by exposing internal services publicly.
- Do not flush shared caches broadly unless cache corruption is the confirmed symptom.

## Escalate when
- Error rate remains high after rollback or capacity mitigation.
- Tenant-isolation or data-correctness issues are suspected.
- A dependency outside the cluster is failing and requires another owning team.