Stage 7 · Master
Capstone — Assembling Fieldwork End-to-End
Load Testing & an On-Call Runbook
Find where Fieldwork actually bends under pressure, then write the document an on-call engineer can use while half-awake at 3am.
What We Tested and Why
Load testing mattered in Fieldwork for one reason: every earlier module had quietly created assumptions about concurrency. The gateway assumes JWT verification is cheap enough to do on every request. The tasks service assumes its connection pool can absorb bursts. Redis assumes cached reads will shield Postgres. Kafka assumes asynchronous work stays off the write path. None of those assumptions become engineering facts until traffic tries to break them.
We rejected the lazy version of performance testing — one endpoint hammered in isolation from a laptop against localhost — because it teaches the wrong bottlenecks. Fieldwork's real shape is mixed traffic: list tasks far more often than create tasks, periodic auth refreshes, bursts when teams open the same workspace after stand-up, and enough write activity to keep the outbox moving. A single synthetic benchmark would have optimized the wrong thing.
| Test style | Why it looks appealing | Why we rejected or kept it |
|---|---|---|
| Single endpoint, local benchmark | Fast to run and easy to graph | Rejected because it hides network hops, shared pools, Redis behavior, and realistic route mix |
| Mixed scenario against a deployed environment | Harder to set up | Kept because it exposes the actual coupling between gateway, service, database, cache, and Kafka |
| Peak-only spike test | Finds dramatic failures quickly | Insufficient alone because many backend issues appear during sustained saturation, not instant shock |
The k6 Scenario We Kept
We used k6 because it was simple enough to keep in the repository and expressive enough to model the request mix we actually cared about. The scenario used one authenticated tenant with seeded workspaces, projects, and users, then split virtual users across task listing, task creation, and task status updates. That mattered more than heroic volume. A realistic ratio tells you which layer becomes dominant first.
import http from "k6/http";
import { check, sleep } from "k6";
export const options = {
scenarios: {
listTasks: {
executor: "ramping-arrival-rate",
startRate: 20,
timeUnit: "1s",
preAllocatedVUs: 40,
maxVUs: 120,
stages: [
{ target: 80, duration: "5m" },
{ target: 120, duration: "10m" },
{ target: 0, duration: "3m" },
],
exec: "listTasks",
},
createTask: {
executor: "constant-arrival-rate",
rate: 12,
timeUnit: "1s",
duration: "15m",
preAllocatedVUs: 20,
maxVUs: 60,
exec: "createTask",
},
updateTask: {
executor: "constant-arrival-rate",
rate: 18,
timeUnit: "1s",
duration: "15m",
preAllocatedVUs: 20,
maxVUs: 60,
exec: "updateTask",
},
},
thresholds: {
http_req_failed: ["rate<0.01"],
http_req_duration: ["p(95)<350", "p(99)<700"],
checks: ["rate>0.99"],
},
};
const baseURL = __ENV.FIELDWORK_BASE_URL;
const token = __ENV.FIELDWORK_TOKEN;
const workspaceID = __ENV.FIELDWORK_WORKSPACE_ID;
const projectID = __ENV.FIELDWORK_PROJECT_ID;
const params = {
headers: {
Authorization: "Bearer " + token,
"Content-Type": "application/json",
},
};
export function listTasks() {
const res = http.get(
baseURL + "/v1/workspaces/" + workspaceID + "/projects/" + projectID + "/tasks?limit=25&sortBy=updatedAt&direction=desc",
params,
);
check(res, { "list tasks returns 200": (r) => r.status === 200 });
sleep(1);
}
export function createTask() {
const payload = JSON.stringify({
title: "Load test task " + Date.now(),
description: "Created by k6 during sustained load.",
assigneeId: "usr_load",
labels: ["perf", "k6"],
});
const res = http.post(
baseURL + "/v1/workspaces/" + workspaceID + "/projects/" + projectID + "/tasks",
payload,
params,
);
check(res, { "create task returns 201": (r) => r.status === 201 });
}
export function updateTask() {
const taskID = "tsk_seed_" + (__VU % 200);
const payload = JSON.stringify({ status: "in_progress" });
const res = http.patch(
baseURL + "/v1/workspaces/" + workspaceID + "/projects/" + projectID + "/tasks/" + taskID,
payload,
params,
);
check(res, { "update task returns 200": (r) => r.status === 200 });
}The exact numbers are less important than the shape: reads dominate, writes are steady, and thresholds reflect a product decision instead of whatever latency happened to appear on a graph.
We also ran the test against a staging environment with the same ingress, gateway, Redis, Kafka, and Postgres topology shape as production. Not identical scale — that would be wasteful — but identical coupling. Local container-compose tests were still useful for fast iteration, just not for calling the system ready.
What Broke First Under Realistic Load
Three bottlenecks appeared before anything dramatic like CPU exhaustion. First, the tasks service ran out of usable Postgres connections because list queries and create-task transactions competed for the same small pool. Second, the project task-list endpoint had a quiet N+1 shape once assignee display names were joined through a separate repository call. Third, when Redis keys expired together, several pods regenerated the same workspace summary concurrently and amplified load back into Postgres. None of these were visible in happy-path development.
gateway_http_request_duration_seconds{route="POST /v1/.../tasks",quantile="0.95"} -> 612ms
fieldwork_tasks_db_wait_count_total -> steep increase after minute 6
fieldwork_tasks_db_wait_duration_seconds_sum -> 48s over 60s window
fieldwork_tasks_cache_miss_total{key="workspace:summary"} -> synchronized spikes
fieldwork_notifications_consumer_lag -> stable
postgres_pg_stat_activity{state="active",application_name="tasks-service"} -> pool ceiling reachedNotice what is not failing first: Kafka. The async side held up. The synchronous read and write path around Postgres bent first, which is where most product-facing latency usually comes from in CRUD-heavy systems.
Once requests queue on the database pool, almost every route looks unhealthy. That is why pool metrics belong on the first dashboard page, not buried in an appendix behind CPU charts.
The Fixes That Moved the Breaking Point
We did not solve the bottlenecks by throwing replicas at them immediately. More pods would only have meant more goroutines waiting for the same constrained database. The better fixes were narrower: tune pool settings for the real concurrency shape, collapse the N+1 query into one join, and add single-flight protection around cache regeneration. Only after that did extra replicas buy anything.
SELECT
t.id,
t.title,
t.status,
t.updated_at,
u.id AS assignee_id,
u.name AS assignee_name
FROM tasks t
LEFT JOIN users u
ON u.tenant_id = t.tenant_id
AND u.id = t.assignee_id
WHERE t.tenant_id = $1
AND t.project_id = $2
ORDER BY t.updated_at DESC, t.id DESC
LIMIT $3 OFFSET $4;The earlier version fetched task rows first and then called usersRepo.Find per assignee. It looked clean in code review and collapsed under read-heavy load.
| Observed issue | Naive reaction | Fix that actually helped |
|---|---|---|
| Postgres pool waits | Add more tasks-service replicas | Tune pool size and query behavior first so replicas do not multiply contention |
| Slow task-list endpoint | Cache the whole endpoint immediately | Remove the N+1 query so the uncached path is sane first |
| Redis expiry spikes | Increase TTL until data feels stale | Use single-flight and staggered expirations so misses do not synchronize |
Writing the On-Call Runbook
The runbook came directly from the load test findings. That is the part teams skip: they discover the failure mode once, fix it, and leave future responders to rediscover the same debugging path under pager pressure. Fieldwork's runbook is deliberately opinionated. It tells the on-call engineer what symptom means, which dashboard to open first, what commands or panels matter, which mitigations are safe, and what not to do without a rollback plan.
# Alert: Fieldwork Task API Latency High
## Trigger
- P95 gateway latency for task routes > 400ms for 10m
- 5xx rate > 1% for task routes for 5m
## First look (2 minutes)
1. Open Grafana dashboard: Fieldwork / Request Path / Tasks
2. Check panels in order:
- gateway route latency by status
- tasks-service DB pool waits
- Postgres active connections and slow query count
- Redis hit ratio and miss spikes
- Kafka consumer lag (to rule out async backlog)
3. Confirm whether impact is read-heavy, write-heavy, or global.
## Safe mitigations
- If DB wait count is spiking and replicas recently increased, scale tasks-service back down before scaling Postgres clients further.
- If one query dominates slow-query logs, enable the read-path feature flag that disables assignee expansion.
- If Redis miss storm is visible, flush only the affected summary key pattern after verifying Postgres headroom.
## Do not do first
- Do not restart Postgres clients blindly; connection storms make recovery slower.
- Do not flush all Redis keys for the tenant unless product leadership accepts a cold-cache surge.
- Do not pause Kafka consumers unless event lag is the confirmed symptom.
## Escalate when
- P95 > 800ms for 15m after mitigation
- Postgres CPU > 85% for 10m
- Error budget burn alert is also firing
- Tenant-specific impact suggests data corruption, not saturationGood runbooks remove choices, especially in the first five minutes. They should feel a little bossy. That is a feature, not a tone problem.
- Every alert page gets: trigger, likely causes, first dashboard, safe mitigations, unsafe actions, and escalation thresholds.
- Runbooks reference exact dashboard names and feature flags instead of vague instructions like 'check the database'.
- Temporary mitigations are written down only if they were exercised during load tests or incidents, not imagined in calm hindsight.
The Minimum Dashboards and Alerts
We kept the dashboard set intentionally small. One overview board for request-path health, one database board, one cache board, and one async-events board were enough for first response. More charts would have looked thorough and made incidents slower. The alert set followed the same rule: page on symptoms users feel or on error-budget burn, not on every infrastructure tremor.
groups:
- name: fieldwork-latency
rules:
- alert: FieldworkTaskAPIP95High
expr: histogram_quantile(0.95, sum by (le, route) (rate(gateway_http_request_duration_seconds_bucket{route=~".*tasks.*"}[5m]))) > 0.4
for: 10m
labels:
severity: page
service: api-gateway
annotations:
summary: Task API latency is above SLO threshold
runbook: https://runbooks.fieldwork.dev/task-api-latency
- alert: FieldworkTaskDBPoolWaitHigh
expr: rate(fieldwork_tasks_db_wait_count_total[5m]) > 5
for: 10m
labels:
severity: ticket
service: tasks-service
annotations:
summary: Tasks service is frequently waiting on the DB pool
runbook: https://runbooks.fieldwork.dev/task-api-latency
- alert: FieldworkErrorBudgetBurnFast
expr: fieldwork:slo_error_budget_burn_rate_1h{service="api-gateway"} > 14
for: 5m
labels:
severity: page
service: api-gateway
annotations:
summary: Error budget burn rate is critically high
runbook: https://runbooks.fieldwork.dev/slo-burnThe page-worthy alerts are latency and burn-rate based because those map to actual product pain. Pool-wait alerts are still useful, but they are supporting signals unless they are already causing user-visible degradation.
The load test told us where Fieldwork bends. The runbook turns that knowledge into a reusable operational asset. Without it, the next incident starts from zero again, and the learning was never really shipped.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.