Stage 7 · Master
Phase 18 — Performance Engineering
Load Testing
Write a k6 script that ramps virtual users against api-gateway, define SLOs before the run instead of after, and find the platform's real saturation point rather than assuming the numbers from a single benchmark generalize.
A Passing Benchmark Does Not Predict Behavior Under Concurrent Real Traffic
Lessons 4 and 5 measured one endpoint in isolation, called directly, with no other traffic competing for the same connection pool, CPU, or Redis cache. Production traffic hits eleven services simultaneously through api-gateway, and the platform's actual failure mode — Lesson 3's pool exhaustion — only appeared under concurrent, cross-service load a single-endpoint benchmark cannot reproduce. A load test exercises the system the way production actually will.
Write the SLO Down Before Running the Test, Not After Reading the Results
Choosing a target after seeing the numbers is how a bad result quietly becomes an acceptable one. The platform's on-call runbook already commits to a public SLO for the gateway's flat-listing route; the load test's pass/fail criteria are copied directly from that document, not invented for the test.
| Metric | SLO target | Source |
|---|---|---|
| p95 latency | < 300ms | Platform SLO document, §4.2 (Gateway routes) |
| p99 latency | < 800ms | Platform SLO document, §4.2 |
| Error rate | < 0.1% | Platform SLO document, §2.1 (all routes) |
| Sustained throughput | ≥ 150 req/s per gateway replica | Capacity plan, current 3-replica gateway deployment |
Write a Ramping k6 Script, Not a Fixed-Rate One
A fixed-rate script only proves the system survives one specific load level. A ramping script — increasing virtual users in stages — finds the point where latency and error rate begin to degrade, which is the actual capacity question the platform needs answered before the next marketing push.
import http from "k6/http";
import { check } from "k6";
export const options = {
scenarios: {
ramp: {
executor: "ramping-vus",
startVUs: 0,
stages: [
{ duration: "30s", target: 20 },
{ duration: "60s", target: 60 },
{ duration: "60s", target: 120 },
{ duration: "60s", target: 200 },
{ duration: "30s", target: 0 },
],
},
},
thresholds: {
// Matches the platform SLO document exactly — a threshold failure here
// is meant to fail the test, not be adjusted to whatever the run produced.
http_req_duration: ["p(95)<300", "p(99)<800"],
http_req_failed: ["rate<0.001"],
},
};
export default function () {
const res = http.get(`${__ENV.GATEWAY_URL}/v1/organizations/current/flats?limit=20`, {
headers: { Authorization: `Bearer ${__ENV.LOAD_TEST_TOKEN}` },
});
check(res, {
"status is 200": (r) => r.status === 200,
"returned a page of flats": (r) => JSON.parse(r.body).items.length > 0,
});
}The thresholds block is what turns this from a script that merely generates traffic into an automated pass/fail check — k6 exits non-zero if any threshold is violated at any point in the run.
Find the Saturation Point, Not Just Pass/Fail
http_req_duration..............: avg=142.31ms p(95)=289.4ms p(99)=612.8ms
http_req_failed................: 0.42% 842 out of 200,014 requests
✗ http_req_duration
✗ 'p(95)<300' rate=0.94 (VUs 0-140: passing; VUs 140-200: p95 crosses 300ms)
✓ 'p(99)<800'
✗ http_req_failed
✗ 'rate<0.001' rate=0.0042 (failures concentrated in the 180-200 VU stage)The p95 threshold fails specifically in the 140-200 VU range, not across the whole run — that range is the platform's actual saturation point at the current gateway replica count, not an abstract 'it broke' result.
Plotting p95 latency against concurrent VUs over time shows a knee — the point where latency stops scaling linearly with load and starts scaling super-linearly. That knee, here around 140 VUs, is the number capacity planning should use, not the peak VU count the script happened to reach.
Connect the Breaking Point to a Specific Resource, Not a Vague 'Too Much Load'
The pgxmetrics dashboard from Lesson 3, watched live during the same run, shows organization's pool acquired/max ratio crossing 0.9 at the same wall-clock moment the k6 summary shows p95 crossing 300ms — the saturation point is the connection pool becoming the bottleneck again, this time at a higher absolute concurrency than the incident that originally motivated Lesson 3's fix, but the same underlying resource.
- Correlate the k6 timeline against Grafana panels for pool acquired/max ratio, CPU, and Redis hit rate during the same window — the metric that saturates first at the same timestamp the SLO breaks is the actual bottleneck.
- Re-run the identical script after a targeted change (e.g., raising gateway replica count, or MaxConns within the shared budget from Lesson 3) to confirm the knee moves, not just that peak throughput number improved.
- Record the new saturation point in the capacity plan alongside the SLO document — a load test that is not written down is a load test that will be re-litigated from scratch next quarter.
A Passing Load Test Against Staging Is Not a Production Guarantee
Staging's largest organization has a fraction of the largest production tenant's row counts, and Redis starts the test cold instead of warmed by real traffic. A staging load test result should be read as directional evidence of where a bottleneck likely is, not as a guaranteed production capacity number — Lesson 6 of Production Deployment's rollout strategy is what actually protects production if this estimate is wrong.
Applied exercise
Load test the membership listing endpoint and find its knee
Lessons 2, 4, and 5 optimized services/organization's membership-listing endpoint individually. It has never been load tested end-to-end through api-gateway.
- Write a ramping k6 script targeting the membership-listing route, with thresholds copied from the platform SLO document.
- Run it against a staging-scale environment and record the VU range where any threshold first fails.
- Correlate the failure point against pgxmetrics and CPU dashboards to identify which resource saturates first.
- Write a one-paragraph capacity recommendation stating the safe sustained VU count and the resource that would need to change to raise it.
Deliverable
A committed k6 script, a summary showing the exact VU range of the SLO breach, a resource-correlation finding, and a written capacity recommendation.
Completion checks
- Thresholds in the script match the platform SLO document's stated numbers exactly, not values chosen after seeing the results.
- The saturation point is stated as a VU range tied to a wall-clock-correlated resource metric, not just 'it got slow'.
- The capacity recommendation names a specific resource and a specific proposed change, not a general suggestion to 'add more servers'.
Why are the k6 script's thresholds (p95<300ms, p99<800ms, error rate<0.1%) copied from an existing SLO document instead of chosen after reviewing the test's results?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.