Stage 7 · Master
Phase 18 — Performance Engineering
Benchmarking
Write table-driven Go benchmarks against a real Testcontainers PostgreSQL instance, read benchstat's statistical comparison instead of a single run, and gate CI on regressions instead of hoping someone notices.
A Benchmark Is a Claim That Needs Evidence, Not a Feeling
Lesson 2 claimed the batched GetByIDs rewrite is faster than the original N+1 loop. That claim was true for round-trip count by construction, but "faster" for the whole request — including marshaling, map building, and the batched query's own cost at different organization sizes — was never actually measured. A benchmark turns "should be faster" into a number with a confidence interval.
package service_test
import (
"context"
"fmt"
"testing"
"github.com/hoa-platform/backend/services/organization/internal/service"
"github.com/hoa-platform/backend/tests/integration/fixtures"
)
var membershipSizes = []int{5, 40, 250}
func BenchmarkListWithProfiles(b *testing.B) {
pool := fixtures.PostgreSQL(b) // shared Testcontainers instance from Part 8
for _, size := range membershipSizes {
orgID := fixtures.SeedOrganizationWithMembers(b, pool, size)
svc := service.NewMembershipService(fixtures.MembershipRepo(pool), fixtures.UserRepo(pool))
b.Run(fmt.Sprintf("members=%d", size), func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer() // exclude fixture seeding from the measured loop
for i := 0; i < b.N; i++ {
views, err := svc.ListWithProfiles(context.Background(), orgID)
if err != nil {
b.Fatal(err)
}
sink = views // prevents the compiler from eliminating the call as dead code
}
})
}
}
// sink is a package-level variable; assigning to it defeats dead-code
// elimination that would otherwise make the benchmark measure nothing.
var sink []service.MemberViewb.ResetTimer() after seeding, b.ReportAllocs() for allocation counts, and the package-level sink variable are the three details that separate a benchmark measuring the real workload from one measuring an optimized-away no-op.
If the Go compiler can prove a computed value is never observed, it is free to eliminate the code that computes it — including inside a benchmark loop. Assigning the result to a package-level variable is the standard defense; a local variable that is never read again is not enough.
Benchmark the Shape of Real Data, Not One Convenient Size
5, 40, and 250 members are not arbitrary — they are this platform's actual p50, p95, and largest-tenant membership counts, pulled from a production query against organization_memberships. A benchmark run only at size 5 would have hidden exactly the scaling behavior Lesson 2 was written to fix.
BenchmarkListWithProfiles/members=5-10 41823 28912 ns/op 4102 B/op 58 allocs/op
BenchmarkListWithProfiles/members=40-10 892 1301884 ns/op 38210 B/op 481 allocs/op
BenchmarkListWithProfiles/members=250-10 104 9812204 ns/op 210144 B/op 2803 allocs/opns/op scales roughly linearly with member count here, which is the expected shape for one batched query plus O(n) map construction — a super-linear curve at this stage would indicate an accidental N+1 reintroduced somewhere in the rewrite.
Compare Before and After With benchstat, Not by Eye
A single before/after number invites confirmation bias — it is easy to see the improvement you expect. benchstat runs a statistical test across repeated samples and reports whether a difference is likely real or within normal run-to-run noise, which matters because the CI runner's own background load varies between commits.
name old time/op new time/op delta
ListWithProfiles/members=40-10 38.2ms ± 4% 1.30ms ± 2% -96.60% (p=0.000 n=10+10)
ListWithProfiles/members=250-10 238.9ms ± 6% 9.81ms ± 3% -95.89% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
ListWithProfiles/members=40-10 61.4kB ± 0% 38.2kB ± 0% -37.79% (p=0.000 n=10+10)p=0.000 across 10 runs each means the difference is not noise — this is the actual evidence behind Lesson 2's claim, not an assumption from reading the code.
Gate CI on a Regression Threshold, Not on the Benchmark Merely Running
A benchmark that runs in CI but whose output nobody reads catches nothing — regressions creep in silently until the next unrelated incident. The gate below stores a committed baseline and fails the build only when a change regresses time/op beyond a documented tolerance, so normal machine-to-machine noise does not produce false failures.
benchmark-regression:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/setup-go@0aaccfd150d50ccaeb58ebd88d36e91967a5f35b # v5.4.0
with:
go-version: "1.26.x"
- run: go test ./services/organization/internal/service/... -bench BenchmarkListWithProfiles -benchmem -run '^$' -count 10 > current.txt
- run: go run golang.org/x/perf/cmd/benchstat@latest .benchstat/baseline.txt current.txt | tee comparison.txt
- run: |
# Fails the job if any case regresses time/op by more than 15%,
# the documented tolerance for normal CI-runner variance.
if grep -E '\+([1-9][5-9]|[2-9][0-9])\.[0-9]+%' comparison.txt; then
echo "::error::Benchmark regression exceeds 15% tolerance"
exit 1
fiThe baseline file is committed and only updated deliberately, in its own reviewed commit, when a change is expected to alter performance — never silently overwritten by CI itself.
Applied exercise
Benchmark the keyset pagination rewrite from Lesson 2
Lesson 2 replaced the flats list endpoint's OFFSET pagination with a keyset cursor but never measured it with a Go benchmark — only with a single EXPLAIN ANALYZE capture.
- Write a table-driven benchmark comparing OFFSET-based and keyset-based repository methods at page depths 1, 50, and 400.
- Use b.ReportAllocs() and a package-level sink to prevent dead-code elimination.
- Run both with -count 10 and compare using benchstat.
- State the tolerance you would set for a CI regression gate on this specific benchmark, and justify the number from the benchstat variance you observed.
Deliverable
A committed benchmark file, an old.txt/new.txt/benchstat comparison showing the expected flattening at page 400, and a written CI tolerance recommendation.
Completion checks
- The benchmark seeds a realistic row count matching the largest tenant's actual flats table size.
- benchstat's p-value indicates statistical significance, not just a favorable single run.
- The stated CI tolerance is derived from observed variance, not an arbitrary round number.
Why does the benchmark assign its result to a package-level 'sink' variable instead of a local variable inside the loop?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.