Stage 7 · Master
Phase 20 — Production Readiness
Production Checklist
Replace 'we believe this is production-ready' with a single executable script that independently re-verifies every control from this module, and use it as the capstone gate for onboarding a fourteenth service.
Seven lessons each built one control: least-privilege database roles and row-level security, off-cluster PITR-capable backups, a measured independent restore drill, a safe API versioning contract, a tamper-evident audit log, staged and audited feature flags, and a burn-rate-aware SLO gate. A checklist that lists those seven items in a wiki page and asks an engineer to tick boxes from memory is exactly the kind of unverified control this whole module has been arguing against. This lesson closes the loop: one script that re-derives the true state of every control by querying the same systems those lessons built, and a capstone exercise that onboards a new service, services/notification-service, through every gate at once.
Distinguish a Checklist of Claims From a Checklist of Verified Facts
Every earlier lesson in this module built a specific, executable verification — a psql session proving DDL is rejected, an archive-lag metric, a restore-drill ledger entry, a Deprecation-header traffic check, an offsite hash-chain verifier, a rollback measured within a TTL window, a burn-rate query. A production checklist that does not run those same checks and instead asks a human to remember and attest to each one has silently regressed to trusting the same unverified assumptions this module exists to eliminate — the PVC dump that 'should' restore, the flag that 'should' roll back fast, the role split that 'should' still be correctly scoped six months after it was set up.
Build One Script That Re-Derives Every Control's True State
#!/usr/bin/env bash
set -euo pipefail
# Usage: deploy/production-readiness/verify.sh <service-name>
# Re-derives, does not assume, the true state of every Production
# Readiness control for the named service.
service="${1:?usage: verify.sh <service-name>}"
failures=0
check() {
local description="$1"
shift
if "$@"; then
echo " [pass] $description"
else
echo " [FAIL] $description"
failures=$((failures + 1))
fi
}
echo "== Security: role isolation for $service =="
check "hoa_${service}_app_1 cannot run DDL" bash -c \
"! PGPASSWORD=\"\$APP_PASSWORD\" psql \"postgres://hoa_${service}_app_1@\$DATABASE_HOST:5432/hoa\" -tAc \"DROP TABLE IF EXISTS ${service}.__verify_probe;\" 2>/dev/null"
check "row-level security is FORCEd on ${service}'s primary table" bash -c \
"psql \"postgres://hoa_${service}_migrator@\$DATABASE_HOST:5432/hoa\" -tAc \"SELECT relforcerowsecurity FROM pg_class WHERE relname = '${service}_primary_table' AND relnamespace = '${service}'::regnamespace;\" | grep -q t"
echo "== Backup: archive lag and off-cluster repository =="
check "WAL archive lag under 120s" bash -c \
"[ \"\$(curl -fsS http://${service}.hoa.svc.cluster.local:9090/metrics | grep -oP 'hoa_wal_archive_lag_seconds \K[0-9.]+')\" -lt 120 ]"
check "repo2 has a full backup within the last 26 hours" \
python3 deploy/production-readiness/assert_recent_backup.py --repo=2 --max-age-hours=26
echo "== Disaster Recovery: latest drill met RPO and RTO =="
check "most recent drill-history.jsonl entry has both targets met" \
python3 deploy/production-readiness/assert_last_drill_passed.py \
--ledger=deploy/disaster-recovery/drill-history.jsonl
echo "== API Versioning: no route serves an unversioned response =="
check "every gateway route for ${service} declares a version prefix" \
python3 deploy/production-readiness/assert_versioned_routes.py --service="${service}"
echo "== Audit Logs: append-only grants and chain integrity =="
check "UPDATE and DELETE are revoked on ${service}.audit_log" bash -c \
"psql \"postgres://hoa_${service}_migrator@\$DATABASE_HOST:5432/hoa\" -tAc \"SELECT has_table_privilege('hoa_${service}_app', '${service}.audit_log', 'UPDATE');\" | grep -q f"
check "offsite audit chain verifies with zero breaks" \
go run -C tools/audit-verify . --topic="hoa.audit.${service}" --check-only
echo "== Feature Flags: rollback takes effect within the cache TTL =="
check "flag cache TTL is 30s or less" \
python3 deploy/production-readiness/assert_flag_ttl.py --service="${service}" --max-seconds=30
echo "== SLI/SLO: recording rules and burn-rate alerts are loaded =="
check "${service}-slo.yaml rules are active in Prometheus" bash -c \
"curl -fsS http://prometheus.hoa.svc.cluster.local:9090/api/v1/rules | grep -q \"${service}-charge-slo\""
echo
if [ "$failures" -eq 0 ]; then
echo "ALL CHECKS PASSED for $service"
exit 0
else
echo "$failures CHECK(S) FAILED for $service — not production ready"
exit 1
fiEvery single check calls out to the real system the corresponding lesson built — a live psql session, a live Prometheus query, the actual drill-history.jsonl ledger, the actual audit-chain verifier — rather than reading a config file that merely describes what should be true; a config-only check would pass even if the role was reverted, the drill never ran, or the chain was silently broken.
Make the Script a Deploy Gate, Not a Manual Step Before a Release
A verification script that exists but is run manually, occasionally, by whoever remembers, decays exactly like an unmonitored backup or an untested restore. Wiring it into the same pipeline that promotes a service to production is what keeps every one of these seven controls continuously true instead of true only on the day someone last ran the script by hand.
name: production-readiness-gate
on:
deployment:
workflow_dispatch:
inputs:
service:
required: true
jobs:
verify:
if: github.event.deployment.environment == 'production' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- name: Run production readiness verification
run: |
service="${{ github.event.deployment.payload.service || inputs.service }}"
deploy/production-readiness/verify.sh "$service"
env:
DATABASE_HOST: ${{ secrets.PRODUCTION_DATABASE_HOST }}
APP_PASSWORD: ${{ secrets.PRODUCTION_APP_PASSWORD }}
- name: Report deployment status
if: always()
uses: actions/github-script@v7
with:
script: |
await github.rest.repos.createDeploymentStatus({
owner: context.repo.owner,
repo: context.repo.repo,
deployment_id: context.payload.deployment.id,
state: "${{ job.status }}" === "success" ? "success" : "failure",
description: "production readiness gate",
});Gating on the deployment event itself — not a separate manual approval step — means a service cannot reach production through any path that skips verification, including a hotfix deploy under pressure, which is precisely the situation where a skipped check is most tempting and most dangerous.
verify.sh takes a service name and only asserts facts about that service's own schema, roles, backup repository, and SLO rules — it never assumes that because maintenance passed, notification or payment must also be compliant. Per-service isolation was the Security lesson's central argument, and a checklist that checked the platform as a whole rather than each service independently would silently reintroduce the exact blast-radius assumption that lesson eliminated.
Capstone: Bring services/notification-service Through Every Gate at Once
services/notification-service — a service that sends the SMS and email alerts other lessons across this course have referenced but never fully built out — is the platform's newest service candidate, and production-readiness for it is not seven separate stories; it is one integrated sequence where each lesson's control depends on the one before it existing correctly.
- Security first — bootstrap hoa_notification_owner/migrator/app/app_1 and enable FORCE ROW LEVEL SECURITY on notification.deliveries, because every later control assumes the role split and tenant isolation already hold.
- Backup and Disaster Recovery next — add notification to the pgBackRest stanza and confirm a drill against it produces a passing drill-history.jsonl entry, because an audit log or a flag rollout is worthless if the schema holding it cannot be restored.
- API Versioning and Audit Logs together — notification's v1 API and its append-only delivery-attempt log ship in the same PR, because every notification send is itself a privileged, auditable action from day one, not retrofitted later.
- Feature Flags and SLI/SLO last — gate notification's new push-notification channel behind a percentage flag whose rollout stage advance is blocked by notification's own burn-rate SLO, exactly as charges_v2_response was gated by payment's SLO, because a channel with no SLI has no way to know if its rollout is going badly.
Applied exercise
Extend verify.sh to catch a control that silently regressed
Six months after services/notification-service passed every check in this lesson, a well-intentioned refactor accidentally re-granted DELETE on notification.audit_log to hoa_notification_app while fixing an unrelated bug, and nobody noticed because verify.sh was last run manually the week notification launched.
- Identify exactly which check in verify.sh would have caught this regression if the script had been run, and explain precisely why it would fail.
- Given the production-readiness-gate.yml workflow only runs on a deployment event, explain one gap in coverage this leaves (a scenario where the regression could persist in production despite the gate existing) and propose a specific change to close it.
- Write the additional check (in the same style as the existing checks in verify.sh) that would need to be added if this exact regression pattern — a grant silently widened outside of a migration — were not already covered by an existing check.
- Explain why this regression is a stronger argument for the capstone's ordering (Security's role grants underpinning every later control) than for treating Audit Logs as an isolated concern.
Deliverable
A precise identification of the existing (or missing) check, a concrete proposal closing the deployment-event gap (such as a scheduled re-run in addition to the deploy-triggered one), any new bash check needed, and a short written argument connecting the regression back to the capstone's dependency ordering.
Completion checks
- The answer correctly traces the regression to the specific has_table_privilege check for DELETE on audit_log, or correctly identifies its absence.
- The proposed fix for the coverage gap adds recurring verification (e.g. a schedule trigger) rather than relying solely on the deployment-event trigger.
- The argument explicitly connects the regression to why Security's controls must be verified continuously, not merely once at onboarding, tying back to this lesson's capstone ordering.
Why does verify.sh query live systems — a real psql session, a real Prometheus endpoint, the actual drill-history.jsonl ledger — instead of reading a static configuration file describing what each control should be?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.