Stage 7 · Master
Phase 20 — Production Readiness
Feature Flags
Ship the v2 charge API and every future risky change behind a typed, auditable flag that can roll back in seconds instead of a redeploy.
The API Versioning lesson gave maintenance a safe path for a breaking response-shape change, but the rollout itself — turning v2 on for real tenants — still happened by deploying code. A feature flag separates those two events: deploying v2's handler and enabling it for any given tenant become independent decisions, so a bad rollout is a config change measured in seconds, not a rollback deploy measured in minutes. This lesson builds a typed flag store backed by Postgres and Redis, ties every flip to the audit log from the previous lesson, and defines the staged rollout that makes 'ship dark, enable gradually' the default instead of the exception.
Model Flags as Typed Values, Not Free-Text Strings
A flag store that returns string and leaves parsing to every call site invites exactly the bug class it should prevent: a typo in "tru", a percentage stored as "50" and compared as a string, a boolean flag silently reinterpreted as a rollout percentage by a caller who assumed the wrong shape. Modeling every flag as one of a small closed set of typed variants makes those mistakes fail at compile time or at flag-registration time, not in production traffic.
package flags
import (
"context"
"github.com/hoa-platform/backend/pkg/tenant"
)
// Kind is a closed set on purpose: adding a new flag shape means
// touching this file and every switch over it, which is the point —
// it surfaces every place in the codebase that needs to learn about
// the new shape instead of letting it silently fall through a default.
type Kind string
const (
KindBoolean Kind = "boolean"
KindPercentage Kind = "percentage" // 0-100, stable hash of tenant ID
KindVariant Kind = "variant" // one of a named enum, e.g. "control"/"treatment"
)
type Definition struct {
Key string
Kind Kind
Description string
Variants []string // only meaningful when Kind == KindVariant
}
type Store interface {
// BooleanEnabled panics if key was not registered with KindBoolean —
// a caller asking a percentage flag "is this on" is a bug, not a
// runtime maybe.
BooleanEnabled(ctx context.Context, tenantID tenant.ID, key string) (bool, error)
PercentageEnabled(ctx context.Context, tenantID tenant.ID, key string) (bool, error)
Variant(ctx context.Context, tenantID tenant.ID, key string) (string, error)
}The three Kind values map exactly to the three rollout shapes this platform actually needs — a hard on/off, a gradual percentage ramp, and an A/B variant — and nothing else is supported, which keeps every flag's evaluation logic simple enough to reason about during an incident.
Postgres for Durable Definitions, Redis for the Request-Path Read
Every flag evaluation sits on maintenance's request path, so it cannot afford a Postgres round trip per request — but the flag's current state must survive a Redis flush and must never silently drift out of sync with what an operator believes they set. Postgres stays the durable source of truth an operator writes to directly; a Redis cache, invalidated on every write, serves the hot path without a network round trip to the primary database on every request.
CREATE TABLE maintenance.flag_state (
flag_key text NOT NULL,
organization_id uuid, -- NULL row is the platform-wide default
kind text NOT NULL CHECK (kind IN ('boolean', 'percentage', 'variant')),
boolean_value boolean,
percentage smallint CHECK (percentage BETWEEN 0 AND 100),
variant_value text,
updated_by uuid NOT NULL,
updated_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (flag_key, organization_id)
);
-- A NULL organization_id row cannot be duplicated per flag_key — this
-- is the platform default every tenant falls back to before any
-- tenant-specific override exists.
CREATE UNIQUE INDEX flag_state_default_idx
ON maintenance.flag_state (flag_key)
WHERE organization_id IS NULL;
REVOKE DELETE ON maintenance.flag_state FROM PUBLIC;
GRANT SELECT, INSERT, UPDATE ON maintenance.flag_state TO hoa_maintenance_app;
DELETE is revoked the same way it was on audit_log: rolling a flag back to the platform default is an UPDATE that sets an explicit state, never a row deletion — so there is always a current, queryable row explaining exactly what a tenant's flag state is, not an absence that could mean 'default' or could mean 'never configured.'
DROP TABLE IF EXISTS maintenance.flag_state;
Dropping the table takes flag_state_default_idx and the REVOKE/GRANT state with it, leaving nothing behind for this migration to clean up separately.
package flags
import (
"context"
"errors"
"fmt"
"hash/fnv"
"time"
sharedflags "github.com/hoa-platform/backend/pkg/flags"
"github.com/hoa-platform/backend/pkg/tenant"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/redis/go-redis/v9"
)
type Store struct {
pool *pgxpool.Pool
redis *redis.Client
}
func New(pool *pgxpool.Pool, redis *redis.Client) *Store {
return &Store{pool: pool, redis: redis}
}
// PercentageEnabled hashes tenantID against the flag key so a given
// tenant's outcome is stable across requests at the same rollout
// percentage — a tenant already in the enabled bucket at 20% never
// flips back out purely from re-hashing on the next request, which
// would look like a flaky rollout rather than a deliberate ramp.
func (s *Store) PercentageEnabled(ctx context.Context, tenantID tenant.ID, key string) (bool, error) {
percentage, err := s.percentageFor(ctx, tenantID, key)
if err != nil {
return false, err
}
h := fnv.New32a()
_, _ = h.Write([]byte(key + ":" + tenantID.String()))
bucket := h.Sum32() % 100
return uint32(percentage) > bucket, nil
}
func (s *Store) percentageFor(ctx context.Context, tenantID tenant.ID, key string) (int, error) {
cacheKey := fmt.Sprintf("flag:%s:%s:pct", key, tenantID.String())
if cached, err := s.redis.Get(ctx, cacheKey).Result(); err == nil {
var pct int
if _, scanErr := fmt.Sscanf(cached, "%d", &pct); scanErr == nil {
return pct, nil
}
}
const query = `
SELECT coalesce(
(SELECT percentage FROM maintenance.flag_state WHERE flag_key = $1 AND organization_id = $2),
(SELECT percentage FROM maintenance.flag_state WHERE flag_key = $1 AND organization_id IS NULL),
0
)`
var pct int
if err := s.pool.QueryRow(ctx, query, key, tenantID.String()).Scan(&pct); err != nil {
return 0, fmt.Errorf("read flag percentage: %w", err)
}
_ = s.redis.Set(ctx, cacheKey, pct, 30*time.Second)
return pct, nil
}
var ErrKindMismatch = errors.New("flag registered with a different kind than requested")The Redis TTL is deliberately short — 30 seconds — rather than cached forever and invalidated on write: a short TTL bounds the worst case ('how long can a flag flip take to reach every replica') to a fixed number even if an invalidation message is ever dropped, instead of depending on cache invalidation always succeeding.
Stage the Rollout and Audit Every Flip Like a Privileged Action
A flag flip is exactly as privileged an action as the credential rotation the Audit Logs lesson covered — it changes production behavior for real tenants without a deploy — so it goes through the same audit.Record call, and it never jumps straight from 0% to 100% for a change with the blast radius of the v2 charge API.
- Stage 1 — internal tenants only (the platform team's own HOA record), percentage flag pinned at 0% with an explicit allowlist override for internal organization_id values.
- Stage 2 — 5% of tenants, held for at least one full SLO observation window (the SLI/SLO lesson's burn-rate alerting must stay green throughout).
- Stage 3 — 50%, held long enough to observe a full weekly traffic cycle, including the highest-volume day.
- Stage 4 — 100%, followed by the API Versioning lesson's deprecation clock starting on the old path, not before.
package admin
import (
"context"
"encoding/json"
"fmt"
"net/http"
"github.com/hoa-platform/backend/services/maintenance-service/internal/audit"
"github.com/hoa-platform/backend/pkg/tenant"
"github.com/jackc/pgx/v5/pgxpool"
)
type setPercentageRequest struct {
FlagKey string `json:"flag_key"`
Percentage int `json:"percentage"`
}
// SetPercentage is the only write path to maintenance.flag_state's
// percentage column. It requires an operator role checked upstream by
// the gateway's admin-scope middleware, and it writes an audit_log
// entry in the same transaction as the flag update, so a flip and its
// record can never diverge — one commits, or neither does.
func (h *FlagHandler) SetPercentage(w http.ResponseWriter, r *http.Request) {
var req setPercentageRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "invalid request body", http.StatusBadRequest)
return
}
if req.Percentage < 0 || req.Percentage > 100 {
http.Error(w, "percentage must be between 0 and 100", http.StatusBadRequest)
return
}
tenantID := tenant.From(r.Context())
actorID := operatorIDFrom(r)
tx, err := h.pool.Begin(r.Context())
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
defer func() { _ = tx.Rollback(context.Background()) }()
var before int
_ = tx.QueryRow(r.Context(),
`SELECT coalesce(percentage, 0) FROM maintenance.flag_state
WHERE flag_key = $1 AND organization_id = $2`,
req.FlagKey, tenantID.String(),
).Scan(&before)
const upsert = `
INSERT INTO maintenance.flag_state (flag_key, organization_id, kind, percentage, updated_by)
VALUES ($1, $2, 'percentage', $3, $4)
ON CONFLICT (flag_key, organization_id)
DO UPDATE SET percentage = $3, updated_by = $4, updated_at = now()`
if _, err := tx.Exec(r.Context(), upsert, req.FlagKey, tenantID.String(), req.Percentage, actorID); err != nil {
http.Error(w, "failed to update flag", http.StatusInternalServerError)
return
}
if err := audit.Record(r.Context(), tx, tenantID, audit.Entry{
ActorUserID: actorID,
Action: "flag.percentage_changed",
ResourceType: "feature_flag",
ResourceID: fmt.Sprintf("%s:%s", req.FlagKey, tenantID.String()),
Before: map[string]int{"percentage": before},
After: map[string]int{"percentage": req.Percentage},
}); err != nil {
http.Error(w, "failed to record audit entry", http.StatusInternalServerError)
return
}
if err := tx.Commit(r.Context()); err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}Writing the flag update and the audit entry in the same transaction is what the Audit Logs lesson's 'one function, not ad hoc inserts' principle requires in practice — a handler that updated the flag and then audited it in a separate transaction could commit the flip while losing the record if the process crashed in between.
The speed advantage of a feature flag over a redeploy only holds if the risky code path checks the flag on every single request, not once at process startup. Caching PercentageEnabled's result for the lifetime of a server process — rather than per-request, as the 30-second Redis TTL above does — turns 'roll back in seconds' back into 'roll back after every replica happens to restart,' which defeats the entire reason this lesson avoided a redeploy in the first place.
Applied exercise
Design a variant flag for a payment retry strategy experiment
services/payment-service/internal/payment wants to compare two retry backoff strategies for failed card charges — 'fixed-delay' versus 'exponential-backoff' — as a KindVariant flag, with the choice sticky per tenant for the life of the experiment.
- Write the flag_state row(s) needed to run this as a 50/50 split, including how the schema you designed for percentage flags would need to differ (or not) to support a two-value variant instead.
- Explain why FNV-hashing tenantID against the flag key (as PercentageEnabled does) is also the right approach for keeping a tenant's variant assignment stable, and adapt it to choose between exactly two named variants instead of a 0-100 threshold.
- Write the audit.Record call site for changing which variant a specific tenant is pinned to, including believable before/after JSON.
- State one reason a variant flag's audit trail matters more for a payment-retry experiment than it did for the charges_v2_response rollout in this lesson.
Deliverable
A flag_state schema note (reusing or extending the migration from this lesson), a hashing scheme mapped to two variants, a filled-in audit.Record call, and a short written justification.
Completion checks
- The variant assignment is deterministic per tenant using the same stable-hash principle as the percentage flag, not re-randomized per request.
- The audit entry captures which variant a tenant moved from and to, not merely 'variant changed.'
- The justification correctly identifies that a payment-affecting experiment has direct financial and compliance consequences that a UI-shape change does not.
Why does PercentageEnabled hash the tenant ID together with the flag key instead of generating a fresh random number on every request?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.