Stage 7 · Master
Phase 4 — Authentication Service
Deployment
Turn the kid-based key rotation designed in the jwt lesson into an operational runbook: a zero-downtime dual-key rollout for auth-service, driven by Kubernetes Secrets rather than a redeploy-and-hope.
Rotating a Signing Key Is Not a Normal Rolling Update
A normal auth-service deploy — say, shipping the RBAC middleware from earlier in this module — is a routine rolling update: new pods come up, old ones drain, nothing about what a valid token looks like changes mid-rollout. Rotating the Ed25519 signing key is different: for a window of up to 15 minutes (one access-token TTL), tokens signed by the old key and the new key must both verify successfully everywhere, or users holding a token signed moments before the switch get spuriously logged out.
Every verifying service holds a map of kid to public key, not a single key. Rotation is therefore a data change (add a key to the map, later remove an old one) layered on top of, but decoupled from, a code deploy — the runbook below is mostly about sequencing that data change safely across every service that verifies tokens.
Keys Live in a Kubernetes Secret, Loaded at Startup
apiVersion: v1
kind: Secret
metadata:
name: auth-signing-keys
namespace: hoa-platform
type: Opaque
stringData:
active-kid: "2025-01-a"
keys.json: |
{
"2025-01-a": { "private": "<base64 ed25519 private key>", "public": "<base64 ed25519 public key>" }
}keys.json intentionally holds every key still valid for verification, keyed by kid, while active-kid names which one Issue() should currently sign with — the same two-field shape as the in-process KeySet struct from the jwt lesson, just persisted.
Five Steps, Each Independently Safe to Pause On
- 1. Generate a new Ed25519 keypair and add it to keys.json under a new kid, alongside the existing key — active-kid is unchanged. Apply the Secret and roll every verifying service (auth-service, user-service, and any future gateway) so all of them now accept both keys.
- 2. Confirm via each service's /healthz or a dedicated /debug/keys endpoint that the new kid is present in every running pod's PublicKeys map before proceeding — a service still missing the new key would reject tokens the new signing key is about to start issuing.
- 3. Update active-kid to the new key's kid and roll auth-service specifically. From this moment, newly issued tokens carry the new kid, while tokens issued under the old key (still up to 15 minutes old) continue verifying everywhere from step 1.
- 4. Wait at least one full access-token TTL (15 minutes) plus a safety margin — this platform uses 30 minutes — before touching the old key at all, guaranteeing every token signed under it has naturally expired.
- 5. Remove the old key from keys.json entirely and roll every verifying service one final time. Only now is the old private key considered fully retired and safe to destroy.
If active-kid is switched (step 3) before every verifying service has actually picked up the new key (step 1-2), users receiving a token signed with the brand-new key would be rejected by any service that hasn't rolled yet — the exact outage this whole runbook exists to avoid. Step 2's explicit check exists because a Kubernetes rolling update completing does not, by itself, guarantee every pod has re-read a mounted Secret without an explicit reload mechanism.
Rehearsing the Rotation Against a Local Cluster
Applied exercise
Write the automated check that gates step 3
Step 3 of the runbook (switching active-kid) currently depends on a human manually confirming step 2 via curl against each pod — error-prone under time pressure during an actual rotation.
- Design a script or CI job that queries every running auth-service and user-service pod's /debug/keys endpoint and fails loudly if any pod is missing the new kid.
- State what this check should do if a pod is mid-restart and briefly unreachable — treat it as a hard failure, or retry with a bounded timeout?
- Propose where this check belongs: a manual pre-flight run by an operator, or a required automated gate in the deployment pipeline before active-kid can be changed.
Deliverable
A written design (or actual script) for the automated key-propagation check, with an explicit decision on pipeline placement.
Completion checks
- The check queries actual running pods rather than assuming rollout status alone implies the Secret was re-read.
- The pipeline-placement decision is explicit, not left as 'someone should check this'.
Why must at least one full access-token TTL plus a safety margin elapse between switching active-kid and removing the old key from every verifier's map?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.