Stage 7 · Master
Phase 9 — Payment Service
Deployment
Ship payment code with secret rotation, migration-first upgrades, and readiness gates that survive webhook retries during rollout.
Provider Secrets Belong in Kubernetes Secrets and Rotate Under a Grace Window
Payment credentials are not configuration in the casual sense; they are signing material and API keys. Put them in a Kubernetes Secret, not a ConfigMap, and wire them through Helm values as references to existing secret keys. Rotation has one payment-specific twist: webhook senders may still sign in-flight retries with the old secret for a short window. If your new pods only accept the new secret immediately, you create a self-inflicted webhook outage during rotation.
func (p *RazorpayProvider) VerifyWebhookSignature(payload []byte, signatureHeader string) bool {
secrets := []string{p.currentWebhookSecret}
if p.previousWebhookSecret != "" {
secrets = append(secrets, p.previousWebhookSecret)
}
for _, secret := range secrets {
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(payload)
expected := hex.EncodeToString(mac.Sum(nil))
if hmac.Equal([]byte(expected), []byte(signatureHeader)) {
return true
}
}
return false
}Accept both the current and previous webhook secret during a short grace window, then remove the previous secret after the provider is confirmed to be signing only with the new value.
apiVersion: v1
kind: Secret
metadata:
name: payment-service-payment
type: Opaque
stringData:
PAYMENT_PROVIDER_KEY: rp_live_example_key
PAYMENT_PROVIDER_SECRET: rp_live_example_secret
PAYMENT_WEBHOOK_SECRET_CURRENT: whsec_current_example
PAYMENT_WEBHOOK_SECRET_PREVIOUS: whsec_previous_exampleThis is a Secret because anyone who can read it can impersonate your gateway client or forge webhook signatures.
payment:
provider: razorpay
existingSecretName: payment-service-payment
secretKeys:
apiKey: PAYMENT_PROVIDER_KEY
apiSecret: PAYMENT_PROVIDER_SECRET
webhookSecretCurrent: PAYMENT_WEBHOOK_SECRET_CURRENT
webhookSecretPrevious: PAYMENT_WEBHOOK_SECRET_PREVIOUS
webhookGraceWindow: 24h
migrations:
enabled: true
backoffLimit: 1
readiness:
path: /readyzThe chart should point at secret keys by name instead of inlining sensitive values into versioned YAML.
During the overlap window, the provider may still retry events signed with the previous secret. Rejecting them all at once looks like an attack in metrics but is actually a deployment mistake.
Payment Schema Migrations Must Finish Before Webhook Pods Receive Traffic
User Service already established the contract: migrations run in their own Job, and the rollout waits for it. Nothing about that changes here, so the chart reuses it. What is payment-specific is the consequence of getting the order wrong. Every other service answers a late migration with a 500 that a human eventually retries; here the *provider* retries, on its own schedule, for hours, against an endpoint that cannot yet record what it is being told. The Helm hook weight is therefore not a stylistic choice — it is what keeps a provider retry storm from starting.
annotations:
- "helm.sh/hook": pre-upgrade
+ "helm.sh/hook": pre-install,pre-upgrade
+ "helm.sh/hook-weight": "-5"
+ "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
- backoffLimit: 0
+ backoffLimit: {{ .Values.payment.migrations.backoffLimit }}
args:
- >-
- migrate -path /app/services/payment-service/db/migrations
+ migrate -path /app/services/payment-service/db/migrations
+ -verbose
-database "$DATABASE_URL"
upThree lines carry the whole lesson: pre-install covers a fresh cluster, hook-weight -5 puts the Job ahead of the workload update, and a non-zero backoffLimit lets a transient database blip retry instead of failing the release while the provider is already retrying against you.
Readiness Gates Turn Provider Retries Into a Safe Rolling-Update Buffer
A brief 5xx during rollout is survivable only because the gateway will retry and your inbox uniqueness constraint will dedupe the eventual success. That does not excuse sloppy readiness. New pods must not receive traffic until database connections are up, secrets are mounted, and the payment adapters have loaded both webhook secrets. Otherwise you convert a self-healing retry into a visible outage with a larger retry storm.
- Failure mode: starting new pods before migrations complete produces 5xx loops on live webhooks because the handler touches columns or tables that do not exist yet.
- Security concern: payment API keys and webhook secrets must live in Secret-backed env vars, never in ConfigMaps or Helm plain text committed to Git.
- Operational safety: dual-secret verification plus strict readiness means a short webhook retry burst during rollout is noisy but self-healing rather than data-losing.
Render, Apply, and Observe the Upgrade Like a Payment-Specific Change
| Rollout mistake | Who notices first | What the provider does | Recovery cost |
|---|---|---|---|
| Pods Ready before the migration Job finishes | The provider, not your dashboard | Retries the same signed events for hours | Low — the inbox dedupes the eventual success |
| New secret deployed without dual verification | The bad-signature alert, which looks like an attack | Keeps signing with the old secret until told otherwise | High — events rejected during the window are gone unless the provider replays |
| Readiness passing before secrets are mounted | Nobody, until reconciliation runs | Receives 500s and retries | Medium — recoverable, but the ledger disagrees until the nightly job |
Applied exercise
Rotate the webhook secret without dropping staged events
Security has issued a rotation ticket for the payment webhook secret, but product has a live payment promotion this evening and cannot tolerate webhook downtime.
- Describe the exact order in which you would add the new secret, deploy dual-secret verification, switch the provider to sign with the new secret, and finally remove the old one.
- List the metrics or logs you would watch during the grace window to confirm old signatures are tapering off instead of failing suddenly.
- Explain what would happen if you skipped the pre-upgrade migration job while deploying the same release.
- Write one rollback rule that keeps stale secrets from lingering longer than necessary.
Deliverable
A deployment runbook for rotating payment webhook secrets during live traffic, including sequencing, observability checks, and rollback notes.
Completion checks
- The plan keeps both old and new webhook secrets valid during the overlap window.
- Migration ordering is called out explicitly before new pods receive traffic.
- The rollback path preserves service availability without normalizing long-lived dual-secret drift.
Why should payment provider keys and webhook secrets be delivered through a Kubernetes Secret rather than a ConfigMap?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.