Stage 7 · Master
Phase 19 — Production Deployment
Secrets
Stop committing base64-encoded credentials as if they were encrypted, sync real database and signing-key values from an external secret manager, and rotate the migration credential without ever rebuilding an image.
A Kubernetes Secret Committed to Git Is a Plaintext Credential With Extra Steps
Organization Service created organization-db-credentials as a Secret and moved on, which was the right scope for that lesson. What it did not answer is where the value comes from. An early draft of this chart answered it the easy way — a YAML manifest with a base64 data: field — and that manifest was one review comment away from being committed alongside the Deployment. The single command below is the whole argument against it, and it is worth running once rather than reading about.
$ kubectl get secret organization-db-credentials -o jsonpath='{.data.password}' | base64 -d
local-only-password-that-was-about-to-become-a-real-oneAnyone with read access to the Git repository — not the cluster, the repository — would have had this password the moment the PR merged. The fix is not a stronger encoding; it is never writing the value to a file Git tracks at all.
A common workaround — generating the Secret YAML locally and applying it directly with kubectl apply -f, never git add-ing it — depends entirely on discipline that a hurried commit can defeat once. External Secrets Operator (below) removes the possibility structurally: no file containing the actual value ever exists in the repository, committed or not.
External Secrets Operator Syncs Real Values Into Kubernetes Secrets Automatically
External Secrets Operator (ESO) runs in-cluster and continuously syncs values from an external secret store — this platform uses AWS Secrets Manager — into native Kubernetes Secret objects. The only thing committed to Git is a reference to where the value lives, never the value itself.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: organization-db-credentials
namespace: {{ .Release.Namespace }}
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: organization-db-credentials
creationPolicy: Owner
data:
- secretKey: DATABASE_URL
remoteRef:
key: {{ .Values.environment }}/organization/db-app-role
- secretKey: PGPASSWORD
remoteRef:
key: {{ .Values.environment }}/organization/db-app-role
property: password{{ .Values.environment }} — already set to staging or production by Lesson 4's values files — resolves to a different secret path per environment automatically, so the same chart template can never accidentally sync production's credential into a staging Secret or vice versa.
Two Credentials for One Service: the API Role and the Migration Role
Lesson 6 introduced the principle: the api container's database role has no DDL privileges, and the migrate container's role is the only one that does. This is where that separation becomes concrete — two ExternalSecret resources, syncing two distinct database roles from the secret store, each mounted into exactly one of the two workloads.
| ExternalSecret | Remote key | Database role privileges | Mounted into |
|---|---|---|---|
| organization-db-credentials | {env}/organization/db-app-role | SELECT, INSERT, UPDATE, DELETE on application tables only | organization-api Deployment |
| organization-migrate-credentials | {env}/organization/db-migrate-role | CREATE, ALTER, DROP on organization's own schema, no DML on other services' tables | organization-migrate pre-upgrade hook Job only |
If a future SQL injection or dependency vulnerability in the api container's own code path is ever exploited, the attacker inherits the app-role's privileges — read and write application data, but no ability to create a new table, drop a constraint, or alter a column. The migrate-role credential, which could do those things, is never present in the long-running api Pod's environment at all.
GHCR Repositories Are Private; Pulling Them Requires Their Own Secret Too
organization-api and organization-migrate (Lesson 6) are published to private GHCR repositories, not public ones — a private repository requires every node that pulls the image to authenticate, which Kubernetes does via an imagePullSecrets reference on the Pod spec, itself synced by ESO rather than hand-created.
spec:
template:
spec:
imagePullSecrets:
- name: ghcr-pull-credentials # synced by its own ExternalSecret, scoped to packages:read only
containers:
- name: api
image: "{{ .Values.image.api.repository }}@{{ .Values.image.api.digest }}"ghcr-pull-credentials is deliberately scoped to packages:read on the organization image repositories only — it cannot push, delete, or read any other GHCR repository, so a leaked pull secret cannot be used to tamper with the images themselves.
Rotate the Migration Credential Without Touching an Image
A quarterly credential rotation replaces the migrate role's password in AWS Secrets Manager directly. ESO's refreshInterval: 1h means the in-cluster Secret updates on its own within an hour, with no Helm upgrade, no new image build, and no code change — the separation between application bytes (Lesson 6) and runtime configuration (this lesson) is what makes that possible.
organization-migrate-credentials is only read when the pre-upgrade hook Job runs during the next release; a rotated value sitting correctly in the Secret object proves ESO's sync worked, but does not by itself prove the new password is valid until an actual migration Job authenticates with it. The next scheduled or forced release is the real verification point.
Scan Git History Itself, Not Just the Current Working Tree
Finding: password: local-only-password-that-was-about-to-become-a-real-one
Secret: kubernetes-secret-value
File: charts/organization/templates/db-credentials-DRAFT.yaml
Commit: 3f9a21c
::error::gitleaks detected 1 leak(s)This exact scan, added to quality-gate.yaml as a required check, is what should have caught the near-miss described at the start of this lesson before merge — and now does, on every future pull request.
Applied exercise
Split auth's signing-key secret into a synced ExternalSecret
services/auth-service currently reads its JWT signing key from a manually created Secret, applied once by hand and never rotated since — nobody remembers the exact command used to create it.
- Write an ExternalSecret syncing auth's signing key from a per-environment path in the secret store.
- Confirm the signing key never appears in any committed YAML file, only a remoteRef path.
- Run gitleaks against the repository's full history and confirm no historical commit contains the old manually-created Secret's value.
- Describe, in one sentence, what would need to happen for a rotated signing key to actually take effect for already-issued tokens.
Deliverable
A committed ExternalSecret for auth's signing key, a clean gitleaks history scan, and a written note on rotation's effect on already-issued tokens.
Completion checks
- The ExternalSecret's remoteRef path varies by {{ .Values.environment }}, matching organization's pattern.
- gitleaks reports zero findings across full history, not just the current tree.
- The rotation note correctly identifies that already-issued tokens signed with the old key become unverifiable once the key rotates, unless a grace-period dual-key strategy is in place.
Why is a Kubernetes Secret's base64-encoded data field not a substitute for keeping the value out of Git?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.