Stage 7 · Master
Phase 19 — Production Deployment
CI/CD
Build the api and migrate images exactly once per commit, scan and sign that one build, and promote the same signed digest from staging to production through Helm — never rebuild, and never let a rebuilt image silently replace the one that passed every gate.
Two Builds of the 'Same' Image Are Not Provably the Same Image
An earlier CI/CD lesson built organization's artifact job: build the api and migrate targets once, scan both with Trivy, generate a CycloneDX SBOM, smoke-test the running container, and save both images to .tar archives as a workflow artifact. That job answers 'is this commit's image trustworthy.' It has never answered a second, equally important question: how does a trustworthy image reach production without anyone rebuilding it — because a second build, even from identical source and a cache hit, is not cryptographically provable to produce identical bytes to the first.
publish:
needs: [artifact]
if: github.ref == 'refs/heads/main'
permissions: { contents: read, packages: write, id-token: write }
outputs:
api-digest: ${{ steps.resolve-api-digest.outputs.digest }}
migrate-digest: ${{ steps.resolve-migrate-digest.outputs.digest }}
steps:
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
with: { name: organization-images-${{ github.sha }} }
# docker load organization-api.tar / organization-migrate.tar, tag, push,
# resolve each pushed digest, cosign sign both — no docker build here.This lesson picks up exactly where that job's outputs leave off: two signed, pushed content digests — one for the api image's own repository, one for the migrate image's own, separate repository — and turns them into a running staging, then production, deployment without a third build ever happening.
Promote to Staging Automatically, Production Only With Approval
GitHub Environments give each deployment target its own required reviewers, its own secrets, and its own deployment history — staging deploys the moment publish succeeds; production requires a human to approve the exact same job using the exact same digests, not a re-triggered pipeline that could pick up a newer commit by accident.
promote-staging:
needs: [publish]
environment: staging
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: azure/setup-helm@fb453306b312929fd8b5f66a8f5d2d84b5a2e7d3 # v4.3.0
- name: Verify signatures before deploying anything
run: |
cosign verify --certificate-identity-regexp "https://github.com/hoa-platform/.+/.github/workflows/release.yaml@.+" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/hoa-platform/organization-api@${{ needs.publish.outputs.api-digest }}
cosign verify --certificate-identity-regexp "https://github.com/hoa-platform/.+/.github/workflows/release.yaml@.+" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/hoa-platform/organization-migrate@${{ needs.publish.outputs.migrate-digest }}
- name: Deploy the exact verified digests to staging
run: |
helm upgrade --install organization charts/organization \
-n hoa-staging --create-namespace \
-f charts/organization/values.yaml \
--set image.api.digest=${{ needs.publish.outputs.api-digest }} \
--set image.migrate.digest=${{ needs.publish.outputs.migrate-digest }} \
--atomic --timeout 3m
promote-production:
needs: [publish, promote-staging]
environment: production # GitHub blocks this job here until platform-leads approves
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: azure/setup-helm@fb453306b312929fd8b5f66a8f5d2d84b5a2e7d3 # v4.3.0
- name: Verify signatures again — never trust a value passed between jobs without re-checking it
run: |
cosign verify --certificate-identity-regexp "https://github.com/hoa-platform/.+/.github/workflows/release.yaml@.+" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
ghcr.io/hoa-platform/organization-api@${{ needs.publish.outputs.api-digest }}
- name: Deploy the identical digests staging just ran — no build-arg differs, no image differs
run: |
helm upgrade --install organization charts/organization \
-n hoa-production \
-f charts/organization/values.yaml -f charts/organization/values-production.yaml \
--set image.api.digest=${{ needs.publish.outputs.api-digest }} \
--set image.migrate.digest=${{ needs.publish.outputs.migrate-digest }} \
--atomic --timeout 5mpromote-production depends on both publish and promote-staging, and uses the exact same needs.publish.outputs.api-digest value staging just deployed — there is no separate build-arg, no separate tag, and no code path that could cause production to run bytes staging never ran.
needs.publish.outputs.api-digest is a plain string passed between jobs — nothing about GitHub Actions' job-output mechanism cryptographically guarantees it was not tampered with in a compromised intermediate step, or that the digest even still corresponds to a signed image if something between jobs went wrong. Re-running cosign verify in promote-production, immediately before the production deploy, costs one API call and closes that gap rather than assuming a check performed one job earlier still holds.
Separate Repositories and Separate Secrets Prevent One Component From Widening the Other's Blast Radius
ghcr.io/hoa-platform/organization-api and ghcr.io/hoa-platform/organization-migrate are two independent GHCR package repositories, not two tags in one repository. Each has its own access policy: the migrate image never needs to be pulled by anything except the release pipeline's own migration Job, so it can be restricted to a narrower packages:read scope than the api image, which every environment's Deployment pulls continuously.
| Component | Repository | Who pulls it | Secret it needs at runtime |
|---|---|---|---|
| organization-api | ghcr.io/hoa-platform/organization-api | Every Deployment replica, continuously | organization-db-credentials (read/write DB role) |
| organization-migrate | ghcr.io/hoa-platform/organization-migrate | Only the pre-upgrade hook Job, once per release | organization-migrate-credentials (DDL-privileged DB role, never granted to the api image's role) |
The api container's DB credential grants INSERT/UPDATE/SELECT/DELETE on application tables but no DDL privileges — a SQL injection or an application bug in the api container cannot alter the schema. The migrate container's credential is the only one with CREATE TABLE/ALTER TABLE privileges, and it is only ever mounted into the short-lived hook Job, never into the long-running api Deployment. Lesson 7 builds this credential separation in full.
Confirm Production Runs the Exact Digest Staging Ran, Not a Coincidentally Similar One
Applied exercise
Add a promote-staging and promote-production job for services/gateway-service
gateway's artifact and publish jobs exist from an earlier lesson, but nothing promotes its published digest to any running environment yet.
- Write promote-staging and promote-production jobs mirroring organization's structure, using gateway's own chart from Lesson 4's exercise.
- Ensure promote-production depends on both publish and promote-staging.
- Re-verify the cosign signature in promote-production even though promote-staging already verified it.
- After a full run, diff the deployed image reference between staging and production and confirm it is empty.
Deliverable
Committed promote-staging and promote-production jobs, a completed workflow run, and a confirmed empty diff between staging's and production's deployed image references.
Completion checks
- production's job requires the production GitHub Environment's approval gate before running.
- The exact same digest string is used in both the staging and production helm upgrade commands.
- cosign verify runs in both jobs, not only in promote-staging.
Why does promote-production run cosign verify again on the same digest that promote-staging already verified, instead of trusting that check?
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.