Continuous Deployment to Kubernetes

Why We Automated the Last Mile

Once CI could prove that an image for api-gateway or tasks-service had been built and tested, leaving the final production deployment as a manual kubectl step stopped making sense. Manual deploys feel safe because a human is present, but they are mostly a source of inconsistency: wrong image tags, forgotten rollout checks, and deploy knowledge concentrated in whoever remembers the shell commands. Fieldwork's goal was straightforward. A merge to main should produce one immutable image digest, update the target Deployment, wait for rollout health, and surface failure immediately. If humans are involved, it should be because the rollout needs a decision, not because the workflow needs a typist.

We still kept the deployment path intentionally narrow. Only merged commits from main could ship, images were referenced by digest rather than mutable tags, and each service deployed independently. That prevented the classic CD failure mode where "latest" means different things on the registry and on the node cache, or where one service's deployment job quietly redeploys everything else because the workflow was written at the repository level instead of the service level.

The Deployment Path from Merge to Cluster

Fieldwork's deployment pipeline was push-based from GitHub Actions to Kubernetes. For this stage of the system, that was the right trade-off. The repository already owned the build, the manifests, and the service-level change detection. Adding a separate controller loop just to sync one cluster would have increased moving parts more than safety. The deployment workflow built and pushed an image, resolved its immutable digest, patched the appropriate Deployment image reference, waited for kubectl rollout status, and failed loudly if the rollout stalled or regressed. This is the part many teams skip: the deploy is not done when kubectl apply returns. It is done when the new pods are actually ready.

name: deploy-service

on:
  push:
    branches: [main]

jobs:
  deploy-tasks-service:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: build
        uses: docker/build-push-action@v6
        with:
          context: .
          file: ./services/tasks-service/Dockerfile
          push: true
          tags: ghcr.io/thesyscoder/fieldwork/tasks-service:${{ github.sha }}
      - uses: azure/setup-kubectl@v4
      - run: |
          kubectl -n fieldwork set image deployment/tasks-service \
            tasks-service=ghcr.io/thesyscoder/fieldwork/tasks-service@${{ steps.build.outputs.digest }}
          kubectl -n fieldwork rollout status deployment/tasks-service --timeout=180s

Rollback Had to Be Real, Not Ceremonial

A rollback plan that has never been exercised is just documentation theater. Fieldwork's CD path included two real rollback options. The first was Kubernetes-native history: kubectl rollout undo deployment/tasks-service if the current rollout failed or the new pods exposed issues immediately after readiness. The second was digest redeploy: pick the last known-good digest from the deployment workflow output and set the image back explicitly. The first is fast. The second is precise and still works when rollout history is noisy because of unrelated manifest changes.

We tested rollback on purpose. That mattered. Readiness and graceful shutdown protect the forward path, but they also influence how safely a rollback can happen under load. If an application cannot drain or re-enter service correctly, rolling back simply repeats the same availability mistakes in the opposite direction. CD without rollback drills is just optimistic automation.

kubectl -n fieldwork rollout undo deployment/tasks-service

kubectl -n fieldwork set image deployment/tasks-service \
  tasks-service=ghcr.io/thesyscoder/fieldwork/tasks-service@sha256:6d9e9d1d5a0f... \
  && kubectl -n fieldwork rollout status deployment/tasks-service --timeout=180s

Why We Did Not Start with Full GitOps

GitOps is a strong model, and Fieldwork may grow into it. We did not start there because the system had one primary cluster, one repository, and one small set of services whose deployment logic already lived in GitHub Actions. Adding Argo CD or Flux immediately would have introduced another controller, another reconciliation surface, and another debugging hop before the team had enough deployment complexity to justify that indirection. The cost was concrete: more components to secure, monitor, and teach before the current workflow had even proven inadequate.

• Trigger production deployment only from merged commits on main.
• Deploy immutable image digests, not mutable tags.
• Wait for rollout status and treat timeout as deployment failure, not a warning.
• Practice both rollback undo and explicit digest redeploy before you need them in anger.

What We Actually Shipped

Fieldwork shipped push-based continuous deployment from GitHub Actions to Kubernetes. A merge to main built one service image, resolved its digest, updated the matching Deployment, and waited for rollout completion. The workflow treated rollout health as part of deployment rather than as somebody else's problem after the YAML applied. That kept the deploy path short, explicit, and aligned with the service boundaries already present in the repository.

Just as important, rollback was part of the design from the beginning. We did not call the pipeline complete because it could move forward quickly. We called it complete because it could also move backward predictably. That is the practical difference between automation that looks impressive in a demo and automation you trust with a production API.