Stage 7 · Master
Security Hardening
Dependency & Image Scanning
Catch vulnerable Go modules and container layers in CI, while the fix is still a PR instead of an incident ticket.
Why We Added Scanning Before the First Report
Fieldwork already had dependency risk the moment it had more than one Go module, a Docker base image, and CI pulling tools from the network. Waiting for the first vulnerability report would have meant teaching the team the wrong operating model: ship first, inventory later. We wanted the opposite. Vulnerability review should happen in the same place as unit tests and lint failures — inside the pull request that introduced the risk.
The alternative we rejected was a nightly 'security scan' workflow that emailed a report after the fact. That sounds operationally gentle, but it pushes ownership away from the author and toward whoever reads the report next morning. By then the change may already be merged, deployed, and referenced by a release candidate. Late visibility turns a cheap patch into coordination work.
The author of the PR is the cheapest person to fix the problem. That is why Fieldwork runs dependency and image scanning in the same CI path as tests, not as a disconnected compliance job.
Scanning Go Modules in CI
For Go code, we kept the first pass deliberately boring: go mod tidy to keep manifests honest, govulncheck for reachable vulnerability analysis, and a lockstep policy that every service scans its own module rather than trusting the workspace root to tell the whole story. A go.work repo makes it easy to forget that each module can drift independently.
name: ci
on:
pull_request:
push:
branches: [main]
jobs:
govulncheck:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
module:
- api-gateway
- auth-service
- tasks-service
- notifications-service
defaults:
run:
working-directory: services/${{ matrix.module }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: services/${{ matrix.module }}/go.mod
- name: Restore module cache
uses: actions/cache@v4
with:
path: |
~/go/pkg/mod
~/.cache/go-build
key: ${{ runner.os }}-${{ matrix.module }}-${{ hashFiles(format('services/{0}/go.sum', matrix.module)) }}
- name: Keep dependencies honest
run: |
go mod tidy
git diff --exit-code go.mod go.sum
- name: Scan reachable vulnerabilities
run: go run golang.org/x/vuln/cmd/govulncheck@latest ./...govulncheck was chosen over a plain manifest scanner for the first gate because it understands whether vulnerable symbols are actually reachable from compiled code. That cut false positives enough that people kept reading the output.
| Approach | What it catches well | What it misses or costs |
|---|---|---|
govulncheck in each Go module | Reachable vulnerable code in compiled packages | Needs a real module context; won't tell you about OS packages in the image |
| Manifest-only scanners | Broad package inventory across ecosystems | Often noisy because they ignore whether code paths are reachable |
| Nightly out-of-band scan | Large estate reporting | Feedback arrives after merge, so remediation cost is higher |
We still kept Dependabot-style updates, but not as the primary control. Automated PRs are great for drift reduction. They are not enough for release gating because they do nothing for a vulnerable module you intentionally pinned yesterday.
Scanning the Built Image, Not Just the Dockerfile
The second decision was equally important: scan the actual image digest we intend to ship, not only the Dockerfile or source tree. Vulnerabilities often arrive through the base image, shared C libraries, package manager metadata, or build-time leftovers that never appear in Go dependency output. A clean go.mod does not mean a clean container.
# services/tasks-service/Dockerfile
FROM golang:1.24.4-alpine3.22 AS build
WORKDIR /src
COPY go.work ./
COPY services/tasks-service/go.mod services/tasks-service/go.sum ./services/tasks-service/
RUN cd services/tasks-service && go mod download
COPY services/tasks-service ./services/tasks-service
RUN cd services/tasks-service && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" -o /out/tasks-service ./cmd/tasks-service
FROM gcr.io/distroless/static-debian12:nonroot@sha256:8d4106c6e4b9f7fd2e4d5ce9a0d46ef0c284bb3c0f7fb0f00baf7a7f3a90c50e
COPY --from=build /out/tasks-service /tasks-service
USER nonroot:nonroot
ENTRYPOINT ["/tasks-service"]Pinning the runtime image by digest matters because 'distroless latest' is not a stable thing to approve. The scanner, the reviewer, and the deploy pipeline all need to reason about one immutable artifact.
image-scan:
runs-on: ubuntu-latest
needs: [govulncheck]
permissions:
contents: read
security-events: write
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- name: Build image
run: |
docker build -f services/tasks-service/Dockerfile -t fieldwork/tasks-service:${{ github.sha }} .
- name: Trivy scan
uses: aquasecurity/trivy-action@0.28.0
with:
image-ref: fieldwork/tasks-service:${{ github.sha }}
format: sarif
output: trivy-results.sarif
severity: CRITICAL,HIGH
ignore-unfixed: true
vuln-type: os,library
scanners: vuln
- name: Upload SARIF
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: trivy-results.sarifWe scan the built artifact before pushing it anywhere. That keeps the registry from becoming a staging area for images we already know we should not promote.
Distroless reduced package surface area for Fieldwork, which helped, but it did not remove the need for scanning. The point of shrinking the image was to reduce attack surface and noise, not to outsource judgment to the base image vendor.
Making Failures Actionable Instead of Noisy
A scanner people ignore is worse than no scanner because it creates a fake sense of coverage. We tuned Fieldwork's policy around two ideas: fail fast on findings that are both severe and relevant, and document every exception in the repo with an expiration date. We rejected the 'block on every CVE' policy because it collapses once the first unfixed base-image advisory shows up in a dependency you cannot meaningfully replace that day.
# .trivy.yaml
severity:
- HIGH
- CRITICAL
ignore-unfixed: true
exit-code: 1
scan:
security-checks:
- vuln
vulnerability:
type:
- os
- library
ignorefile: .trivyignoreignore-unfixed: true was a practical decision, not a philosophical one. If a distro has not shipped a patched package yet, blocking every PR does not improve security. It just trains engineers to search for bypasses.
# .trivyignore
# CVE-2026-10412 — openssl advisory inherited from build image only.
# Not present in distroless runtime image. Remove by 2026-08-15 after builder refresh.
CVE-2026-10412
# GO-2026-3014 — vulnerable code path not imported by notifications-service.
# Re-check when webhook delivery is enabled.
GO-2026-3014An ignore file is acceptable only when it behaves like debt with a due date. Anonymous, permanent suppression is how scanners become theater.
| Policy choice | Why we kept it | What we avoided |
|---|---|---|
| Fail PRs on HIGH/CRITICAL findings | Serious issues are visible while authors still own the code | The false urgency of medium-severity noise on every change |
| Allow expiring exceptions in-repo | Reviewers can see the justification beside the code it affects | A secret spreadsheet of ignored findings nobody audits |
| Scan both modules and images | Go packages and OS layers fail differently | Blind spots caused by assuming one scanner understands the entire artifact |
The Release Gate We Kept
The final release gate was intentionally modest: the build cannot publish a service image unless module scanning and image scanning both pass, or there is an explicit reviewed exception in the repository. We did not add a separate security sign-off queue because that would have recreated the same ownership gap we were trying to eliminate.
- Run vulnerability checks in the pull request path so the author sees them while the change is still small.
- Scan each Go module in a
go.workrepo independently; workspace-level assumptions hide drift. - Build and scan the exact image digest you intend to publish, including OS packages and copied artifacts.
- Keep scanner policy strict enough to matter and narrow enough that engineers still trust the signal.
Fieldwork chose PR-time ownership over after-the-fact reporting. Everything else — govulncheck, image scans, pinned digests, exception files — exists to support that one operational decision.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.