CI Pipeline for a Multi-Module Repo

Why Full-Repo CI Stopped Scaling

Fieldwork lives in one go.work repository, but it does not behave like one deployable unit. api-gateway, auth-service, and tasks-service each build, test, and ship independently. Running the full lint, test, and image build pipeline for every service on every pull request was defensible when the repository was tiny. Once the services accumulated their own integration tests, mocks, and Docker builds, it became wasted compute. A change to tasks-service's task filter logic should not wait for auth-service image assembly if no auth code or shared package was touched. Slow CI is not just annoying; it trains engineers to batch too many changes together so they pay the queue cost less often.

We rejected the other extreme too: fully separate repositories with fully separate pipelines. That would have removed selective CI complexity, but it would also have made shared contracts, synchronized refactors, and go.work development worse. The actual decision was to keep the monorepo and make CI understand its boundaries. In other words, teach automation the repository's architecture instead of flattening the architecture to suit weak automation.

How We Detected Affected Services

The hard part of selective CI is not the matrix job. It is defining what "affected" really means. A direct file change under services/tasks-service obviously affects tasks-service. But a change under services/shared/authz may affect both api-gateway and tasks-service if both import that package. Fieldwork solved this with a small mapping layer: direct service path changes mark that service dirty, and changes in shared modules fan out to the list of services that depend on them. This was explicit configuration, not a clever but fragile attempt to infer everything from git paths alone.

We considered dynamic dependency graph extraction from go list output inside CI. It was attractive because it sounded more automatic. We rejected it for the first iteration because CI needed a stable, reviewable mapping that the team could reason about quickly. A small checked-in script plus dependency map was easier to trust than a pipeline whose behavior changed with every transitive import graph detail. Automation is only helpful when engineers can predict it.

#!/usr/bin/env bash
set -euo pipefail

base_ref="${1:-origin/main}"
changed_files=$(git diff --name-only "$base_ref"...HEAD)
services=()

mark_service() {
  local service="$1"
  [[ " ${services[*]} " =~ " ${service} " ]] || services+=("$service")
}

while IFS= read -r file; do
  case "$file" in
    services/api-gateway/*)
      mark_service api-gateway
      ;;
    services/auth-service/*)
      mark_service auth-service
      ;;
    services/tasks-service/*)
      mark_service tasks-service
      ;;
    services/shared/authz/*)
      mark_service api-gateway
      mark_service tasks-service
      ;;
    services/shared/httpx/*)
      mark_service api-gateway
      mark_service auth-service
      mark_service tasks-service
      ;;
  esac
done <<< "$changed_files"

printf '%s\n' "${services[@]}" | jq -R . | jq -s '{service: .}'

Matrix Jobs Kept Service Work Isolated

Once the changed service list existed, matrix jobs handled the rest cleanly. Each affected service got its own lint, test, and image-build path, which meant failures stayed localized. A broken tasks-service test did not drown in logs from auth-service. More importantly, the pipeline structure mirrored ownership. Teams could look at one job and understand the exact binary under test instead of reading a monolithic 2,000-line CI log. Selective CI is partly about speed and partly about preserving signal.

name: fieldwork-ci

on:
  pull_request:

jobs:
  detect:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.detect.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - id: detect
        run: echo "matrix=$(./scripts/changed-services.sh origin/main)" >> "$GITHUB_OUTPUT"

  service-ci:
    needs: detect
    if: ${{ fromJson(needs.detect.outputs.matrix).service[0] != null }}
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJson(needs.detect.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.work
      - run: go test ./services/${{ matrix.service }}/...
      - run: docker build -f ./services/${{ matrix.service }}/Dockerfile .

What Stayed Global Even in Selective CI

Selective did not mean "nothing global ever runs." Some checks were intentionally repository-wide because their failure mode crosses service boundaries. go.work consistency checks, generated contract drift in api-gateway, and formatting or lint rules for shared tooling still ran globally or under their own trigger paths. The key was to distinguish shared invariants from service-local work. If a rule only protects one binary, scope it to that binary. If a rule protects the whole workspace contract, keep it global and accept the cost.

• Run service-local unit tests, Docker builds, and integration tests only for affected services.
• Run workspace-level checks like go work sync validation once per workflow when shared manifests change.
• Treat shared package edits as a fan-out event, not as proof that every service must rebuild forever.
• Keep the change-detection script checked in and code-reviewed like any other production logic.

What We Actually Shipped

Fieldwork shipped a selective CI workflow for its go.work repository. A checked-in detection script mapped changed paths and shared packages to affected services, then GitHub Actions fanned out lint, test, and build jobs only for those services. Global invariants still ran where appropriate, but we stopped paying full-repository cost for service-local changes.

That decision kept the monorepo benefits we wanted while removing the false simplicity of all-or-nothing pipelines. The lesson was not that every repository needs path-based CI. The lesson was that CI should understand the same boundaries humans already rely on. If the repo is modular, the pipeline should be modular too.