Why Our CI Pipeline Took 40 Minutes and What We Actually Cut to Get to 8

40 minutes from push to a merged PR having a green check. That was our pipeline six months ago. Nobody could tell me exactly why — the general answer was always "tests are slow" or "it's just a big monorepo." Both were true and both were almost irrelevant. The real breakdown, once I actually timed every single step instead of accepting the vague consensus, looked nothing like what anyone assumed.

Step one: stop guessing, start timing

Before changing anything, I added timing output to every discrete step in the workflow and pulled 20 recent runs. The actual breakdown surprised almost everyone on the team, including me.

Fix #1: Docker layer caching (14 min → 3 min)

We were building the image from a cold cache on every run — no --cache-from, no registry cache, nothing persisted between runs. Every single build reinstalled every OS package and every dependency layer from scratch, every time, even when only application code had changed.

- name: Build image
  run: docker build -t app:latest .

- uses: docker/setup-buildx-action@v3

- uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    tags: ghcr.io/org/app:${{ github.sha }}
    cache-from: type=registry,ref=ghcr.io/org/app:buildcache
    cache-to: type=registry,ref=ghcr.io/org/app:buildcache,mode=max

This alone cut the build step from 14 minutes to about 3 — the only work left on a typical PR was rebuilding the top layer or two where application code actually changed.

Fix #2: Parallelize the test suite (11 min → 4 min)

We ran the full test suite sequentially on a single runner. GitHub Actions supports matrix jobs natively — we just weren't using them. Splitting by package (this is a Go monorepo with ~14 internal packages) and running 4 shards in parallel cut wall-clock test time to roughly the slowest single shard instead of the sum of all of them.

strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: go test ./... -run . -shard=${{ matrix.shard }}/4

Fix #3: Actually cache dependencies correctly (9 min → 90 sec)

We had a dependency cache configured — but keyed only on the branch name, not on the lockfile hash. Every push invalidated it, so it was effectively caching nothing. actions/cache needs a key that changes exactly when the thing you're caching changes, not more, not less.

- uses: actions/cache@v4
  with:
    path: ~/go/pkg/mod
    key: ${{ runner.os }}-${{ github.ref }}

- uses: actions/cache@v4
  with:
    path: ~/go/pkg/mod
    key: ${{ runner.os }}-gomod-${{ hashFiles('go.sum') }}
    restore-keys: |
      ${{ runner.os }}-gomod-

Fix #4: Don't run everything on every PR

A docs-only PR was running the full Docker build, the full test matrix, and a deploy-preview step — none of which touch anything a docs change affects. We added path filters so jobs only run when relevant files change.

on:
  pull_request:
    paths-ignore:
      - 'docs/**'
      - '**.md'

The result

flowchart TB
    subgraph before["Before — one runner, sequential, 40 min"]
        direction LR
        A1[checkout + install\n9 min] --> A2[docker build\nno cache · 14 min] --> A3[tests\nsequential · 11 min] --> A4[lint + types\n4 min] --> A5[deploy\n2 min]
    end
    subgraph after["After — cached + parallel runners, ~8 min"]
        direction LR
        B1[checkout + install\ndep cache · 1.5 min] --> B2{fan out}
        B2 --> B3[docker build\nregistry cache · 3 min]
        B2 --> B4[tests\n4-way shard · 4 min]
        B2 --> B5[lint + types\n3 min]
        B3 --> B6[deploy\n2 min]
        B4 --> B6
        B5 --> B6
    end

If your pipeline feels slow and nobody can point to exactly why, that's the tell. Add real per-step timing before you touch anything — the fix is usually obvious once you can see where the minutes actually are.