Caching Dependencies

Speeding npm, pip, Maven, Gradle, and Docker layer builds with keyed restore caches.

6 min readCI/CD & GitOpsPlatform

Why Caching Matters

Dependency installation is one of the slowest parts of a pipeline. Installing npm packages, pip dependencies, or Maven artifacts from the network can take minutes. Caching these dependencies between pipeline runs reduces build time dramatically — often by 50-80%.

The key to effective caching is using cache keys that change when dependencies change. A cache key based on the lock file hash ensures the cache is invalidated when dependencies are updated, preventing stale dependency issues.

GitHub Actions Cache

GitHub Actions caching patterns


jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Cache npm dependencies
      - name: Cache npm
        uses: actions/cache@v4
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            npm-${{ runner.os }}-

      - run: npm ci

      # Cache Next.js build
      - name: Cache Next.js
        uses: actions/cache@v4
        with:
          path: .next/cache
          key: nextjs-${{ runner.os }}-${{ hashFiles('src/**/*.ts') }}-${{ hashFiles('next.config.js') }}
          restore-keys: |
            nextjs-${{ runner.os }}-${{ hashFiles('src/**/*.ts') }}-
            nextjs-${{ runner.os }}-

      - run: npm run build

The npm cache stores the package manager cache directory, not node_modules. This is faster because npm does not need to download packages. The Next.js cache stores build artifacts for incremental builds.

Use setup-node built-in caching

The actions/setup-node action has built-in cache support. Add cache: npm to cache dependencies automatically without the explicit cache action. This is simpler and handles edge cases for you.

GitLab CI Cache

GitLab CI cache configurations


# Global cache for all jobs
default:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
      - .npm/
    policy: pull-push

build:
  stage: build
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
    policy: pull
  script:
    - npm ci
    - npm run build

test:
  stage: test
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
    policy: pull
  script:
    - npm test

# Multi-language caching
python-build:
  stage: build
  cache:
    key:
      files:
        - requirements.txt
    paths:
      - .venv/
  script:
    - python -m venv .venv
    - source .venv/bin/activate
    - pip install -r requirements.txt

GitLab CI cache key can be based on file contents using the files keyword. The policy keyword controls whether the job uploads (push), downloads (pull), or both. Use pull for jobs that only read the cache.

Docker Layer Caching

Docker layer caching with BuildKit


jobs:
  docker-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/myorg/myapp:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          build-args: |
            NODE_ENV=production

BuildKit GHA cache backend stores Docker layers in the GitHub Actions cache. mode=max caches all layers, not just the final image layers. This significantly speeds up builds when only a few layers change.

Cache Strategies

Lock-file hashing — use the hash of package-lock.json, requirements.txt, or go.sum as the cache key. This ensures exact dependency versions are cached.
Multi-level restore — use restore-keys with partial key matches. This allows cache hits even when the exact key does not match, using the closest available cache.
Split caches — separate caches for different dependency types (production vs dev, build vs test) to improve hit rates.
Cache rotation — set appropriate expiration times. Development caches can expire in days; release caches should persist longer.

Cache Pitfalls

Caches can cause subtle issues when not managed carefully. Stale caches cause phantom bugs. Corrupted caches cause mysterious failures. Overly broad cache keys waste storage and reduce hit rates.

Never cache node_modules with npm ci

npm ci deletes node_modules before installing. If you cache node_modules, npm ci removes it first. Instead, cache the npm cache directory (~/.npm on Linux) which stores downloaded packages. This is faster and avoids conflicts.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 5 · Platform

Building Pipelines