Pipeline Anatomy

What Is a Pipeline?

A CI/CD pipeline is a automated sequence of steps that takes code from a commit to a deployed artifact. It validates, builds, tests, and releases software without manual intervention. Every push to a repository triggers the pipeline, which runs a defined set of jobs and reports back with success or failure.

Pipelines are the backbone of modern software delivery. They enforce consistency, catch regressions early, and provide a single source of truth for how software is built and released. Without a pipeline, teams rely on manual steps that are slow, error-prone, and impossible to audit.

Stages, Jobs, and Steps

Pipelines are organized into a hierarchy: stages contain jobs, and jobs contain steps. Understanding this hierarchy is critical to writing effective pipeline configurations.

A step is the smallest unit of work — a single command or action. Steps run sequentially within a job. A job groups related steps that share the same execution environment. Stages group jobs that serve the same purpose (build, test, deploy).

Directed Acyclic Graph

Modern CI systems model pipelines as directed acyclic graphs (DAGs). Each job is a node, and dependencies between jobs are edges. A job can only run after all its dependencies complete. This allows independent jobs to run in parallel, reducing total pipeline duration.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run build

  test-unit:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: npm test -- --unit

  test-integration:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: npm test -- --integration

  deploy:
    needs: [test-unit, test-integration]
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh

GitHub Actions Anatomy

GitHub Actions workflows live in .github/workflows/ and are triggered by events. Each workflow contains one or more jobs, and each job contains steps that either run shell commands or invoke reusable actions from the marketplace.

name: CI Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"

      - name: Install dependencies
        run: npm ci

      - name: Run linter
        run: npm run lint

      - name: Run tests
        run: npm test

      - name: Build application
        run: npm run build

GitLab CI Anatomy

GitLab CI uses a .gitlab-ci.yml file at the repository root. It defines stages explicitly and assigns jobs to stages. Jobs can use needs to define fine-grained dependencies beyond stage ordering.

stages:
  - build
  - test
  - deploy

variables:
  NODE_VERSION: "20"

build-app:
  stage: build
  image: node:${NODE_VERSION}
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  script:
    - npm ci --cache .npm
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

test-unit:
  stage: test
  image: node:${NODE_VERSION}
  needs: [build-app]
  script:
    - npm ci
    - npm test -- --unit

test-integration:
  stage: test
  image: node:${NODE_VERSION}
  services:
    - postgres:16
  needs: [build-app]
  script:
    - npm ci
    - npm test -- --integration

deploy-production:
  stage: deploy
  needs: [test-unit, test-integration]
  script:
    - ./deploy.sh
  environment:
    name: production
  when: manual

Runners and Executors

A runner is the agent that executes pipeline jobs. GitHub provides hosted runners (ubuntu-latest, windows-latest, macos-latest). GitLab offers shared runners or self-hosted runners. Self-hosted runners give you control over the environment, networking, and hardware.