Runner Hardening

Isolating self-hosted runners with ephemeral VMs, network egress controls, and scoped tokens.

8 min readCI/CD & GitOpsPlatform

Runner Threat Model

Self-hosted runners execute arbitrary code from pull requests. A malicious PR can access the runner's network, read files, steal credentials, and pivot to other systems. GitHub-hosted runners are isolated per-job, but self-hosted runners persist between jobs.

The threat is real — supply chain attacks have targeted CI runners to steal credentials and inject malicious code into builds. Hardening runners is not optional for organizations with self-hosted infrastructure.

Ephemeral Runners

Ephemeral runners with auto-scaling


# GitHub Actions ephemeral runner configuration
# Each job gets a fresh runner that is destroyed after completion

# Runner group configuration (GitHub Enterprise)
# Settings > Actions > Runner groups > New runner group
# - Name: ephemeral-builders
# - Visibility: Selected repositories
# - Flow: Allow access to public repositories: No

# Terraform configuration for ephemeral runners
resource "aws_launch_template" "github_runner" {
  name_prefix = "github-runner-"
  image_id    = "ami-abc123"
  instance_type = "m5.xlarge"

  user_data = base64encode(<<-EOF
    #!/bin/bash
    # Install and configure runner
    cd /home/ec2-user
    curl -sL https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz | tar xz
    ./config.sh --url https://github.com/myorg --token $RUNNER_TOKEN --labels ephemeral --unattended
    ./run.sh

    # Shutdown after job completion
    shutdown -h now
  EOF)

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name = "github-ephemeral-runner"
    }
  }
}

resource "aws_autoscaling_group" "github_runners" {
  name                = "github-ephemeral-runners"
  min_size            = 0
  max_size            = 10
  desired_capacity    = 2
  launch_template {
    id      = aws_launch_template.github_runner.id
    version = "$Latest"
  }
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 50
    }
  }
}

Ephemeral runners are created for each job and destroyed after completion. The shutdown -h now command terminates the instance after the runner exits. The autoscaling group maintains a pool of warm instances.

Network Isolation

Network policies for runners


apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: runner-network-policy
  namespace: ci-runners
spec:
  podSelector:
    matchLabels:
      app: github-runner
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Allow GitHub webhook traffic
    - from:
        - ipBlock:
            cidr: 140.82.112.0/20
      ports:
        - port: 443
          protocol: TCP
  egress:
    # Allow HTTPS to GitHub
    - to:
        - ipBlock:
            cidr: 140.82.112.0/20
      ports:
        - port: 443
          protocol: TCP

    # Allow DNS
    - to:
        - namespaceSelector: {}
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

    # Block access to internal networks
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8
            except:
              - 10.0.1.0/24  # Exception for artifact registry

    # Allow artifact registry access
    - to:
        - ipBlock:
            cidr: 10.0.1.0/24
      ports:
        - port: 443
          protocol: TCP

Network policies restrict which traffic runners can send and receive. Egress rules allow only GitHub and the artifact registry. Ingress rules allow only GitHub webhooks. This limits the blast radius of a compromised runner.

Token Scoping

Minimum-privilege runner tokens


# GitHub Actions workflow with scoped permissions
name: Build
on: push

permissions:
  contents: read

jobs:
  build:
    runs-on: self-hosted
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test

  deploy:
    needs: build
    runs-on: self-hosted
    permissions:
      contents: read
      id-token: write
    environment: production
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/deploy
          aws-region: us-east-1

# GitLab CI with protected variables
deploy:
  stage: deploy
  script:
    - ./deploy.sh
  variables:
    AWS_ACCESS_KEY_ID: $PROD_AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY: $PROD_AWS_SECRET_ACCESS_KEY
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual
  environment:
    name: production

Scope tokens to the minimum required permissions. The build job only needs read access. The deploy job uses OIDC for short-lived credentials. Protected variables ensure only authorized pipelines can access production credentials.

Container Isolation

Running jobs in containers with resource limits


jobs:
  build:
    runs-on: self-hosted
    container:
      image: node:20
      options: >-
        --cpus 2
        --memory 4g
        --network none
        --read-only
        --tmpfs /tmp:size=1G
        --security-opt no-new-privileges
        --cap-drop ALL
        --cap-add NET_BIND_SERVICE

    services:
      postgres:
        image: postgres:16
        options: >-
          --cpus 1
          --memory 1g
          --network ci-network

    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test

Container isolation limits what the build job can do. --network none prevents network access. --read-only prevents filesystem modification. --cap-drop ALL removes all Linux capabilities. Resource limits prevent resource exhaustion.

Runner Monitoring

Audit logs — track which jobs ran on which runners with what commands.
Network monitoring — alert on unexpected outbound connections from runners.
Resource monitoring — track CPU, memory, and disk usage to detect anomalies.
Process monitoring — alert on unexpected processes running on runner hosts.
Credential rotation — rotate runner registration tokens regularly.

Treat runners as untrusted execution environments

Assume that any code running on a runner is hostile. Isolate runners from production systems, use short-lived credentials, and monitor for anomalous behavior. This mindset prevents supply chain attacks through CI.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 5 · Platform

Supply-Chain Security