How to Test GitHub Actions Workflows Before Changing Runner Images

GitHub Actions workflows depend on more than the commands written in a YAML file. They also depend on the operating system, installed tools, programming language versions, environment variables, file permissions, and other software available on the runner.

A workflow that works correctly on an existing runner image may fail after the image changes. A newer image might include different versions of Git, Node.js, Python, Java, or other development tools. It may also remove preinstalled software, change default shell behavior, or introduce operating system updates that affect your build.

These failures can interrupt deployments, delay releases, and make it difficult to determine whether the problem comes from the application or the runner environment.

The safest approach is to test your workflows against the proposed runner image before changing the production workflow. This article explains how to identify runner dependencies, test alternative images, compare execution results, and introduce runner changes without unnecessarily disrupting your CI/CD pipeline.

Why Runner Image Changes Can Break Workflows

GitHub-hosted runners provide virtual machines with operating systems and development tools already installed. For example, a workflow might use an Ubuntu runner to build a .NET application, execute tests, and publish a deployment package.

The workflow file may look simple:

name: Build Application

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Build application
        run: dotnet build MyApplication.sln --configuration Release

      - name: Run tests
        run: dotnet test MyApplication.sln --configuration Release

Although the YAML specifies the runner operating system, it does not explicitly define every tool used by the build.

The workflow assumes that the required .NET SDK is available. It may also rely on Git behavior, shell utilities, native libraries, environment variables, or other software installed on the runner.

When the underlying image changes, these assumptions may no longer hold.

Common problems include:

• A required SDK or runtime is missing or has a different version.

• A build tool behaves differently after an operating system update.

• A native library required by the application is unavailable.

• A script depends on a shell command that is no longer installed.

• File permissions or filesystem behavior cause tests to fail.

• A third-party GitHub Action is incompatible with the new environment.

• A deployment script depends on environment variables or tools that were previously available.

The important distinction is that changing a runner image can change the execution environment even when the workflow file remains exactly the same.

Identify Your Current Runner Dependencies

Before testing a new runner, determine what the existing workflow actually uses. This makes it easier to identify compatibility risks and investigate failures.

Review the runs-on Configuration

Start by examining the runner selected by each job.

For example:

jobs:
  build:
    runs-on: ubuntu-22.04

  test:
    runs-on: ubuntu-22.04

  deploy:
    runs-on: ubuntu-22.04

All three jobs use Ubuntu 22.04. If you plan to move to another Ubuntu image, identify which jobs need testing.

Explicit operating system labels make the current environment easier to understand. By contrast, a moving label such as ubuntu-latest can change its underlying operating system over time.

For controlled migrations, consider using an explicit runner label while evaluating the replacement environment. Once the migration is complete, decide whether to retain the explicit version or return to a moving label with a documented update process.

Inspect Installed Tools

Create a temporary diagnostic step to record important environment details.

- name: Inspect runner environment
  shell: bash
  run: |
    echo "Operating system:"
    cat /etc/os-release

    echo "Architecture:"
    uname -m

    echo "Git version:"
    git --version

    echo "Node.js version:"
    node --version || true

    echo "Python version:"
    python3 --version || true

    echo ".NET SDK:"
    dotnet --info || true

This step records useful information in the workflow log.

The commands use error-tolerant checks for optional tools because not every project requires Node.js, Python, or .NET. However, do not use this pattern to hide missing dependencies that the application actually requires.

For example, if the workflow needs the .NET SDK, a missing SDK should cause the build to fail rather than silently continue.

Also remember that installed software can change between runner image releases. Recording tool versions helps you compare the current environment with the proposed one.

Search for Hidden Environment Assumptions

Review shell scripts, build files, Dockerfiles, and deployment commands for dependencies that are not explicitly installed.

For example, a script might invoke a command such as jq, rsync, or a database client without checking whether it is available.

Another workflow might depend on a globally installed package or a particular version of Java.

Where possible, declare important versions and install required tools explicitly. This reduces the number of assumptions tied to the runner image.

Test the New Runner Without Replacing the Existing One

The safest migration strategy is to run the same workload against both the current and proposed environments.

Do not immediately replace the runner used by every production workflow. First, create a controlled test that allows you to compare the results.

Create a Separate Validation Workflow

For example, suppose the existing build uses Ubuntu 22.04 and you want to evaluate Ubuntu 24.04.

You can create a separate workflow that runs on pull requests and manual requests.

name: Runner Image Compatibility

on:
  pull_request:
  workflow_dispatch:

jobs:
  validate-current:
    name: Validate Current Runner
    runs-on: ubuntu-22.04

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Display environment
        run: |
          cat /etc/os-release
          git --version

      - name: Build application
        run: dotnet build MyApplication.sln --configuration Release

      - name: Run tests
        run: dotnet test MyApplication.sln --configuration Release

  validate-candidate:
    name: Validate Candidate Runner
    runs-on: ubuntu-24.04

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Display environment
        run: |
          cat /etc/os-release
          git --version

      - name: Build application
        run: dotnet build MyApplication.sln --configuration Release

      - name: Run tests
        run: dotnet test MyApplication.sln --configuration Release

This example executes the build and test commands on both runner images.

The two jobs run independently, which makes the comparison convenient. They use the same source revision and similar build steps, allowing you to identify environment-specific differences.

Before using this workflow, verify that both runner labels are supported for your repository and that the project has the necessary SDK setup. If the project requires a specific SDK version, install or select that version explicitly in both jobs.

The example also assumes that the solution can be built directly. If your normal workflow restores packages, configures credentials, builds multiple projects, or prepares test services, include those steps in the validation workflow too.

A reduced test workflow is useful for initial diagnosis, but it cannot establish compatibility with production if it omits important parts of the real pipeline.

Avoid Duplicating Complex Workflows

Copying a large workflow into a second file can create maintenance problems. The original workflow may receive new tests or deployment steps while the validation workflow remains unchanged.

A better option is to reuse common steps through a reusable workflow or composite action.

For example, a reusable workflow can accept a runner label as an input:

name: Reusable Build Validation

on:
  workflow_call:
    inputs:
      runner:
        description: Runner label to use
        required: true
        type: string

jobs:
  build:
    runs-on: ${{ inputs.runner }}

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Set up .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'

      - name: Restore dependencies
        run: dotnet restore MyApplication.sln

      - name: Build
        run: dotnet build MyApplication.sln --configuration Release --no-restore

      - name: Test
        run: dotnet test MyApplication.sln --configuration Release --no-build

A caller can invoke this workflow with different runner labels:

name: Compare Runner Images

on:
  pull_request:
  workflow_dispatch:

jobs:
  current:
    uses: ./.github/workflows/reusable-build.yml
    with:
      runner: ubuntu-22.04

  candidate:
    uses: ./.github/workflows/reusable-build.yml
    with:
      runner: ubuntu-24.04

This approach keeps the build steps consistent between environments.

The reusable workflow must exist at the specified repository path. Also ensure the runner label is appropriate for the project and that all required permissions, secrets, and inputs are configured correctly.

The versions of GitHub Actions shown here are examples. Review action versions and pinning policies according to your repository's security requirements.

Compare More Than Build Success

A successful build is necessary, but it does not prove that the new runner is fully compatible.

A runner image can introduce differences that appear only during integration tests, packaging, artifact generation, or deployment.

Compare the following areas before approving the change.