How to Test GitHub Actions Workflows Before Changing Runner Images

GitHub Actions workflows depend on more than the commands written in a YAML file. They also depend on the operating system, installed tools, programming language versions, environment variables, file permissions, and other software available on the runner.

A workflow that works correctly on an existing runner image may fail after the image changes. A newer image might include different versions of Git, Node.js, Python, Java, or other development tools. It may also remove preinstalled software, change default shell behavior, or introduce operating system updates that affect your build.

These failures can interrupt deployments, delay releases, and make it difficult to determine whether the problem comes from the application or the runner environment.

The safest approach is to test your workflows against the proposed runner image before changing the production workflow. This article explains how to identify runner dependencies, test alternative images, compare execution results, and introduce runner changes without unnecessarily disrupting your CI/CD pipeline.

Why Runner Image Changes Can Break Workflows

GitHub-hosted runners provide virtual machines with operating systems and development tools already installed. For example, a workflow might use an Ubuntu runner to build a .NET application, execute tests, and publish a deployment package.

The workflow file may look simple:

name: Build Application

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Build application
        run: dotnet build MyApplication.sln --configuration Release

      - name: Run tests
        run: dotnet test MyApplication.sln --configuration Release

Although the YAML specifies the runner operating system, it does not explicitly define every tool used by the build.

The workflow assumes that the required .NET SDK is available. It may also rely on Git behavior, shell utilities, native libraries, environment variables, or other software installed on the runner.

When the underlying image changes, these assumptions may no longer hold.

Common problems include:

• A required SDK or runtime is missing or has a different version.

• A build tool behaves differently after an operating system update.

• A native library required by the application is unavailable.

• A script depends on a shell command that is no longer installed.

• File permissions or filesystem behavior cause tests to fail.

• A third-party GitHub Action is incompatible with the new environment.

• A deployment script depends on environment variables or tools that were previously available.

The important distinction is that changing a runner image can change the execution environment even when the workflow file remains exactly the same.

Identify Your Current Runner Dependencies

Before testing a new runner, determine what the existing workflow actually uses. This makes it easier to identify compatibility risks and investigate failures.

Review the runs-on Configuration

Start by examining the runner selected by each job.

For example:

jobs:
  build:
    runs-on: ubuntu-22.04

  test:
    runs-on: ubuntu-22.04

  deploy:
    runs-on: ubuntu-22.04

All three jobs use Ubuntu 22.04. If you plan to move to another Ubuntu image, identify which jobs need testing.

Explicit operating system labels make the current environment easier to understand. By contrast, a moving label such as ubuntu-latest can change its underlying operating system over time.

For controlled migrations, consider using an explicit runner label while evaluating the replacement environment. Once the migration is complete, decide whether to retain the explicit version or return to a moving label with a documented update process.

Inspect Installed Tools

Create a temporary diagnostic step to record important environment details.

- name: Inspect runner environment
  shell: bash
  run: |
    echo "Operating system:"
    cat /etc/os-release

    echo "Architecture:"
    uname -m

    echo "Git version:"
    git --version

    echo "Node.js version:"
    node --version || true

    echo "Python version:"
    python3 --version || true

    echo ".NET SDK:"
    dotnet --info || true

This step records useful information in the workflow log.

The commands use error-tolerant checks for optional tools because not every project requires Node.js, Python, or .NET. However, do not use this pattern to hide missing dependencies that the application actually requires.

For example, if the workflow needs the .NET SDK, a missing SDK should cause the build to fail rather than silently continue.

Also remember that installed software can change between runner image releases. Recording tool versions helps you compare the current environment with the proposed one.

Search for Hidden Environment Assumptions

Review shell scripts, build files, Dockerfiles, and deployment commands for dependencies that are not explicitly installed.

For example, a script might invoke a command such as jq, rsync, or a database client without checking whether it is available.

Another workflow might depend on a globally installed package or a particular version of Java.

Where possible, declare important versions and install required tools explicitly. This reduces the number of assumptions tied to the runner image.

Test the New Runner Without Replacing the Existing One

The safest migration strategy is to run the same workload against both the current and proposed environments.

Do not immediately replace the runner used by every production workflow. First, create a controlled test that allows you to compare the results.

Create a Separate Validation Workflow

For example, suppose the existing build uses Ubuntu 22.04 and you want to evaluate Ubuntu 24.04.

You can create a separate workflow that runs on pull requests and manual requests.

name: Runner Image Compatibility

on:
  pull_request:
  workflow_dispatch:

jobs:
  validate-current:
    name: Validate Current Runner
    runs-on: ubuntu-22.04

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Display environment
        run: |
          cat /etc/os-release
          git --version

      - name: Build application
        run: dotnet build MyApplication.sln --configuration Release

      - name: Run tests
        run: dotnet test MyApplication.sln --configuration Release

  validate-candidate:
    name: Validate Candidate Runner
    runs-on: ubuntu-24.04

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Display environment
        run: |
          cat /etc/os-release
          git --version

      - name: Build application
        run: dotnet build MyApplication.sln --configuration Release

      - name: Run tests
        run: dotnet test MyApplication.sln --configuration Release

This example executes the build and test commands on both runner images.

The two jobs run independently, which makes the comparison convenient. They use the same source revision and similar build steps, allowing you to identify environment-specific differences.

Before using this workflow, verify that both runner labels are supported for your repository and that the project has the necessary SDK setup. If the project requires a specific SDK version, install or select that version explicitly in both jobs.

The example also assumes that the solution can be built directly. If your normal workflow restores packages, configures credentials, builds multiple projects, or prepares test services, include those steps in the validation workflow too.

A reduced test workflow is useful for initial diagnosis, but it cannot establish compatibility with production if it omits important parts of the real pipeline.

Avoid Duplicating Complex Workflows

Copying a large workflow into a second file can create maintenance problems. The original workflow may receive new tests or deployment steps while the validation workflow remains unchanged.

A better option is to reuse common steps through a reusable workflow or composite action.

For example, a reusable workflow can accept a runner label as an input:

name: Reusable Build Validation

on:
  workflow_call:
    inputs:
      runner:
        description: Runner label to use
        required: true
        type: string

jobs:
  build:
    runs-on: ${{ inputs.runner }}

    steps:
      - name: Checkout source
        uses: actions/checkout@v4

      - name: Set up .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'

      - name: Restore dependencies
        run: dotnet restore MyApplication.sln

      - name: Build
        run: dotnet build MyApplication.sln --configuration Release --no-restore

      - name: Test
        run: dotnet test MyApplication.sln --configuration Release --no-build

A caller can invoke this workflow with different runner labels:

name: Compare Runner Images

on:
  pull_request:
  workflow_dispatch:

jobs:
  current:
    uses: ./.github/workflows/reusable-build.yml
    with:
      runner: ubuntu-22.04

  candidate:
    uses: ./.github/workflows/reusable-build.yml
    with:
      runner: ubuntu-24.04

This approach keeps the build steps consistent between environments.

The reusable workflow must exist at the specified repository path. Also ensure the runner label is appropriate for the project and that all required permissions, secrets, and inputs are configured correctly.

The versions of GitHub Actions shown here are examples. Review action versions and pinning policies according to your repository's security requirements.

Compare More Than Build Success

A successful build is necessary, but it does not prove that the new runner is fully compatible.

A runner image can introduce differences that appear only during integration tests, packaging, artifact generation, or deployment.

Compare the following areas before approving the change.

Compare Test Results

Run the same test suite against both environments and inspect failures individually.

A test that fails only on the candidate runner may reveal a real compatibility problem. However, it might also expose an existing flaky test, an external service failure, or a race condition.

Do not automatically classify every candidate-only failure as an operating system regression. Re-run the failing test where appropriate, inspect the logs, and identify the underlying cause.

Compare Build Artifacts

For workflows that publish packages, compare the resulting artifacts as well as the test results.

For example, a .NET build might produce assemblies, configuration files, static assets, and deployment packages. Confirm that the candidate runner produces all required files and that the application can start in a representative test environment.

Binary files may differ even when the application behaves correctly because build environments can introduce timestamps, metadata changes, or other non-deterministic output.

Therefore, byte-for-byte comparison is most useful when your build is designed to be reproducible. Otherwise, compare package contents, required files, metadata, and application behavior.

Measure Execution Time

Record the duration of important workflow steps on both runners.

A slower workflow does not necessarily indicate a defect, but a substantial increase can affect developer productivity and release timelines.

For a useful comparison, run the workflows multiple times under similar conditions. Account for package caches, dependency downloads, concurrent workloads, and external service latency.

Avoid making a decision based on a single unusually fast or slow run.

Make the Build Independent of Preinstalled Software

One of the most effective ways to reduce runner migration risk is to declare the environment your application needs.

Select Tool Versions Explicitly

For example, a .NET workflow can install its required SDK instead of relying on whichever SDK happens to be preinstalled.

- name: Set up .NET SDK
  uses: actions/setup-dotnet@v4
  with:
    dotnet-version: '10.0.x'

- name: Display .NET information
  run: dotnet --info

- name: Restore dependencies
  run: dotnet restore MyApplication.sln

The version pattern in this example selects a compatible SDK from the .NET 10 release line. For stricter reproducibility, use a suitable exact version and manage SDK selection through the repository's global.json file.

A tool setup action reduces dependence on the runner's preinstalled SDK, but it does not make the entire environment identical. Operating system libraries, shell utilities, and other dependencies can still differ.

Install Required System Packages

If a build requires a command-line utility or native library, install it explicitly where practical.

For example:

- name: Install required tools
  shell: bash
  run: |
    sudo apt-get update
    sudo apt-get install -y jq

This makes the dependency visible in the workflow.

However, package names, package availability, and default versions can vary between operating system releases. Test the installation commands on the candidate runner before using them in production.

For highly reproducible builds, consider using a controlled container image with explicitly managed dependencies. Containers can reduce differences in user-space tools, although the host runner, kernel, architecture, and other external factors can still matter.

Keep Environment Configuration in Source Control

Store build scripts, version configuration, and non-secret environment settings in the repository where appropriate.

Use GitHub Actions secrets or an approved secret-management system for credentials. Never print tokens, passwords, private keys, or other sensitive values while diagnosing runner differences.

The objective is to make the workflow's requirements explicit so that a new runner does not unexpectedly change the application build.

Common Mistakes to Avoid

Changing the Production Runner Before Testing

Replacing the runner label in the primary workflow without testing can introduce failures into every pull request or release.

Use a candidate workflow or a limited rollout first. Keep a clear path back to the previous runner while the migration is being evaluated.

Testing Only a Simple Build

A successful compilation does not confirm that integration tests, packaging, deployment, or native dependencies will work.

Run the meaningful stages of your actual pipeline, including those that depend on external services or operating system features.

Assuming a Moving Runner Label Is Stable

Labels such as ubuntu-latest refer to a moving runner target. Their underlying operating system can change as GitHub updates its runner offerings.

If you use a moving label, monitor runner-image announcements and maintain a regular compatibility-testing process. Do not assume that a workflow will always run against the same image simply because its YAML has not changed.

Ignoring Third-Party Actions

Your workflow may depend on actions maintained by other teams. Those actions can have their own runtime requirements and compatibility constraints.

Review action documentation, release notes, and execution logs when a failure occurs. Also maintain your repository's policy for reviewing and pinning third-party actions.

Comparing Different Source Revisions

If the current runner tests one commit and the candidate runner tests another, a code change may explain the difference.

Run both jobs against the same commit whenever possible. This makes the results more meaningful and reduces the number of variables under investigation.

Suppressing Errors During Validation

Commands such as || true can be useful for collecting optional environment information, but they should not hide failures in required build steps.

A validation workflow must fail when a required dependency is missing, a test fails, or a necessary artifact is not produced.

Best Practices for a Safe Runner Migration

Follow these practices when planning a runner-image change:

1. Inventory the runner labels used by all important workflows.

2. Record the current operating system, architecture, SDK versions, and required system tools.

3. Create a candidate workflow that tests the proposed image against the same commit.

4. Reuse the existing build and test logic to prevent validation drift.

5. Run unit tests, integration tests, packaging, and other relevant pipeline stages.

6. Compare build outputs, workflow duration, logs, and failure patterns.

7. Investigate every candidate-only failure before deciding whether it is a regression.

8. Test deployment-related steps in a safe environment before changing production deployment workflows.

9. Roll out the new image to a limited set of workflows before expanding its use.

10. Document the migration, including the previous runner, candidate runner, known differences, and rollback procedure.

Not every repository needs a lengthy migration process for every runner update. A small project with few dependencies may require only a focused compatibility test. A production system with native libraries, complex build tooling, and deployment automation needs broader validation.

Choose the test scope according to the impact of a potential failure.

Troubleshooting Runner Compatibility Failures

When the candidate workflow fails, use the following approach to narrow down the cause.

Start with the first meaningful error rather than the final failure message. A missing system package, for example, can cause several later steps to fail. Fixing the original dependency may resolve the entire chain.

If the cause remains unclear, temporarily add diagnostic commands to both workflows. Keep the output focused and ensure it does not expose secrets.

Advantages and Disadvantages of Testing Runner Images in Parallel

Running the current and candidate environments side by side offers several benefits, but it also introduces costs.

For most important workflows, the additional validation cost is justified by the reduction in migration risk.

To control that cost, run candidate tests on pull requests that affect build infrastructure, on a scheduled basis, or before planned runner migrations. You can also use a smaller initial test suite before running the full pipeline.

Summary

Changing a GitHub Actions runner image can affect your workflow even when the application code and YAML configuration remain unchanged. Operating system updates, missing tools, SDK differences, native libraries, and third-party actions can all introduce compatibility problems.

The safest approach is to test the proposed runner alongside the existing environment. Use the same source revision, reuse common workflow logic, and compare build results, tests, artifacts, execution time, and deployment behavior.

Reduce future migration risk by selecting tool versions explicitly, declaring required system dependencies, and avoiding unnecessary reliance on software that happens to be preinstalled on a runner.

Most importantly, treat runner-image changes as infrastructure changes that require validation. A controlled migration gives your team time to identify failures, correct environment assumptions, and move to the new runner without unnecessarily disrupting development or releases.