Introduction
CI/CD pipelines have become part of the application's security boundary.
A vulnerable application can be a problem, but an insecure build workflow can be equally serious because GitHub Actions workflows may have access to source code, deployment credentials, packages, cloud resources, and repository secrets.
This is why security analysis should not stop at application source code. The workflow files themselves need to be reviewed and tested.
CodeQL 2.26.3 includes several improvements to GitHub Actions analysis. The release improves the accuracy of multiple Actions security queries, including checks related to cache poisoning, untrusted checkouts, environment-variable injection, output clobbering, and workflow triggers. It also recognizes untrusted data from github.event.merge_group for workflows triggered by the merge_group event.
For development teams, the interesting question is not simply whether CodeQL can analyze workflow files.
The better question is:
Can CodeQL reliably detect dangerous GitHub Actions patterns before they become CI/CD security vulnerabilities?
This article explains how to build a practical test environment for that question.
Why GitHub Actions Workflows Need Security Testing
A workflow can execute commands automatically when an event occurs.
A simplified pipeline looks like this:
Pull Request
|
v
GitHub Event
|
v
GitHub Actions
|
+-- Checkout Code
+-- Install Dependencies
+-- Run Tests
+-- Build
+-- Deploy
The workflow may also have access to:
Repository Contents
Secrets
GITHUB_TOKEN
Caches
Artifacts
Cloud Credentials
Deployment Environments
If untrusted input reaches a privileged operation, an attacker may be able to influence the workflow.
This is why workflow security should be treated as application security rather than merely CI configuration.
What Changed in CodeQL 2.26.3?
CodeQL 2.26.3 contains several GitHub Actions analysis improvements.
The release includes improvements to queries covering:
It also improves how the analysis classifies the schedule event and recognizes untrusted data associated with merge_group.
One notable change is that the actions/envvar-injection/critical query now requires the untrusted source and privileged context to originate from the same trigger event. The cache-poisoning queries also account for read-only cache access in certain low-trust scenarios.
These changes make 2.26.3 particularly useful for testing workflow security detection rather than simply scanning application code.
Understand the Threat Model
Before testing CodeQL, define what an attacker is allowed to control.
A useful model is:
Untrusted Contributor
|
v
Pull Request / Event
|
v
Workflow
|
v
Privileged Runner
|
+-- Token
+-- Secrets
+-- Cache
+-- Deployment
The security problem occurs when attacker-controlled data crosses into a privileged operation without an appropriate boundary.
For example:
Untrusted Input
|
v
Workflow Expression
|
v
Shell Command
|
v
Code Execution
The benchmark should intentionally create safe test cases representing these patterns.
Create a Safe Test Repository
Do not test security detection directly against production workflows.
Create a dedicated repository containing synthetic examples.
A simple structure can be:
codeql-actions-test/
.github/
workflows/
safe.yml
unsafe-checkout.yml
cache-test.yml
env-test.yml
output-test.yml
schedule-test.yml
src/
sample.cs
README.md
The repository should contain no real secrets, credentials, or production deployment information.
The objective is to test detection, not reproduce a real attack.
Start With a Safe Workflow
A minimal workflow might be:
name: Build
on:
push:
pull_request:
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build
run: dotnet build --configuration Release
The workflow uses an explicit permission block:
permissions:
contents: read
This demonstrates an important security principle:
Give the workflow only the permissions it actually needs.
The exact action versions and workflow structure should follow the repository's supported configuration.
Create a Controlled Untrusted-Checkout Test
One important workflow security category involves checking out attacker-controlled code in a privileged context.
A test fixture can represent the risky pattern without containing real credentials or deployment actions.
For example, create a workflow that combines:
Pull Request Event
+
Privileged Context
+
Untrusted Checkout
Then run CodeQL against it.
The expected benchmark result is not "the workflow must execute."
The expected result is:
Workflow Pattern
|
v
CodeQL Analysis
|
v
Security Alert
The test should confirm whether the security issue is identified and whether the alert points developers toward the relevant workflow expression.
CodeQL 2.26.3 specifically improves the actions/untrusted-checkout/critical query and its alert path.
Test Environment-Variable Injection
Environment variables can become dangerous when attacker-controlled workflow input is placed into a privileged environment.
Conceptually:
Untrusted Input
|
v
Environment Variable
|
v
Privileged Step
The benchmark should contain both:
Unsafe Pattern
and:
Safe Pattern
This is important because a security scanner should not simply report everything that resembles user input.
CodeQL 2.26.3 refined the critical environment-variable injection analysis so that the untrusted source and privileged context must originate from the same trigger event.
Test Output Clobbering
GitHub Actions steps can communicate through outputs.
A security problem can occur when untrusted input influences an output in a way that allows later workflow behavior to be modified.
The conceptual flow is:
Untrusted Data
|
v
Workflow Output
|
v
Later Step
|
v
Security Impact
A test fixture should contain:
Unsafe Output Handling
and a safe equivalent.
The goal is to determine whether CodeQL correctly distinguishes them.
CodeQL 2.26.3 improves the actions/output-clobbering/high query, including reducing reports for simple JSON-preserving jq path filters while retaining relevant cases. It also addresses a performance issue involving regular-expression input.
Test Cache Poisoning
Caching can improve CI performance, but an improperly designed cache can create a trust-boundary problem.
A simplified scenario looks like:
Low-Trust Workflow
|
v
Writes Cache
|
v
Privileged Workflow
|
v
Reads Cache
If the privileged workflow consumes attacker-controlled cached content, the cache becomes a potential attack path.
The benchmark should test several cases:
Case 1
Writable by untrusted workflow
Case 2
Read-only access
Case 3
Default-branch scope
Case 4
Privileged workflow
CodeQL 2.26.3 updated cache-poisoning queries to account for read-only cache access on certain low-trust triggers and retain findings for cache scopes that the trigger can actually write.
This makes cache behavior an important part of a modern workflow-security benchmark.
Test Workflow Trigger Analysis
The event that starts a workflow is part of its security context.
Examples include:
push
pull_request
schedule
merge_group
workflow_dispatch
These events have different trust characteristics.
A benchmark should therefore classify workflows by trigger:
| Trigger | Test |
|---|
| push | Yes |
| pull_request | Yes |
| schedule | Yes |
| merge_group | Yes |
| workflow_dispatch | Yes |
The exact security impact depends on the workflow and permissions.
CodeQL 2.26.3 improves classification of the schedule event when determining whether a workflow can be externally triggered. It also recognizes untrusted data in github.event.merge_group for workflows triggered by the merge_group event.
Test Positive and Negative Cases
A good security benchmark must contain both vulnerable and safe workflows.
For example:
Test Suite
|
+-- Unsafe Checkout
+-- Safe Checkout
|
+-- Unsafe Cache
+-- Safe Cache
|
+-- Unsafe Environment
+-- Safe Environment
|
+-- Unsafe Output
+-- Safe Output
Then evaluate:
Expected Finding
|
v
Actual Finding
|
v
Match?
This helps measure both detection and noise.
Measure Detection Precision
A basic security-testing matrix can use:
| Test Case | Vulnerable? | Alert Expected? | Alert Found? |
|---|
| Unsafe checkout | Yes | Yes | Measure |
| Safe checkout | No | No | Measure |
| Unsafe cache | Yes | Yes | Measure |
| Safe cache | No | No | Measure |
| Unsafe environment | Yes | Yes | Measure |
| Safe environment | No | No | Measure |
| Unsafe output | Yes | Yes | Measure |
| Safe output | No | No | Measure |
The benchmark should record actual observations.
Do not claim a detection percentage unless it has been measured across a defined test suite.
Measure False Positives
A security scanner can create operational problems when developers receive too many irrelevant alerts.
For each alert, classify it:
True Positive
False Positive
Needs Review
Then calculate:
False Positive Rate =
False Positives
-----------------
Total Alerts
The exact formula and sample size should be documented in the benchmark.
The goal is not simply to maximize the number of alerts.
The goal is to identify meaningful security risks without overwhelming developers.
Measure False Negatives
False negatives are more difficult to measure because they are vulnerabilities the scanner did not report.
A controlled test suite helps.
If the repository contains ten intentionally vulnerable patterns and CodeQL reports eight:
Known Vulnerabilities = 10
Detected = 8
Not Detected = 2
The two undetected cases should be investigated.
Possible reasons include:
Do not automatically classify every missed case as a CodeQL defect.
Run CodeQL in CI
A workflow can run CodeQL analysis as part of CI.
A simplified configuration can look like:
name: Security Analysis
on:
push:
pull_request:
permissions:
contents: read
security-events: write
jobs:
analyze:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: csharp
- name: Build
run: dotnet build --configuration Release
- name: Analyze
uses: github/codeql-action/analyze@v3
The exact workflow should be adapted to the repository's languages and security configuration.
The important point is that the security analysis should become part of the normal development process rather than a manual activity.
Test GitHub Actions Analysis Specifically
If the benchmark focuses on workflow security, make sure the test repository contains workflow files that exercise the relevant GitHub Actions queries.
Do not assume that scanning a C# project automatically provides a comprehensive test of workflow security.
The benchmark should explicitly include:
.github/workflows/*.yml
with controlled test cases.
This separates:
Application Code Analysis
from:
GitHub Actions Workflow Analysis
Compare CodeQL Versions
The article's benchmark should ideally compare the target release against a baseline.
For example:
Baseline
CodeQL 2.26.2
Target
CodeQL 2.26.3
Run the same test repository against both.
Then compare:
Alerts Added
Alerts Removed
Alerts Changed
False Positives
False Negatives
Analysis Time
This is more useful than simply saying that the new version has "better detection."
Keep the Test Corpus Stable
A version comparison becomes unreliable if the test cases change between runs.
Use the same:
Repository
Workflow Files
Source Code
Configuration
Query Suite
Then change only the CodeQL version.
Conceptually:
Same Test Corpus
|
+--------+--------+
| |
v v
CodeQL Baseline CodeQL 2.26.3
| |
+--------+--------+
|
v
Compare
This makes differences easier to attribute to the CodeQL update.
Measure Analysis Performance
Security detection quality is the primary concern, but CI performance also matters.
Record:
Analysis duration
CPU usage where available
Memory usage where practical
Number of analyzed workflow files
Number of generated alerts
Do not claim that a release is faster or slower without measurement.
CodeQL 2.26.3 specifically includes a fix for a performance issue in the actions/output-clobbering/high query caused by unescaped regular-expression input.
That makes analysis time an interesting metric for a controlled before-and-after experiment.
Test Custom Queries
Organizations sometimes create custom CodeQL queries for internal workflow patterns.
For example:
Internal Deployment Workflow
|
v
Custom Security Requirement
|
v
Custom CodeQL Query
Version upgrades can affect custom queries.
CodeQL 2.26.3 includes a breaking change for GitHub Actions analysis: the codeql.actions.security.SelfHostedQuery module was removed because runner labels do not reliably distinguish self-hosted runners from managed runners. Custom queries depending on that module need to be updated.
Therefore, custom-query compatibility should be included in an upgrade test.
Validate Alert Locations
Detection alone is not enough.
An alert should help the developer locate the risky workflow expression.
For example:
Security Alert
|
v
Workflow File
|
v
Relevant Expression
|
v
Developer Fix
CodeQL 2.26.3 improves the starting paths for some cache-poisoning and untrusted-checkout queries so alerts can point more directly to expressions controlling risky behavior.
During benchmarking, record whether alerts are actionable rather than merely counting them.
Test Safe Workflow Patterns
Security analysis should distinguish between dangerous and acceptable configurations.
For example, a workflow that uses:
permissions:
contents: read
should be evaluated differently from a workflow granting broad permissions.
Likewise, a workflow using trusted, fixed values should not be treated the same as one that directly consumes attacker-controlled input.
The benchmark should contain paired examples:
Unsafe Pattern
|
v
Expected Alert
Safe Pattern
|
v
Expected No Alert
This is important for measuring precision.
Common Mistakes
Testing Only Vulnerable Workflows
Without safe examples, false-positive behavior cannot be evaluated.
Measuring Only Alert Count
More alerts do not automatically mean better security.
Changing the Test Corpus Between Versions
This makes version-to-version comparison unreliable.
Using Real Secrets
Security testing should use synthetic values.
Testing Only Application Code
Workflow security requires workflow-specific test cases.
Ignoring Custom Queries
A CodeQL version change can affect organization-specific analysis.
Treating Every Miss as a Scanner Defect
First verify that the benchmark case is actually covered by the intended query and configuration.
Ignoring Alert Location
An accurate detection with an unclear location may still be difficult for developers to remediate.
Troubleshooting
CodeQL Does Not Report an Expected Workflow Issue
Check:
The workflow is included in the analysis.
The relevant query suite is enabled.
The test pattern matches the intended vulnerability class.
The CodeQL version is the expected version.
The workflow syntax is valid.
Then compare the result with a minimal reproduction.
A Safe Workflow Produces an Alert
Determine whether the workflow actually satisfies the conditions of the security query.
For CodeQL 2.26.3, several GitHub Actions queries were specifically refined to improve accuracy, so the exact workflow context matters.
Custom Query Fails After Upgrade
Check whether it depends on a removed or changed CodeQL module.
For 2.26.3, custom Actions queries using codeql.actions.security.SelfHostedQuery need to be updated.
Analysis Takes Longer
Compare:
Previous Version
|
v
Same Repository
|
v
New Version
Check whether the difference is associated with a particular query, repository structure, or configuration.
Developers Receive Too Many Alerts
Classify alerts before changing the security configuration.
Determine whether the problem is:
Do not disable broad security analysis simply because developers receive alerts.
Advantages
Helps detect insecure CI/CD patterns earlier.
Provides a repeatable way to evaluate CodeQL updates.
Makes workflow-security testing measurable.
Can reduce manual review effort.
Helps teams identify changes in detection behavior.
Supports safer development and deployment pipelines.
Disadvantages
Static analysis cannot prove that every workflow is secure.
Custom workflow patterns may require additional modeling.
Security findings still require developer review.
Maintaining a representative benchmark corpus requires effort.
Version comparisons can become complicated when query suites or configurations change.
False positives can reduce trust if not managed carefully.
A Practical CodeQL Workflow Security Test Architecture
A complete benchmark can look like this:
Test Repository
|
+------------+------------+
| |
v v
Safe Workflows Vulnerable Workflows
| |
+------------+------------+
|
v
CodeQL Analysis
|
+------------+------------+
| | |
v v v
Alerts Locations Analysis Time
| | |
+------------+------------+
|
v
Test Evaluator
|
+------------+------------+
| |
v v
Expected Results Actual Results
| |
+------------+------------+
|
v
Comparison
This approach turns a security-scanning update into a measurable engineering experiment.
Example Benchmark Matrix
A practical test suite can contain:
| Security Area | Safe Case | Vulnerable Case | Detection |
|---|
| Untrusted checkout | Test | Test | Measure |
| Cache poisoning | Test | Test | Measure |
| Environment injection | Test | Test | Measure |
| Output clobbering | Test | Test | Measure |
| Workflow triggers | Test | Test | Measure |
| Untrusted merge-group data | Test | Test | Measure |
| Custom query compatibility | Test | Test | Measure |
Then compare the results across CodeQL versions:
| Metric | Baseline | 2.26.3 |
|---|
| Total Alerts | Measure | Measure |
| Expected Alerts | Measure | Measure |
| Missed Cases | Measure | Measure |
| False Positives | Measure | Measure |
| Analysis Time | Measure | Measure |
| Custom Query Failures | Measure | Measure |
The actual values should come from the benchmark environment rather than being assumed.
How to Interpret the Results
Suppose a benchmark shows that CodeQL 2.26.3 reports more alerts than the previous version.
That does not automatically mean the new version is better.
Investigate:
More Alerts
|
+-- New Detection?
+-- Previously Missed Case?
+-- False Positive?
+-- Changed Query Behavior?
Likewise, fewer alerts do not automatically mean reduced security coverage.
The useful question is:
Did the new version identify the intended security patterns more accurately and provide actionable results?
That is why a controlled test corpus is more valuable than a simple alert-count comparison.
Conclusion
GitHub Actions workflows are part of the software supply chain, and vulnerabilities in CI/CD configuration can expose source code, credentials, caches, artifacts, and deployment infrastructure. CodeQL 2.26.3 introduces several improvements specifically affecting GitHub Actions analysis, including changes to cache-poisoning, untrusted-checkout, environment-variable injection, output-clobbering, and workflow-trigger analysis.
The most useful way to evaluate those improvements is to build a controlled workflow-security benchmark containing both safe and intentionally vulnerable examples. Run the same test corpus against the baseline and target CodeQL versions, then measure detection accuracy, false positives, missed cases, alert quality, analysis time, and custom-query compatibility.
The goal is not to produce the largest possible number of security alerts. The goal is to identify real CI/CD security risks early, explain them clearly to developers, and verify that security analysis continues to work correctly as the CodeQL engine and workflow environment evolve.