CodeQL queries are often treated like configuration because they run as part of automated security analysis. In practice, a query change can affect what security issues an organization discovers, how many alerts developers receive, and whether existing CI workflows continue to behave as expected.
This becomes especially important when teams maintain custom CodeQL queries or modify query packs used across multiple repositories. A small change to a query can increase the number of results, eliminate valid findings, introduce false positives, or change the way developers interpret security alerts.
Testing these changes before organization-wide rollout helps separate a useful improvement from a change that simply produces more alerts.
Why CodeQL Query Changes Need Testing
A CodeQL query defines a pattern that CodeQL uses to identify potentially interesting behavior in source code. When the query changes, its result set can change even when application code has not changed.
For example, a query might originally identify unsafe calls to a particular API:
from FunctionCall call
where call.getTarget().hasName("ExecuteQuery")
select call, "Potential unsafe query execution."
A later change could broaden the matching logic to include additional APIs:
from FunctionCall call
where
call.getTarget().hasName("ExecuteQuery") or
call.getTarget().hasName("ExecuteRawQuery")
select call, "Potential unsafe query execution."
The change may be correct, but it can also create new findings across hundreds of repositories.
Before rolling it out, the team should answer questions such as:
Did the query identify the expected new cases?
Did it start reporting unrelated code?
Did existing valid results disappear?
How many repositories are affected?
Does the new query behave consistently across supported languages and frameworks?
Does the query still run within an acceptable analysis time?
These are testing questions, not simply code review questions.
Start With a Baseline
The first step is to establish what the current query produces.
Run the existing query against a representative set of repositories and record the results.
Useful baseline information includes:
Metric | Why It Matters |
|---|
Total findings | Shows the current result volume |
Findings by repository | Identifies concentrated impact |
Findings by severity | Helps understand security relevance |
Known true positives | Confirms expected detections |
Known false positives | Provides comparison data |
Analysis duration | Detects performance regressions |
Query errors | Identifies compatibility problems |
The baseline does not need to cover every repository initially. A representative test set is usually more useful than immediately running an experimental query across the entire organization.
Build a Representative Test Corpus
A query should not be tested only against the application that originally motivated the change.
Create a test corpus containing different patterns of code.
For example, a security query might need test cases for:
Unsafe API usage
Safe API usage
Wrapped API usage
Inherited API usage
Aliased API usage
Generated code
Test code
Third-party integration
Edge cases
The purpose is to test both detection and exclusion.
A good query should identify code that matches the intended security condition while avoiding code that merely looks similar.
Test Positive Cases
Positive test cases contain code that the query is expected to detect.
Suppose the query is designed to identify unsafe command execution:
public void Run(string input)
{
Process.Start(input);
}
The expected result is a CodeQL finding.
Create several variations instead of relying on one example:
public void Run(string input)
{
var command = input;
Process.Start(command);
}
Another variation might pass the value through a helper method:
public void Run(string input)
{
StartProcess(input);
}
private void StartProcess(string command)
{
Process.Start(command);
}
These cases help determine whether the query handles the data flow patterns it is intended to recognize.
Test Negative Cases
Negative testing is equally important.
A query that finds every call to an API may appear effective, but it can create a large number of irrelevant alerts.
For example:
public void Run()
{
Process.Start("known-safe-command");
}
If the query is specifically intended to identify user-controlled command execution, this case may need to be excluded or classified differently.
Negative cases should represent legitimate application patterns that should not trigger the query.
This is one of the most useful ways to detect false-positive growth before developers start receiving new alerts.
Test Boundary Conditions
The difficult cases are often between obvious positive and negative examples.
Consider input that comes from configuration:
var command = configuration["Command"];
Process.Start(command);
Now compare it with input obtained directly from an HTTP request:
var command = Request.Query["command"];
Process.Start(command);
Both values are external to the immediate function, but their security characteristics may be different depending on the application's architecture and the query's intended model.
Boundary cases should be explicitly included in the test suite.
Common examples include:
User input passed through several methods
Values returned by helper functions
Framework-specific request objects
Nullable values
Generic methods
Inheritance
Interface implementations
Lambda expressions
Wrapper APIs
Sanitization functions
Validation methods
Compare Query Results, Not Just Query Compilation
A query successfully compiling does not mean the query is correct.
A useful validation process compares the old and new result sets.
Conceptually:
Existing query results
+
New query results
↓
Result comparison
↓
Added findings
Removed findings
Unchanged findings
Pay particular attention to findings that appear only in the new version.
For each newly detected case, determine whether it is:
Expected detection
False positive
Duplicate detection
Previously missed valid finding
Also investigate findings that disappeared.
A query modification can accidentally remove an existing detection while attempting to improve another part of the query.
Test Custom Queries Against Known Vulnerabilities
If a query is designed to detect a specific vulnerability pattern, include known vulnerable examples in the test corpus.
For example:
public string BuildQuery(string username)
{
return "SELECT * FROM Users WHERE Name = '" + username + "'";
}
The test should confirm that the query identifies the intended unsafe behavior.
Then include a safer version:
public User FindUser(string username)
{
return database.Users
.FirstOrDefault(u => u.Name == username);
}
The query should not report the safer pattern merely because both pieces of code interact with database operations.
This gives the team a practical regression test for the security logic.
Test Query Performance
Correct results are only one part of the evaluation.
A query that produces accurate findings but significantly increases analysis time can still create operational problems.
Measure the query against repositories of different sizes.
For example:
Repository Type | What to Check |
|---|
Small | Basic correctness |
Medium | Normal developer workload |
Large | Query scalability |
Legacy | Complex code patterns |
Monorepo | Large dependency and source graph |
Compare analysis duration before and after the change.
Do not automatically reject a slower query. A security improvement may justify additional analysis time. The important point is to understand the trade-off before deployment.
Test False-Positive Controls
If the query supports exclusions or sanitization logic, test those paths explicitly.
For example, suppose the query recognizes a validation method:
if (IsSafeCommand(command))
{
Process.Start(command);
}
The test suite should verify whether IsSafeCommand is actually modeled correctly.
A common mistake is assuming that a function name is enough to establish security.
For example:
bool IsSafeCommand(string value)
{
return value.Length > 0;
}
This function does not necessarily make command execution safe.
Query logic should be based on the security semantics the query is designed to model, rather than convenient naming assumptions.
Test the Query in Pull Request Analysis
A query may behave differently from expectations when integrated into the normal developer workflow.
Test the modified query in a pull request scenario.
Check:
Does the expected alert appear?
Is the alert attached to the correct code location?
Does the result contain useful information?
Does the alert appear as a new issue when appropriate?
Does the workflow complete successfully?
Does analysis time remain acceptable?
This matters because developers interact with CodeQL findings through the workflow, not through the query source itself.
Use a Staged Rollout
Avoid making a new query version the default for every repository immediately.
A safer rollout can look like this:
Query change
↓
Unit-style query tests
↓
Representative repository corpus
↓
Internal validation
↓
Pilot repositories
↓
Result comparison
↓
Organization-wide rollout
Pilot repositories should represent different application types and development patterns.
For example, do not select only small, modern repositories if the organization also maintains large legacy applications.
Define Acceptance Criteria
Before testing starts, define what would make the query acceptable.
For example:
Required:
- Existing true positives remain detectable
- New intended cases are detected
- No critical false-positive regression
- Query completes successfully
- Analysis time remains within the agreed threshold
Investigate:
- Large increase in findings
- Significant increase in analysis duration
- Findings disappearing unexpectedly
- New repository-specific failures
This prevents teams from deciding whether the query is successful based only on the total number of alerts.
More alerts do not automatically mean better security analysis.
Common Mistakes
Testing Only the New Scenario
A query change is often created to solve one missed detection. Testing only that scenario can hide regressions elsewhere.
Always include existing positive and negative cases.
Using Alert Count as the Main Success Metric
An increase from 100 findings to 500 findings does not prove that the new query is better.
The additional 400 findings must be classified.
Skipping Large Repositories
A query may work well on a small project but become expensive on a large codebase.
Performance testing should include realistic repository sizes.
Ignoring Removed Findings
Teams often focus on new alerts and forget to check what disappeared.
Always compare both additions and removals.
Rolling Out Immediately
Organization-wide deployment makes debugging more difficult because many repositories can begin producing unexpected results at the same time.
Use staged deployment whenever practical.
Best Practices for CodeQL Query Testing
Keep query tests close to the query's intended behavior.
Maintain a test corpus containing:
Expected positive cases
Expected negative cases
Edge cases
Previously fixed false positives
Previously missed vulnerabilities
Framework-specific patterns
When changing a query, treat existing test cases as regression tests.
Record why important test cases exist. Without that context, a future developer may remove a seemingly unnecessary case and reintroduce an old problem.
It is also useful to separate query correctness from rollout readiness. A query can be logically correct but still require performance tuning or additional validation before organization-wide deployment.
Advantages and Disadvantages
Advantages
Reduces unexpected alert increases.
Detects false-positive regressions early.
Protects existing security coverage.
Provides measurable evidence for query changes.
Makes organization-wide rollout safer.
Helps identify performance problems before developers are affected.
Disadvantages
Requires maintaining a test corpus.
Large repositories can make validation expensive.
Security semantics can be difficult to model precisely.
Some framework-specific behavior requires additional test cases.
Query testing adds work to the security engineering process.
The trade-off is usually worthwhile for queries that are shared across many repositories because the cost of one poorly tested change can multiply across the organization.
Production Rollout Checklist
Before rolling out a CodeQL query change, verify the following:
[ ] Baseline results captured
[ ] Positive cases tested
[ ] Negative cases tested
[ ] Edge cases tested
[ ] Previously fixed regressions tested
[ ] Added findings reviewed
[ ] Removed findings reviewed
[ ] False-positive rate investigated
[ ] Large repositories tested
[ ] Analysis performance compared
[ ] Pull request behavior verified
[ ] Pilot repositories selected
[ ] Rollout criteria defined
[ ] Query version documented
This checklist creates a repeatable process instead of relying on manual inspection after deployment.
Summary
CodeQL query changes should be tested like application logic, especially when the same query is used across multiple repositories.
Start with a baseline, create representative positive and negative cases, test edge conditions, compare old and new result sets, and measure performance. Most importantly, investigate both newly reported findings and findings that disappear.
A successful query change is not simply one that produces more alerts. It is one that improves the intended security coverage without creating unnecessary noise or removing useful detections.
For organization-wide CodeQL usage, a small amount of query testing before rollout can prevent much larger security review and maintenance problems later.
Join the conversation! Your thoughts help the community grow.