Code coverage is commonly used in CI/CD pipelines to understand how much of an application's source code is exercised by automated tests. Teams use coverage reports to identify untested areas, enforce quality gates, and prevent important code paths from being introduced without tests.
However, coverage systems depend on more than just the test framework. They also depend on Git branches, commits, changed files, report formats, and the way a CI platform compares the current code with the repository's history.
This becomes especially important when developers create new branches. A new branch may not have enough historical coverage information for a coverage check to perform the same comparison that it performs on an established branch. When that comparison fails, developers can end up with a failed check even though the application builds and the tests themselves pass.
GitHub has addressed an issue involving code coverage failures on new branches. Understanding this type of problem is useful because it shows why code coverage should be treated as a CI/CD engineering concern rather than simply a percentage displayed in a report.
What Is Code Coverage?
Code coverage measures which parts of an application's source code are executed while automated tests run.
Consider this simple C# method:
public int CalculateDiscount(int amount)
{
if (amount >= 1000)
{
return 100;
}
return 0;
}A test that only checks an amount below 1,000 exercises the second path:
[Fact]
public void ReturnsZeroForSmallOrder()
{
var service = new DiscountService();
var result = service.CalculateDiscount(500);
Assert.Equal(0, result);
}The test does not execute the amount >= 1000 branch.
A coverage tool can identify this difference and report that part of the method was not covered.
Coverage can therefore help answer questions such as:
Which lines were executed?
Which branches were executed?
Which methods have tests?
Which files have little or no coverage?
Did coverage change with this pull request?
Why Coverage Is Useful in CI/CD
Coverage becomes more valuable when it is connected to a pull request or branch workflow.
A basic pipeline might look like this:
Developer
|
v
Create Branch
|
v
Write Code
|
v
Add Tests
|
v
Push Changes
|
v
GitHub Actions
|
+--> Build
|
+--> Test
|
+--> Coverage
|
v
Pull Request CheckThe purpose is not necessarily to force every project to achieve a specific percentage. The more useful goal is to identify meaningful changes that are not adequately tested.
For example, a team may require that a pull request does not significantly reduce coverage for the code it changes.
Why New Branches Can Cause Coverage Problems
Coverage comparison often depends on a baseline.
Suppose the main branch contains:
main
├── CustomerService.cs
├── OrderService.cs
└── PaymentService.csThe coverage system has historical information about these files.
Now a developer creates a new branch:
main
|
+---- feature/new-payment-flowThe branch introduces a new file:
PaymentValidationService.csThe coverage system may need to determine:
What was the previous coverage?
What is the current coverage?
Which lines are new?
Which files changed?
What should be used as the comparison baseline?A new branch can expose edge cases in this process because the branch may not yet have its own coverage history.
That does not necessarily mean the application has a testing problem. It can be a problem with how coverage data is compared or how the CI system handles the branch state.
Coverage Percentage vs Coverage Diff
These two concepts are often confused.
Overall Coverage
Overall coverage measures the percentage of the analyzed code that was executed by tests.
For example:
Lines covered: 850
Total lines: 1000
Coverage: 85%Coverage Diff
Coverage diff compares coverage between two versions or branches.
For example:
main: 84%
feature: 86%
Change: +2%A pull-request system may also focus only on changed lines:
Changed lines: 120
Covered lines: 110
Changed-line coverage: 91.6%Coverage comparison is more complicated than simply calculating a percentage because the system must know what changed and what should be used as the baseline.
Why Branch Creation Is an Important Edge Case
A normal branch usually has a clear relationship with an existing branch:
main
|
+---- feature/loginBut from the perspective of a coverage system, the feature branch may initially have:
No previous coverage result
No branch-specific history
New files
Renamed files
Deleted files
Changed test configuration
Different build outputs
A coverage check that assumes historical data always exists can therefore behave incorrectly.
This is particularly noticeable when a project uses coverage as a required GitHub check.
A Typical Failure Scenario
Imagine a repository has a workflow like this:
name: Test and Coverage
on:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: 8.0.x
- name: Test
run: dotnet test --collect:"XPlat Code Coverage"The tests may pass successfully:
Test Run Successful
Passed: 124
Failed: 0But the coverage check can still fail because the coverage comparison or report processing does not have the expected baseline.
This distinction is important:
A test failure and a coverage-check failure are not necessarily the same problem.
How a Coverage Pipeline Should Be Structured
A robust coverage pipeline should separate the different stages.
Checkout
|
v
Restore Dependencies
|
v
Build
|
v
Run Tests
|
v
Generate Coverage
|
v
Process Coverage Report
|
v
Compare With Baseline
|
v
Publish CheckEach stage should have a clear responsibility.
If the tests pass but the coverage comparison fails, developers should be able to determine that the problem is in the coverage stage rather than debugging application code unnecessarily.
Example With .NET
For a .NET project, coverage can be collected using the built-in test collection support:
dotnet test \
--collect:"XPlat Code Coverage"The command generates coverage output during the test run.
A project can also use a coverage tool such as Coverlet through the appropriate .NET testing configuration.
For example:
dotnet test \
/p:CollectCoverage=true \
/p:CoverletOutputFormat=coberturaThe exact command depends on how the project has configured its testing and coverage tooling.
The important part is that the coverage report generated by the test run must be in a format that the downstream coverage system understands.
New Branches and Coverage Baselines
A good coverage system should understand that a newly created branch may not have an independent historical coverage record.
For example:
main
Coverage: 82%
|
+---- feature/customer-import
|
v
New branch
No history yetThe system may need to compare the branch against the base branch instead.
Conceptually:
Base branch
|
| Coverage baseline
v
Pull request
|
| Current coverage
v
ComparisonThis approach is more meaningful because the pull request is evaluated against the code it is intended to modify.
Why Changed-Line Coverage Can Be More Useful
Suppose an old application has 500,000 lines of code and only 55% overall coverage.
A developer adds 2,000 new lines and tests 1,900 of them.
The overall percentage may barely move because the existing codebase is so large.
Changed-line coverage tells a different story:
New lines: 2,000
Covered lines: 1,900
Coverage: 95%This can give teams a more useful signal for new development.
It also avoids forcing developers to immediately rewrite every old test in a legacy system just to improve the global percentage.
Common Mistakes
Treating Coverage as a Quality Score
An 80% coverage number does not automatically mean that the application is well tested.
A test suite can execute a line without checking whether the result is correct.
For example:
var result = CalculateTotal(order);does not provide much value unless the test actually verifies the expected result.
Ignoring Branch Coverage
Line coverage may report good numbers while important conditional paths remain untested.
Using Only Overall Coverage
For large legacy repositories, overall coverage can hide the quality of newly added code.
Failing the Build for Minor Coverage Changes
A very strict threshold can make development frustrating if normal refactoring causes tiny coverage changes.
Coverage policies should be meaningful rather than arbitrary.
Assuming Every Failure Is a Code Problem
If tests pass but the coverage check fails, investigate the coverage configuration, report generation, baseline, and branch comparison before changing application code.
Best Practices
Test the Code That Matters
Focus on business-critical behavior rather than chasing a percentage.
For an e-commerce application, important areas might include:
Payment processing
Authentication
Order calculation
Inventory updates
Authorization
Data validation
These areas deserve strong tests even if less important utility code has lower coverage.
Use Coverage as a Regression Signal
A useful policy can prevent a pull request from significantly reducing coverage in changed code.
This turns coverage into a regression detector rather than a vanity metric.
Use the Correct Baseline
Pull-request coverage should normally be evaluated against the appropriate base branch.
This is especially important for new branches.
Keep Coverage Configuration in Source Control
For reproducibility, coverage settings should be part of the repository rather than being configured differently on individual developer machines.
Make CI Failures Actionable
A failed coverage check should tell developers:
What failed
Which files are affected
What coverage changed
What threshold was violated
Which branch was used as the baseline
A message such as "coverage failed" is not enough.
Troubleshooting Coverage Failures on a New Branch
When a new branch produces an unexpected coverage failure, follow a structured process.
Step 1: Check the Tests
Run the tests locally:
dotnet testIf the tests fail, fix those failures first.
Step 2: Generate Coverage Locally
Run the project's configured coverage command:
dotnet test --collect:"XPlat Code Coverage"Verify that a coverage report is actually generated.
Step 3: Check the Report
Confirm that the expected source files appear in the report.
A missing file can indicate an incorrect build, test, or coverage configuration.
Step 4: Check the Base Branch
Determine which branch the pull request is comparing against.
The expected baseline may not be the same as the branch currently checked out locally.
Step 5: Inspect the CI Logs
Look at the coverage-processing step separately from the test step.
For example:
Build: Passed
Tests: Passed
Coverage: Generated
Comparison: FailedThis immediately narrows the problem.
Step 6: Check New Files
Pay particular attention to files that did not exist in the base branch.
New files may have no historical coverage information.
Advantages
Better Developer Feedback
Coverage checks can identify code that needs additional tests before merging.
Safer Refactoring
Coverage reports provide another signal when developers change existing behavior.
Useful Pull Request Visibility
Developers can understand how a change affects test coverage before it reaches the main branch.
Better CI/CD Quality Gates
Coverage can be combined with testing and static analysis to create a broader quality process.
Disadvantages and Limitations
Coverage Does Not Measure Test Quality
A high percentage does not guarantee meaningful tests.
Additional CI Complexity
Coverage collection and reporting add processing steps to the pipeline.
Baseline Problems Can Create False Failures
New branches, renamed files, and unusual repository structures can expose issues in coverage comparison.
Coverage Can Encourage the Wrong Behavior
If teams focus only on the percentage, developers may write superficial tests simply to satisfy a threshold.
Large Repositories Need More Care
Monorepos and repositories with multiple applications can require more sophisticated coverage configuration.
A Practical Coverage Strategy
For most development teams, a balanced approach is better than chasing a single number.
A useful strategy is:
Unit Tests
|
v
Integration Tests
|
v
Coverage Measurement
|
v
Changed-Code Review
|
v
Security and Quality Checks
|
v
Pull RequestUse coverage to identify gaps, but let developers and reviewers decide whether the tests actually protect important behavior.
For new branches, make sure the CI system has a clear baseline and does not assume that the branch already has historical coverage information.
Production Checklist
Before enforcing coverage checks in a repository, verify:
Tests run successfully in CI.
Coverage reports are generated consistently.
The correct base branch is used for comparisons.
New files are handled correctly.
Renamed and deleted files do not create misleading results.
Coverage thresholds are documented.
Developers can understand why a check failed.
Coverage is not treated as the only quality metric.
Important business logic has meaningful tests.
Coverage configuration is version-controlled.
CI failures distinguish test failures from coverage-processing failures.
Summary
Code coverage is a useful part of a software quality process, but it becomes more complicated when CI/CD systems compare coverage across branches.
New branches are a particularly important edge case because they may not have their own historical coverage data. A coverage system that expects a previous result can therefore produce failures even when the application's tests are passing.
The fix for this class of problem is not simply to lower the coverage threshold. Teams should understand how the coverage tool determines its baseline, how new files are handled, and how pull-request coverage is calculated.
For production teams, the best approach is to use coverage as a testing and regression signal, especially around changed code, while combining it with meaningful unit tests, integration tests, code review, and security analysis.
The goal is not to achieve an impressive coverage number. The goal is to make sure that new and changed code has enough meaningful tests to give the team confidence when it is merged and deployed.
Join the conversation! Your thoughts help the community grow.