AI-powered code review can help developers identify bugs, security issues, and potential improvements before merging a pull request. However, there is an important question every development team should ask: How do we know whether an AI code reviewer can identify problems that actually affect our application?
A review that only points out formatting issues or suggests better variable names may look useful, but it does not necessarily protect an application from broken code.
One practical way to evaluate GitHub Copilot code review is to introduce controlled defects into a test repository, create pull requests, and check whether Copilot identifies the problems.
This approach helps teams understand what Copilot can detect, where it needs improvement, and why automated builds and tests must remain part of the review process.
Why Test AI Code Reviews With Real Defects?
AI code review tools analyze code changes and provide feedback based on the available context. They can identify potential bugs, suspicious logic, incorrect assumptions, and other issues.
However, receiving a review comment does not automatically mean the code is safe to merge.
For example, consider a pull request that introduces an incorrect method name. A developer may expect Copilot to identify the issue, but the actual compiler is responsible for confirming whether the project builds successfully.
Similarly, a method can compile correctly while producing an incorrect result at runtime. A code review may identify the suspicious logic, but an automated test provides repeatable evidence of the failure.
Testing Copilot with deliberately introduced defects helps answer several questions:
- Does Copilot identify compilation errors?
- Can it recognize logic that produces incorrect results?
- Does it notice missing null checks?
- Can it identify changes that break existing tests?
- Does it provide useful explanations rather than generic suggestions?
- Can it distinguish an actual defect from a stylistic preference?
The objective is not to prove that Copilot catches every bug. It is to evaluate its usefulness within a complete software development workflow.
Create a Controlled Test Repository
Start with a separate repository or test branch. Do not introduce intentional defects into an active production branch.
For this example, create a small ASP.NET Core project with a service responsible for calculating discounts.
The service contains a method that accepts an order amount and applies a discount.
public class DiscountService
{
public decimal CalculateDiscount(decimal amount)
{
if (amount <= 0)
{
return 0;
}
return amount * 0.10m;
}
}
Create a unit test to verify the expected behavior.
public class DiscountServiceTests
{
[Fact]
public void CalculateDiscount_ReturnsTenPercent()
{
var service = new DiscountService();
var result = service.CalculateDiscount(100);
Assert.Equal(10, result);
}
}
This gives us a working baseline.
Before introducing defects, run the project build and test commands:
dotnet restore
dotnet build --no-restore
dotnet test --no-build
Make sure the tests pass. This is important because a test environment that already contains failures will make it difficult to determine whether a new change caused a problem.
Introduce Defects That Copilot Should Review
Once the baseline is working, introduce one defect at a time.
Testing individual defects makes the results easier to understand. If a pull request contains several unrelated problems, it becomes difficult to identify which issue Copilot detected and which it missed.
Introduce a compilation error
Change the return statement to call a method that does not exist.
public class DiscountService
{
public decimal CalculateDiscount(decimal amount)
{
if (amount <= 0)
{
return 0;
}
return CalculatePercentage(amount);
}
}
The method CalculatePercentage has not been defined.
The project should fail during compilation.
This test checks whether Copilot notices the invalid method reference and explains why the change will prevent the application from building.
The compiler remains the authoritative source for this failure. Copilot's review is an additional opportunity to identify the problem before the CI pipeline completes.
Introduce a logic error
Now replace the discount calculation with an incorrect formula.
public class DiscountService
{
public decimal CalculateDiscount(decimal amount)
{
if (amount <= 0)
{
return 0;
}
return amount * 10;
}
}
The method compiles successfully, but the calculation is incorrect.
For an order amount of 100, the method returns 1,000 instead of 10.
This is a useful test because compilation alone cannot detect the problem. A reviewer needs to understand the intended behavior and recognize that the calculation is incorrect.
A unit test should catch this issue.
Introduce a null-related defect
Consider a service that receives customer information.
public class CustomerService
{
public string GetCustomerName(Customer customer)
{
return customer.Name.ToUpper();
}
}
If customer is null, the method throws a NullReferenceException.
A safer implementation would validate the input:
public class CustomerService
{
public string GetCustomerName(Customer customer)
{
ArgumentNullException.ThrowIfNull(customer);
return customer.Name.ToUpper();
}
}
This test helps evaluate whether Copilot recognizes a potentially unsafe operation and recommends an appropriate correction.
The expected behavior should be clear: either the method rejects null input or the application handles it according to the business requirement.
Introduce a regression that breaks an existing test
Not every defect is obvious from the changed method.
Suppose an existing application expects a discount of 10 percent, but a developer changes the calculation to 5 percent.
The new code compiles, and the application may start successfully. However, an existing test will fail because the business behavior has changed.
This scenario is particularly useful because it demonstrates why code review and automated testing serve different purposes.
Copilot may identify the unexpected behavior if the surrounding code and requirements provide enough context. The test suite provides a repeatable check against the expected result.
Request a Copilot Code Review
After introducing a defect, commit the change to a separate branch and open a pull request.
GitHub Copilot can be requested as a reviewer from the pull request interface.
Under the Reviewers section in the pull request sidebar, select Copilot and request a review.
You can also use GitHub CLI when creating a pull request:
gh pr create --reviewer @copilot
For an existing pull request, use:
gh pr edit 123 --add-reviewer @copilot
Replace 123 with the actual pull request number.
Once the review is complete, examine the comments carefully.
Do not evaluate the review only by counting comments. A useful review should identify meaningful problems, explain their potential impact, and suggest a correction that matches the application's requirements.
Also remember that Copilot's default review is generally a comment review. It should not be treated as an automatic replacement for required human approvals or repository protection rules.
Build an Automated Validation Pipeline
Copilot code review should work alongside continuous integration, not replace it.
A GitHub Actions workflow can run the project build and unit tests whenever a pull request is created or updated.
Create a workflow file at .github/workflows/dotnet-ci.yml.
name: .NET Build and Test
on:
pull_request:
branches:
- main
push:
branches:
- main
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Set up .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '10.0.x'
- name: Restore dependencies
run: dotnet restore
- name: Build application
run: dotnet build --no-restore --configuration Release
- name: Run unit tests
run: dotnet test --no-build --configuration Release
Adjust the .NET version to match the target framework used by your project.
This workflow performs three important operations:
- Restore downloads the dependencies required by the project.
- Build checks whether the application can compile successfully.
- Test executes the automated test suite.
If a deliberately introduced compilation error exists, the build step should fail.
If the code compiles but changes the expected business behavior, the relevant unit test should fail.
These results provide objective validation that an AI review comment alone cannot provide.
Evaluate Copilot's Review Results
After each pull request receives a Copilot review and the CI workflow completes, record the results.
A simple test matrix makes it easier to compare the behavior across different defect types.
| Test scenario | Expected validation | What to evaluate |
|---|
| Invalid method reference | Build failure | Did Copilot identify the invalid reference? |
| Incorrect discount calculation | Unit test failure | Did Copilot recognize the incorrect result? |
| Missing null validation | Runtime or unit test failure | Did Copilot identify the potential exception? |
| Changed business behavior | Regression test failure | Did Copilot understand the expected behavior? |
For each scenario, record:
- Whether Copilot identified the defect.
- Whether its explanation was technically correct.
- Whether the suggested correction was appropriate.
- Whether the build or test pipeline detected the issue.
- Whether the review produced unnecessary or misleading comments.
You can also repeat the same scenario with slightly different implementations. This helps determine whether a result is consistent or depends heavily on how the code is written.
Avoid treating a single successful review as proof that Copilot can reliably detect the same category of defects across every project.
Improve Review Quality With Repository Instructions
Copilot's review quality can depend on the context available in the repository.
A project with clear naming conventions, architecture rules, and testing expectations gives reviewers more information about what the code is supposed to do.
You can add a .github/copilot-instructions.md file to provide repository-specific guidance.
For example:
# Code Review Guidelines
- Follow the existing project architecture.
- Identify potential runtime exceptions.
- Check null handling in public methods.
- Review business logic changes carefully.
- Identify missing validation for external input.
- Check whether existing tests cover changed behavior.
- Avoid suggesting unrelated formatting changes.
- Prefer simple and maintainable solutions.
- Consider performance when reviewing database operations.
- Never recommend removing validation only to make a test pass.
These instructions communicate the team's expectations and coding conventions.
However, instructions should not be treated as a substitute for executable rules. If a requirement can be checked through a compiler, analyzer, unit test, or security scanner, automate that check.
Keep instructions focused and specific. A long document containing contradictory or vague requirements may make it harder to maintain consistent review expectations.
Common Mistakes When Testing AI Code Reviews
Introducing too many defects in one pull request
If a pull request contains several unrelated defects, it becomes difficult to determine which issues Copilot detected correctly.
Use separate branches or pull requests for individual test scenarios.
Treating every review comment as a confirmed bug
AI-generated feedback can be incorrect or unnecessary. Review each suggestion against the actual requirements and application behavior.
A comment should not automatically become a code change simply because it comes from Copilot.
Assuming a successful Copilot review means the build is safe
A review can miss a compilation error, a failing test, or a configuration problem.
Keep build and test checks as required status checks before merging.
Ignoring the surrounding application context
A method may look incorrect when reviewed independently but be valid because of an existing business rule or another component.
Provide clear requirements, useful tests, and repository instructions so the reviewer has enough context.
Not requesting another review after fixing an issue
When you push additional commits, Copilot does not necessarily review the new changes automatically.
You can manually request another review or configure automatic reviews for new pushes through the repository's code review settings.
Best Practices for Using Copilot Code Review
A practical approach is to combine AI feedback with established engineering checks.
- Use a dedicated test repository for deliberately broken changes.
- Introduce one defect at a time.
- Keep a clean baseline before testing.
- Use unit tests to verify expected application behavior.
- Run build and test workflows for every pull request.
- Review Copilot comments before accepting suggested changes.
- Provide repository-specific instructions for important conventions.
- Keep human approval requirements for important changes.
- Re-review significant changes before merging.
- Track missed defects and false positives to understand where the review process needs improvement.
Most importantly, evaluate the quality of the feedback rather than the number of comments generated.
Summary
Testing GitHub Copilot code review with deliberately broken changes provides a practical way to understand its strengths and limitations. By introducing compilation errors, incorrect business logic, null-related problems, and regressions in a controlled repository, development teams can evaluate whether Copilot identifies issues that matter.
However, AI code review should remain one part of a broader quality process. Compilers, unit tests, integration tests, static analysis, and CI pipelines provide independent validation that AI feedback cannot replace.
When these tools work together, developers get an additional layer of feedback without giving up the reliability of established software testing practices.
GitHub CopilotAI code reviewbuild failuresunit testingASP.NET CoreCI/CD
Join the conversation! Your thoughts help the community grow.