GitHub Copilot Code Review for Azure Repos: Detecting False Positives and Missed Bugs

Saurav Kumar
11h
171
0
0

Article

AI-assisted code review can help developers identify potential defects without manually inspecting every line of a pull request. GitHub Copilot code review can analyze changes and provide review comments, making it useful as an additional review layer.

However, AI-generated review comments are not automatically correct.

An AI reviewer can report a problem that is not actually a defect, known as a false positive. It can also fail to identify a real problem, creating a false negative or missed bug.

For teams using Azure Repos and .NET applications, this makes validation important. The goal should not be to measure how many comments Copilot generates. The useful question is whether those comments help identify meaningful defects without creating excessive review noise.

What Are False Positives and Missed Bugs?

Consider this code:

public async Task<Customer?> GetCustomerAsync(
    int customerId)
{
    return await repository
        .FindByIdAsync(customerId);
}

An AI reviewer might suggest additional validation for customerId.

That could be useful in one application but unnecessary in another if the repository already guarantees valid identifiers.

This is a potential false positive:

AI Review
    ↓
Potential issue
    ↓
Application context
    ↓
Not actually a defect

A missed bug is the opposite:

Actual defect
    ↓
AI review
    ↓
No finding

Both cases matter when evaluating an AI code-review system.

Why Context Matters

AI code review operates on available repository and pull-request context.

Consider:

public async Task<Order?> GetOrderAsync(
    int orderId,
    int tenantId)
{
    return await dbContext.Orders
        .FirstOrDefaultAsync(x =>
            x.Id == orderId &&
            x.TenantId == tenantId);
}

The tenant filter is important in a multi-tenant application.

An AI reviewer that does not understand the application's authorization model might:

Miss a missing tenant condition.
Flag the condition as unnecessary.
Recommend a different implementation that does not preserve the security boundary.

This demonstrates why code-review evaluation should use known test cases with explicit expected findings.

Designing a Code Review Benchmark

A practical benchmark can contain intentionally selected pull requests.

For example:

Benchmark Repository
       |
       +-- Security bugs
       +-- Logic bugs
       +-- Performance issues
       +-- Reliability issues
       +-- Safe changes

Each change should have a predefined expected result.

For example:

Test Case	Expected Result
SQL injection vulnerability	Detect
Missing authorization check	Detect
Null-reference risk	Detect
Unnecessary warning	Avoid
Correct refactoring	Avoid false positive
Safe parameterized query	Avoid false positive

This creates a controlled environment for evaluating review quality.

Use Realistic .NET Examples

A benchmark becomes more useful when it resembles production application code.

For example, this is unsafe:

var sql =
    $"SELECT * FROM Users WHERE Id = {userId}";

await using var command =
    new NpgsqlCommand(sql, connection);

A strong review should recognize the risk of constructing SQL from interpolated input.

The safer approach is:

const string sql = """
    SELECT id, name
    FROM users
    WHERE id = @id;
    """;

await using var command =
    new NpgsqlCommand(sql, connection);

command.Parameters.AddWithValue("id", userId);

The benchmark can then test whether the AI reviewer:

Unsafe version → Finding
Safe version   → No unnecessary finding

False Positive Testing

False-positive testing is often overlooked.

Suppose a code reviewer flags:

var customer =
    await repository.FindByIdAsync(id);

with a recommendation to validate id.

If the repository contract already guarantees that id is valid, the recommendation may not be useful.

A benchmark should record:

Finding generated
        ↓
Developer review
        ↓
Valid defect?

Then classify the result:

True Positive
False Positive

This allows the team to measure review precision.

Missed Bug Testing

Now introduce an actual vulnerability:

public async Task<Order?> GetOrderAsync(
    int orderId)
{
    return await dbContext.Orders
        .FirstOrDefaultAsync(x =>
            x.Id == orderId);
}

Suppose the application is multi-tenant and the query should also include:

x.TenantId == tenantId

If the AI reviewer produces no relevant security warning, that is a missed bug.

Record it as:

Expected Finding → Yes
AI Finding        → No
Result            → Missed Bug

This is more useful than simply counting the number of AI comments.

Precision and Recall

Two useful concepts for evaluating an AI code reviewer are precision and recall.

Precision

Precision asks:

Of the issues reported by the reviewer, how many were actually valid?

Conceptually:

Precision =
True Positives /
(True Positives + False Positives)

High precision means fewer unnecessary review comments.

Recall

Recall asks:

Of the real issues present in the benchmark, how many did the reviewer identify?

Conceptually:

Recall =
True Positives /
(True Positives + False Negatives)

High recall means fewer real bugs are missed.

A strong review system needs a useful balance between the two.

Example Evaluation

Suppose a benchmark contains:

20 real defects

The AI reviewer identifies:

15 real defects

and also reports:

5 invalid findings

Then:

True Positives = 15
False Positives = 5
False Negatives = 5

Precision:

15 / (15 + 5) = 75%

Recall:

15 / (15 + 5) = 75%

These values are illustrative. They should not be presented as actual GitHub Copilot performance.

Build a Ground-Truth Dataset

A benchmark requires a reliable answer key.

For each pull request, document:

Pull Request
Expected Issue
Severity
Relevant File
Relevant Line
Reason

For example:

PR	Issue	Severity	Expected
PR-01	SQL injection	Critical	Detect
PR-02	Missing tenant filter	High	Detect
PR-03	Safe refactoring	None	No finding
PR-04	Incorrect null handling	Medium	Detect

This dataset becomes the ground truth.

Without ground truth, it is difficult to determine whether an AI review comment is actually correct.

Test Different Bug Categories

Do not create a benchmark containing only SQL injection.

Include multiple categories:

Security
Logic
Null handling
Concurrency
Performance
Error handling
Authentication
Authorization
Resource management
API contracts

For .NET applications, examples could include:

async/await mistakes
incorrect cancellation handling
improper disposal
LINQ logic errors
Entity Framework queries
Npgsql usage
ASP.NET Core authorization

This provides a broader evaluation of review quality.

Test Safe Code Too

A benchmark containing only vulnerable code encourages reviewers to report something everywhere.

Add deliberately safe examples.

For example:

await using var connection =
    await dataSource.OpenConnectionAsync(
        cancellationToken);

The test should verify that the reviewer does not invent security or reliability problems without evidence.

Safe examples are essential for measuring false positives.

Review Comment Classification

After running the AI review, classify every comment.

Use categories such as:

True Positive
False Positive
Duplicate
Suggestion
Informational
Missed Bug

A simple evaluation table can be:

Review Comment	Expected Issue?	Classification
SQL injection	Yes	True Positive
Missing validation	No	False Positive
Security finding repeated twice	Yes	Duplicate
Performance suggestion	No	Suggestion

This gives the team more information than raw comment counts.

Severity Matters

Not every finding has the same impact.

A useful classification is:

Critical
High
Medium
Low
Informational

For example:

SQL injection
    ↓
Potentially critical

Unused variable
    ↓
Potentially low

The benchmark should evaluate whether important defects are identified, not simply whether any comment was generated.

Testing Review Quality in Azure Repos

If the source repository is hosted in Azure Repos, verify how the GitHub Copilot review workflow accesses and reviews the code.

Do not assume that every GitHub Copilot code-review capability available for GitHub-hosted repositories applies identically to Azure Repos.

For a connected workflow, document:

Repository provider
Integration mechanism
Authentication
Pull-request source
Code-review trigger
Review output

The benchmark should evaluate the actual integration used by the organization rather than a different repository configuration.

Review Workflow for .NET Teams

A practical workflow can look like:

Developer
    ↓
Pull Request
    ↓
Automated Build
    ↓
Automated Tests
    ↓
AI Code Review
    ↓
Human Review
    ↓
Merge

AI review should supplement rather than replace human review.

A reviewer should still evaluate:

Business logic
Architecture
Security boundaries
Performance
Maintainability

Testing Security Findings

Security-related test cases should be especially controlled.

For example:

var sql =
    "SELECT * FROM users WHERE name = '" +
    userInput +
    "'";

Expected:

Security finding → Yes

Then:

const string sql = """
    SELECT id, name
    FROM users
    WHERE name = @name;
    """;

Expected:

Injection finding → No

This allows the benchmark to test both detection and restraint.

Testing Performance Findings

AI reviewers may identify inefficient database access.

For example:

var users =
    await dbContext.Users.ToListAsync();

var activeUsers =
    users.Where(x => x.IsActive).ToList();

A review might recommend filtering at the database level:

var activeUsers =
    await dbContext.Users
        .Where(x => x.IsActive)
        .ToListAsync();

The benchmark should determine whether such a recommendation is appropriate for the specific scenario.

Not every optimization suggestion is automatically a defect.

Testing Logic Bugs

Consider:

if (order.Status == "Paid" ||
    order.Status == "Shipped")
{
    CancelOrder(order);
}

If cancellation should only be allowed for unpaid orders, this may represent a business-logic problem.

AI code review may or may not detect it depending on the available context.

This demonstrates an important limitation:

Code
  +
Repository Context
  +
Business Requirements

is often necessary to identify semantic bugs.

Common False Positive Sources

AI reviewers can generate unnecessary findings when:

Existing validation occurs elsewhere.
A framework already handles the concern.
A code pattern is intentional.
The reviewer lacks business context.
A performance recommendation is workload-dependent.
A warning duplicates an existing analyzer finding.

Teams should therefore classify findings before turning them into mandatory fixes.

Common Missed Bug Sources

Missed bugs can occur when:

The defect depends on business rules.
Required context is outside the changed file.
The vulnerability requires multiple code paths.
The issue depends on runtime configuration.
The problem involves concurrency.
The application has complex authorization rules.

These limitations reinforce the role of human review.

Troubleshooting Poor Review Results

Too Many False Positives

Review the prompts, repository context, coding conventions, and benchmark cases.

Important Bugs Are Missed

Add more repository context and test cases covering the missing defect category.

Duplicate Findings

Compare AI findings with existing static analyzers and linters.

Security Issues Are Missed

Use dedicated security analysis tools in addition to AI review.

Results Vary Between Runs

Record the model configuration, repository state, and review environment so results can be reproduced as closely as practical.

Best Practices

Build a ground-truth benchmark before evaluating AI review quality.
Include both vulnerable and safe code.
Test security, logic, performance, and reliability issues.
Measure false positives and missed bugs separately.
Use precision and recall when appropriate.
Classify findings by severity.
Test realistic .NET code and repository structures.
Keep human review in the approval process.
Avoid treating every AI suggestion as a mandatory defect.
Compare AI findings with existing static-analysis tools.
Record benchmark conditions so results can be repeated.
Re-evaluate the system when models, integrations, or repository structures change.

Advantages and Disadvantages

Advantages

Can provide an additional review layer.
Can identify some defects before human review.
Can highlight potential security and performance problems.
Can review repetitive changes consistently.
Can reduce the amount of code a developer must inspect manually.

Disadvantages

Can generate false positives.
Can miss real defects.
May misunderstand business logic.
Review quality depends on available repository context.
Findings still require human validation.
AI review should not replace dedicated security and static-analysis tooling.

Conclusion

AI code review is most useful when treated as an additional engineering control rather than an automatic source of truth.

For a meaningful evaluation, use a controlled benchmark:

Known Code Changes
        ↓
Ground-Truth Issues
        ↓
AI Code Review
        ↓
Classify Findings
        ↓
True Positives
False Positives
False Negatives
        ↓
Measure Review Quality

For teams working with .NET and Azure Repos, the most valuable test is not simply counting how many comments an AI reviewer produces. Measure whether it identifies meaningful defects while avoiding unnecessary warnings.

Security vulnerabilities, authorization failures, logic errors, performance problems, and safe refactorings should all be represented in the test suite.

The final decision should remain with human reviewers. AI can expand the review surface and identify issues developers may overlook, but production code still requires engineering judgment, automated tests, static analysis, and security controls.

A well-designed benchmark makes that relationship measurable: the goal is not more AI comments; the goal is better code review with fewer missed bugs and less review noise.