GitHub Copilot Code Review: Measuring Effort Levels Against Defect Detection

Code review is one of the most important quality gates in a software development workflow. A reviewer needs to understand what changed, identify potential defects, check maintainability, and decide whether the change is safe to merge.

AI-assisted code review changes that workflow. Instead of relying entirely on a developer to inspect every line, an AI reviewer can analyze a pull request and highlight potential problems.

GitHub Copilot code review includes configurable effort levels that affect how much reasoning and analysis the review performs. That creates an interesting engineering question: Does increasing review effort actually improve defect detection enough to justify the additional processing?

Rather than assuming that a higher effort level is automatically better, teams can measure the relationship between review effort, detected defects, false positives, and developer validation.

What Does AI Code Review Actually Need to Detect?

A useful code review system should identify more than syntax problems.

Consider a simple change:

public async Task<Order?> GetOrderAsync(int id)
{
    return await dbContext.Orders
        .FirstOrDefaultAsync(x => x.Id == id);
}

The code may compile correctly.

But a reviewer might still need to consider:

Authorization
Tenant isolation
Performance
Null handling
Logging
Data exposure
Transaction behavior
Existing application conventions

AI code review is therefore most useful when it can identify risks that are not immediately visible from compilation.

Understanding Review Effort

The idea behind configurable effort is straightforward.

A lower-effort review can prioritize speed, while a higher-effort review can spend more computational work analyzing the change.

Conceptually:

Pull Request
     |
     v
AI Code Review
     |
     +---- Low Effort
     |       |
     |       v
     |    Faster Review
     |
     +---- Medium Effort
     |       |
     |       v
     |    Deeper Analysis
     |
     +---- High Effort
             |
             v
        More Analysis

The important question is whether additional effort produces additional useful findings.

That should be measured rather than assumed.

Designing a Defect Detection Experiment

A practical experiment needs a controlled dataset.

Start with pull requests or code changes containing known issues.

For example:

Defect Category	Example
Security	Missing authorization
Reliability	Unhandled failure
Performance	Inefficient database query
Correctness	Incorrect condition
Concurrency	Race condition
Resource management	Missing disposal
API contract	Incorrect response
Validation	Missing input validation

The defects should be independently verified before running the experiment.

Otherwise, it becomes difficult to determine whether the AI actually discovered a problem.

Establish a Human-Verified Ground Truth

Before evaluating an AI reviewer, create a ground-truth set.

For each change, document:

Change ID
Defect
Location
Severity
Expected Finding

For example:

PR-042
Defect: Missing tenant filter
Location: OrderRepository.cs
Severity: High
Expected Finding:
Query must restrict records to the authenticated tenant.

The ground truth should be established without using the AI being evaluated.

This prevents circular evaluation.

Measure More Than the Number of Findings

Suppose one review configuration produces 20 findings and another produces 8.

That does not mean the first one is better.

The 20 findings could include many false positives.

Measure at least:

True positives
False positives
False negatives
Review effort
Time to validate findings
Severity distribution

A simple table can help:

Metric	Low Effort	High Effort
True positives	Measure	Measure
False positives	Measure	Measure
False negatives	Measure	Measure
Precision	Calculate	Calculate
Recall	Calculate	Calculate
Validation time	Measure	Measure

Precision and Recall

Two useful metrics are precision and recall.

Precision

Precision answers:

Of the issues reported by the AI, how many were actually valid?

Precision =
True Positives /
(True Positives + False Positives)

A high precision means developers are less likely to waste time investigating incorrect findings.

Recall

Recall answers:

Of the known defects, how many did the AI find?

Recall =
True Positives /
(True Positives + False Negatives)

A review configuration can have high precision but low recall.

For example, it might report only a few issues, but those issues are usually correct.

Another configuration may detect more real defects but also produce more noise.

Both dimensions matter.

Creating a Sample Code Change

Consider this example:

public async Task<Order?> GetOrderAsync(
    int orderId)
{
    return await dbContext.Orders
        .Include(x => x.Customer)
        .FirstOrDefaultAsync(x => x.Id == orderId);
}

Suppose the endpoint is authenticated but does not verify that the requested order belongs to the current tenant.

The implementation may look reasonable in isolation.

A security-focused review should ask:

Does the authenticated user have permission
to access this specific order?

This is exactly the type of defect that makes code review different from compilation.

Testing Multiple Effort Levels

Run the same pull request through each available review configuration.

Keep everything else consistent.

Pull Request
     |
     +----> Effort A
     |
     +----> Effort B
     |
     +----> Effort C
     |
     v
Compare Findings

Do not modify the code between runs.

Do not change the prompt or review instructions unless that variable is explicitly part of the experiment.

This makes the results easier to interpret.

Keep the Evaluation Blind

If possible, reviewers validating AI findings should not know which effort level produced a finding.

For example:

Finding A
Finding B
Finding C

The reviewer determines whether each finding is valid without knowing whether it came from a low- or high-effort review.

This reduces confirmation bias.

Measure Developer Validation Time

Finding defects is only part of the review workflow.

A developer still needs to inspect the finding.

Consider:

Review Configuration	Findings	Valid Findings	Validation Time
A	10	6	18 min
B	12	8	20 min
C	15	9	32 min

Configuration C detects the most valid issues but also requires substantially more validation effort.

Whether that tradeoff is worthwhile depends on the severity of the defects and the team's workflow.

The numbers above are examples only and should not be treated as benchmark results.

Severity Matters

Not all defects have the same value.

A useful evaluation should classify findings.

For example:

Critical
High
Medium
Low
Informational

A review configuration that detects one high-impact security vulnerability may be more valuable than one that identifies ten formatting issues.

Therefore, report findings by severity rather than using only a total count.

Example: Measuring Review Results in C#

The results can be represented with a simple model:

public sealed record ReviewFinding(
    string Category,
    string Severity,
    bool IsCorrect,
    int ValidationMinutes);

Then calculate precision:

public static double CalculatePrecision(
    IEnumerable<ReviewFinding> findings)
{
    var list = findings.ToList();

    var truePositives =
        list.Count(x => x.IsCorrect);

    if (list.Count == 0)
        return 0;

    return (double)truePositives / list.Count;
}

Recall requires the number of known defects:

public static double CalculateRecall(
    IEnumerable<ReviewFinding> findings,
    int knownDefects)
{
    if (knownDefects == 0)
        return 0;

    var truePositives =
        findings.Count(x => x.IsCorrect);

    return (double)truePositives / knownDefects;
}

These calculations are simple, but they make the evaluation more objective.

Build a Defect Taxonomy

A good benchmark should contain different classes of problems.

For example:

Security
|
+-- Authorization
+-- Injection
+-- Data exposure

Correctness
|
+-- Incorrect condition
+-- Boundary error
+-- Null handling

Performance
|
+-- N+1 query
+-- Excessive allocation
+-- Unnecessary database call

Reliability
|
+-- Missing retry
+-- Resource leak
+-- Incorrect failure handling

This prevents the experiment from being biased toward a single type of defect.

Avoid Synthetic-Only Testing

Artificially inserting obvious bugs can make an AI reviewer look more effective than it would be in normal development.

A stronger evaluation combines:

Carefully constructed test cases.
Historical defects.
Realistic code changes.
Different application layers.
Multiple defect categories.

Historical defects are particularly useful because they represent problems developers actually encountered.

Common Mistakes

Counting Every Finding as a Success

More findings do not necessarily mean better review quality.

Measure correctness.

Ignoring False Positives

Too many incorrect findings can reduce developer trust.

Using Only Trivial Bugs

Simple syntax or obvious null checks do not adequately test sophisticated review behavior.

Changing the Test Between Effort Levels

If the code changes between runs, the comparison is no longer controlled.

Ignoring Developer Time

An AI review that finds more issues but requires excessive manual validation may not improve the overall workflow.

Treating AI Review as Final Approval

AI findings should support human review and established engineering controls rather than becoming an automatic replacement for critical review decisions.

Security-Sensitive Code Requires Extra Care

AI code review is especially useful as an additional signal for security-sensitive changes.

Examples include:

Authentication
Authorization
Secret handling
Database queries
File uploads
Deserialization
External API calls

However, the absence of an AI finding should never be interpreted as proof that the code is secure.

Security review should remain layered.

Developer Review
      |
      +-- AI Review
      |
      +-- Static Analysis
      |
      +-- Security Testing
      |
      +-- Automated Tests
      |
      v
Release Decision

Best Practices for Measuring AI Code Review

Create a verified defect dataset.
Include multiple defect categories.
Keep the code changes identical across tests.
Compare multiple review effort levels.
Measure true positives.
Measure false positives.
Measure false negatives.
Calculate precision and recall.
Track severity.
Measure developer validation time.
Blind the validation process where practical.
Include historical defects.
Separate security-critical findings from cosmetic findings.
Do not treat AI review as a substitute for other security and quality controls.

Advantages and Disadvantages

Advantages

Can provide another review signal for pull requests.
May identify defects developers overlook.
Can help reviewers focus attention on potentially risky code.
Effort controls create an opportunity to evaluate speed versus analysis depth.
Findings can be measured using standard information-retrieval metrics.

Disadvantages

AI findings can contain false positives.
Some real defects may remain undetected.
Higher analysis effort can increase processing time.
Results can vary across codebases and defect types.
AI review does not eliminate the need for human judgment.

A Practical Evaluation Workflow

A repeatable experiment can look like this:

Historical / Controlled PRs
          |
          v
Ground-Truth Review
          |
          v
Run Effort Level A
          |
          v
Run Effort Level B
          |
          v
Run Effort Level C
          |
          v
Blind Finding Validation
          |
          v
Precision / Recall / Time
          |
          v
Engineering Recommendation

The final recommendation should be based on the team's actual results.

For example, a team might discover that a higher effort level provides significantly better detection of security defects but offers little additional value for routine formatting or simple CRUD changes.

That could justify using different review configurations for different classes of pull requests.

Conclusion

AI-assisted code review is most useful when it is evaluated like an engineering system rather than treated as a feature that is automatically better at higher effort.

The meaningful measurements are not simply the number of comments generated. Teams should examine true positives, false positives, false negatives, severity, and the amount of developer time required to validate findings.

Configurable review effort makes this comparison particularly interesting because it allows teams to investigate whether deeper analysis produces enough additional defect detection to justify its cost.

The best result may not be a single effort level for every repository. A mature engineering workflow can instead use evidence from its own codebase to determine where deeper AI review provides measurable value and where a faster review is sufficient.