Copilot  

Measuring AI Code Review Quality with GitHub Copilot Metrics

Unmeasured AI Code Quality in Review Pipelines

As enterprise engineering organizations roll out GitHub Copilot across development teams, code generation velocity increases significantly. Developers write more functions, refactor larger code blocks, and submit pull requests (PRs) at an unprecedented rate. However, accelerating initial code generation without measuring the quality and review throughput of AI-assisted code introduces severe operational bottlenecks in DevOps pipelines:

  • PR Review Bottlenecking: Generating code faster creates a higher volume of pull requests, overwhelming senior human reviewers and increasing PR queue waiting times (Cycle Time).

  • Unmonitored Defect Rates: Merging AI-suggested code without tracking acceptance patterns, review revision cycles, or static analysis security debt risks introducing silent bugs into production.

  • Lack of Visibility into AI Review Efficacy: Organizations adopting automated features like Copilot Code Review lack visibility into whether AI-driven code reviews actually surface actionable defects or merely generate noise.

  • Fragmented Engineering Metrics: Relying on subjective developer surveys or vanity adoption metrics (such as raw seat activation counts) provides no actionable telemetry on actual code quality, defect escape rates, or pull request merge velocity.

To balance speed with software quality, engineering leaders and DevOps teams must implement a data-driven measurement pipeline. By querying the GitHub Copilot Metrics REST API (/orgs/{org}/copilot/metrics) alongside Pull Request review metrics in .NET, teams can correlate AI suggestion acceptance rates with pull request review efficiency, code scanning alerts, and cycle time performance.

Architectural Topology: Copilot Telemetry to DevOps Analytics

The GitHub Copilot Metrics API exposes detailed telemetry covering IDE suggestions, chat interactions, pull request review summaries, and repository-level code generation metrics. A .NET background service polls these endpoints, calculates quality indicators, and persists metrics into an enterprise analytics database for real-time visualization.

┌─────────────────────────────────────────────────────────────┐
│                   GitHub Telemetry Engine                   │
│   (Copilot Usage API / Pull Request & Code Review APIs)     │
└──────────────────────────────┬──────────────────────────────┘
                               │
                REST API (JSON Telemetry Payload)
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│             .NET Metrics Ingestion Service                  │
│     (Processes Acceptance Rates, PR Reviews, & Quality)     │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│            Enterprise Analytics & Storage Engine            │
│       (TimescaleDB / Azure Data Explorer / Power BI)        │
└─────────────────────────────────────────────────────────────┘

The table below contrasts subjective developer surveys against automated API-driven telemetry for measuring AI code review quality:

Measurement DimensionSubjective Developer SurveysAutomated Copilot Telemetry & Metrics API
Data SourcePeriodic manual developer questionnaires.Direct GitHub REST API streams (/copilot/metrics).
Data PrecisionLow; subject to recall bias and subjective opinion.High; captures exact lines suggested, accepted, and reviewed.
Evaluation FrequencyMonthly or quarterly.Continuous, daily aggregated metric execution.
PR Quality CorrelationNone; disconnected from git commit histories.Direct linkage between acceptance rates, PR cycle time, and review comments.
Automated AlertingImpossible; data is static and backward-looking.Native; triggers alerts when code review quality or acceptance drops.

Implementing a Copilot Metrics Ingestion Pipeline in .NET

The following step-by-step implementation demonstrates how to build a C# service that retrieves Copilot usage metrics from the GitHub REST API, calculates code review quality indicators, and evaluates PR review efficiency.

Step 1: Install Package Dependencies

Add the required HTTP, JSON, and resilience extensions to your .NET project:

Bash

dotnet add package Microsoft.Extensions.Http
dotnet add package System.Text.Json
dotnet add package Polly

Step 2: Define GitHub Copilot Metrics API Data Contracts

Define strongly typed C# records matching the GitHub Copilot Usage Metrics REST API JSON response schemas.

C#

using System.Text.Json.Serialization;

public record CopilotDayMetrics(
    [property: JsonPropertyName("day")] string Day,
    [property: JsonPropertyName("total_active_users")] int TotalActiveUsers,
    [property: JsonPropertyName("total_engaged_users")] int TotalEngagedUsers,
    [property: JsonPropertyName("copilot_ide_code_completions")] CopilotIdeCompletions? IdeCompletions,
    [property: JsonPropertyName("copilot_ide_chat")] CopilotIdeChat? IdeChat,
    [property: JsonPropertyName("copilot_dotcom_pull_requests")] CopilotPullRequestMetrics? PullRequestMetrics);

public record CopilotIdeCompletions(
    [property: JsonPropertyName("total_suggestions_count")] int TotalSuggestionsCount,
    [property: JsonPropertyName("total_acceptances_count")] int TotalAcceptancesCount,
    [property: JsonPropertyName("total_lines_suggested")] int TotalLinesSuggested,
    [property: JsonPropertyName("total_lines_accepted")] int TotalLinesAccepted,
    [property: JsonPropertyName("editors")] List<CopilotEditorMetric>? Editors);

public record CopilotEditorMetric(
    [property: JsonPropertyName("name")] string Name,
    [property: JsonPropertyName("models")] List<CopilotModelMetric>? Models);

public record CopilotModelMetric(
    [property: JsonPropertyName("name")] string Name,
    [property: JsonPropertyName("languages")] List<CopilotLanguageMetric>? Languages);

public record CopilotLanguageMetric(
    [property: JsonPropertyName("name")] string Name,
    [property: JsonPropertyName("total_code_suggestions")] int TotalSuggestions,
    [property: JsonPropertyName("total_code_acceptances")] int TotalAcceptances);

public record CopilotPullRequestMetrics(
    [property: JsonPropertyName("total_pr_summaries_created")] int TotalPrSummariesCreated,
    [property: JsonPropertyName("total_pr_reviews_created")] int TotalPrReviewsCreated);

public record CodeReviewQualityScore(
    DateTime Date,
    double AcceptanceRatePercentage,
    double LineAcceptanceRatio,
    int TotalPullRequestSummaries,
    int TotalPullRequestReviews,
    string QualityRating);

Step 3: Implement the GitHub Copilot Metrics API Client

Construct a C# service that authenticates with GitHub using a Personal Access Token (PAT) or GitHub App installation token to fetch metrics for an enterprise organization.

C#

using System.Net.Http.Headers;
using System.Text.Json;

public class GitHubCopilotMetricsClient
{
    private readonly HttpClient _httpClient;

    public GitHubCopilotMetricsClient(HttpClient httpClient, string githubPatToken)
    {
        _httpClient = httpClient;
        _httpClient.BaseAddress = new Uri("https://api.github.com/");
        _httpClient.DefaultRequestHeaders.UserAgent.Add(new ProductInfoHeaderValue("DotNetCopilotMetricsService", "1.0"));
        _httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", githubPatToken);
        _httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/vnd.github+json"));
        _httpClient.DefaultRequestHeaders.Add("X-GitHub-Api-Version", "2022-11-28");
    }

    public async Task<List<CopilotDayMetrics>> FetchOrganizationMetricsAsync(string organizationName, CancellationToken ct = default)
    {
        string requestUri = $"orgs/{organizationName}/copilot/metrics";

        using var response = await _httpClient.GetAsync(requestUri, ct);
        response.EnsureSuccessStatusCode();

        string json = await response.Content.ReadAsStringAsync(ct);
        var metricsList = JsonSerializer.Deserialize<List<CopilotDayMetrics>>(json);

        return metricsList ?? new List<CopilotDayMetrics>();
    }
}

Step 4: Calculate Quality and Review Efficiency Metrics

Implement a processor that evaluates raw telemetry and produces actionable engineering quality indicators.

C#

public class CopilotQualityEvaluator
{
    public CodeReviewQualityScore EvaluateDailyQuality(CopilotDayMetrics dayData)
    {
        int totalSuggestions = dayData.IdeCompletions?.TotalSuggestionsCount ?? 0;
        int totalAcceptances = dayData.IdeCompletions?.TotalAcceptancesCount ?? 0;

        int linesSuggested = dayData.IdeCompletions?.TotalLinesSuggested ?? 0;
        int linesAccepted = dayData.IdeCompletions?.TotalLinesAccepted ?? 0;

        // Calculate Acceptance Rate Percentages
        double acceptanceRate = totalSuggestions > 0 
            ? (double)totalAcceptances / totalSuggestions * 100.0 
            : 0.0;

        double lineAcceptanceRatio = linesSuggested > 0 
            ? (double)linesAccepted / linesSuggested * 100.0 
            : 0.0;

        int prSummaries = dayData.PullRequestMetrics?.TotalPrSummariesCreated ?? 0;
        int prReviews = dayData.PullRequestMetrics?.TotalPrReviewsCreated ?? 0;

        // Categorize Code Quality Health Rating based on acceptance stability
        string rating = DetermineQualityRating(acceptanceRate, lineAcceptanceRatio);

        DateTime parsedDate = DateTime.TryParse(dayData.Day, out var dt) ? dt : DateTime.UtcNow;

        return new CodeReviewQualityScore(
            Date: parsedDate,
            AcceptanceRatePercentage: Math.Round(acceptanceRate, 2),
            LineAcceptanceRatio: Math.Round(lineAcceptanceRatio, 2),
            TotalPullRequestSummaries: prSummaries,
            TotalPullRequestReviews: prReviews,
            QualityRating: rating);
    }

    private static string DetermineQualityRating(double acceptanceRate, double lineRatio)
    {
        // High acceptance with balanced line ratio indicates targeted, useful AI code completions
        if (acceptanceRate >= 30.0 && lineRatio >= 25.0)
        {
            return "Optimal (High Usage & High Acceptance)";
        }
        if (acceptanceRate < 15.0)
        {
            return "Needs Calibration (High Noise / Low Acceptance)";
        }
        return "Standard (Healthy Assistance)";
    }
}

Architectural Advantages and Disadvantages

Advantages

  • Data-Driven Engineering Visibility: Replaces subjective feedback with exact, quantitative API metrics covering suggestion volumes and acceptance rates.

  • Early Detection of Code Noise: Identifies when developers are repeatedly rejecting Copilot completions, signaling outdated instructions or improper prompt setup.

  • Correlates AI Usage with PR Throughput: Tracks how AI pull request summaries and automated reviews influence code review cycle times.

Disadvantages

  • Lacks Direct Runtime Defect Attribution: Metrics API numbers track IDE acceptances but do not directly measure whether accepted code caused production bugs.

  • Requires Organization-Level API Permissions: Accessing /orgs/{org}/copilot/metrics requires Organization Owner or View Copilot Metrics administrative permissions.

Enterprise Best Practices

  1. Correlate Metrics with DORA Key Indicators: Combine Copilot acceptance data with DORA metrics (Deployment Frequency, Change Failure Rate, Cycle Time) to verify that AI generation translates into safe delivery gains.

  2. Break Down Telemetry by Programming Language: Use the API's language breakdown fields (total_code_acceptances per language) to see if specific languages (e.g., C# vs. Python) yield different quality scores.

  3. Set Up Threshold Alerts for Acceptance Drops: Trigger alerts when team acceptance rates drop below 15%, which usually indicates prompt instruction drift or framework mismatches.

  4. Combine IDE Metrics with Automated SAST Scanning: Pair Copilot metrics with static code analysis tools (such as SonarQube or GitHub CodeQL) to verify that higher code volume does not increase security debt.

Common Mistakes to Avoid

  • Treating Acceptance Rate as a Strict Developer KPI: Forcing developers to hit a target acceptance rate encourages blind acceptance of AI code without proper human review.

  • Ignoring Pull Request Review Bottlenecks: Tracking code generation speed while ignoring PR review duration allows PR queues to clog up downstream.

  • Conflating Active Users with Engaged Quality Users: Relying solely on total_active_users without measuring line acceptance ratios masks idle or unproductive tool usage.

Troubleshooting Guide

Issue 1: HTTP 403 Forbidden Response from Copilot Metrics API

  • Root Cause: The personal access token or GitHub App installation token lacks the required manage_billing:copilot or read:org administrative scopes.

  • Resolution: Ensure the token carries administrative metrics access permissions and that the Copilot Metrics policy is enabled in the GitHub organization settings.

Issue 2: Metrics Payload Returns Empty or Zero Values

  • Root Cause: Telemetry collection requires a 24-hour aggregation window, or client IDEs have disabled telemetry data sharing.

  • Resolution: Verify that developer IDEs have telemetry enabled and query historical daily endpoints (/copilot/metrics) rather than real-time intraday endpoints.

Issue 3: Discrepancies Between IDE Completions and PR Metrics

  • Root Cause: Users generated code in the IDE but used external tools to open PRs without generating automated Copilot PR summaries.

  • Resolution: Ensure GitHub Copilot for Pull Requests is configured on the organization repositories.

Frequently Asked Questions (FAQs)

1. What endpoints does the GitHub Copilot Metrics API provide?

GitHub provides REST API endpoints at both the organization and enterprise levels (/orgs/{org}/copilot/metrics and /enterprises/{enterprise}/copilot/metrics), returning daily aggregated usage data for IDE completions, chat, and pull requests.

2. How is acceptance rate calculated in Copilot metrics?

Acceptance rate is calculated as the total number of accepted suggestions divided by the total number of suggestions presented to the developer inside the IDE.

3. Does the Copilot Metrics API expose sensitive source code?

No. The API returns aggregated telemetry metrics (counts, line numbers, programming language names, and feature interactions). It never transmits or exposes source code text or prompt contents.

Conclusion

Measuring AI code review quality with GitHub Copilot metrics shifts AI adoption from guesswork to a data-driven engineering discipline. By building automated telemetry pipelines in .NET to track acceptance rates, line ratios, and pull request review metrics, DevOps leaders can optimize AI workflows, eliminate review bottlenecks, and ensure software quality at scale.

Membership not found