Measuring GitHub Copilot Adoption Without Storing Developer Conversations

When a development team introduces GitHub Copilot, managers often want to know whether developers are using it and which features are becoming part of their daily workflow. At the same time, developers may have concerns about how their activity is being monitored.

These concerns are reasonable. A developer might use Copilot to understand an unfamiliar codebase, investigate a production error, or write a unit test. Measuring whether Copilot is being adopted does not automatically require collecting the text of those conversations.

GitHub provides usage metrics that organizations can use to understand Copilot activity. With the right reporting design, a team can track active users, compare feature adoption, and monitor trends without building a separate system to store prompts and generated responses.

The challenge is to collect enough information to answer useful engineering questions while avoiding unnecessary data collection. This article explains how to approach that problem, process usage reports, calculate adoption rates, and build a reporting pipeline that respects developer privacy.

Understanding Copilot Usage Metrics

Before writing any code, decide what the organization actually needs to measure.

The number of assigned licenses is not the same as the number of active users. Likewise, frequent Copilot usage does not necessarily mean that a developer produces better code or completes tasks faster.

A useful adoption report can include the following metrics.

Metric	Purpose	Limitation
Assigned users	Shows how many developers have access	Does not prove that they use Copilot
Active users	Measures usage according to the report's activity definition	Does not explain why someone uses Copilot
Feature adoption	Shows activity involving supported features	Does not measure the quality of generated code
Code suggestion metrics	Provides information about suggestion activity and acceptance where available	Acceptance does not guarantee correctness
Adoption trends	Shows how usage changes over time	Does not independently measure productivity

The exact fields depend on the Copilot usage report, subscription, and API available to the organization.

For example, a team may want to know whether developers are using Copilot Chat more frequently after an internal training session. Feature-level activity can help answer that question without collecting the conversations themselves.

However, usage metrics should not be confused with proof of productivity improvements. To evaluate productivity, teams need additional evidence, such as developer feedback, code review findings, and the time required to complete comparable tasks.

Choosing the Right Reporting Method

GitHub provides built-in Copilot usage dashboards and APIs for organizations that need more control over reporting.

The dashboard is a sensible starting point because it avoids the need to maintain a separate data pipeline. An API-based approach becomes useful when a team needs scheduled reports, historical comparisons, or integration with an existing analytics system.

A custom reporting process generally follows these steps:

Authenticate using an identity with the required permissions.
Request the appropriate Copilot usage report.
Download the report if the API returns a temporary download link.
Extract the fields required for adoption reporting.
Calculate aggregate metrics.
Store the resulting summary rather than retaining unnecessary raw data.

The API endpoint and response format depend on the report being requested. Some reports return data directly, while others provide a download URL. Always check the actual response rather than assuming that every request returns the same structure.

Retrieving a Usage Report

For example, an organization can request a daily Copilot usage report using an appropriate organization-level endpoint.

GET /orgs/{org}/copilot/metrics/reports/organization-1-day?day=2026-10-12

A request can be made using curl:

curl --request GET \
  --url "https://api.github.com/orgs/ORG_NAME/copilot/metrics/reports/organization-1-day?day=2026-10-12" \
  --header "Accept: application/vnd.github+json" \
  --header "Authorization: Bearer ${GITHUB_TOKEN}" \
  --header "X-GitHub-Api-Version: 2026-03-10"

Replace ORG_NAME with the organization name and supply an appropriately authorized token through the GITHUB_TOKEN environment variable.

The example uses a fixed reporting date for clarity. A production job should calculate the required reporting date according to its schedule and the endpoint's supported date range.

The token must have the required permissions to read Copilot metrics. Store it in a secret manager or protected CI/CD configuration rather than placing it directly in source code.

Also, do not assume the response contains the report itself. If GitHub returns a temporary download URL, the application must download the report before processing it. Such URLs can expire, so retrieve the file promptly and avoid recording signed URLs in application logs.

Processing Usage Data Without Retaining Unnecessary Fields

Usage reports may contain more information than a simple adoption dashboard needs. Some exports use newline-delimited JSON, also called NDJSON, in which each line represents a separate JSON object.

The following is an illustrative example of a simplified report:

{"reporting_period":"2026-10-12","active_users":72}
{"reporting_period":"2026-10-13","active_users":75}

These records demonstrate the format; they are not actual GitHub API response samples.

A Python script can read the file and extract only the reporting date and active-user count.

import json
from pathlib import Path

report_path = Path("usage-report.ndjson")
daily_totals = []

with report_path.open("r", encoding="utf-8") as report:
    for line in report:
        if not line.strip():
            continue

        record = json.loads(line)

        daily_totals.append({
            "reporting_period": record["reporting_period"],
            "active_users": record["active_users"]
        })

for item in daily_totals:
    print(
        f'{item["reporting_period"]}: '
        f'{item["active_users"]} active users'
    )

The script expects the simplified fields shown in the example. A production implementation must map the fields from the actual report schema and validate their types before processing them.

Notice that the script does not copy the complete source records into the output. It extracts only the information needed for this report.

This approach reduces unnecessary data handling, but it does not guarantee that the original download contains no identifying information. The source file should still be protected and deleted when it is no longer needed.

Calculating Adoption Rates

Once the usage data has been extracted, the reporting system can calculate the percentage of assigned users who were active during the selected period.

For example, assume an organization has 100 assigned Copilot users and 72 active users.

Adoption Rate = (Active Users / Assigned Users) * 100
              = (72 / 100) * 100
              = 72%

A Python function can perform the calculation while checking for invalid input.

def calculate_adoption_rate(
    assigned_users: int,
    active_users: int
) -> float:
    if assigned_users < 0 or active_users < 0:
        raise ValueError("User counts cannot be negative.")

    if active_users > assigned_users:
        raise ValueError(
            "Active users cannot exceed assigned users."
        )

    if assigned_users == 0:
        return 0.0

    return (active_users / assigned_users) * 100


rate = calculate_adoption_rate(100, 72)
print(f"Adoption rate: {rate:.1f}%")

Output:

Adoption rate: 72.0%

The validation assumes that both counts refer to the same user population and reporting period. If assigned users are counted at the end of the month but active users are counted over a different scope, the resulting percentage may be misleading.

In a production dashboard, consider displaying the reporting period alongside the adoption rate. If there are no assigned users, displaying "Not applicable" may be more meaningful than displaying zero percent.

Building a Privacy-Conscious Reporting Pipeline

A reliable reporting system should separate data retrieval, processing, aggregation, and storage.

1. Retrieve the report securely

Use an authorized identity with the minimum required permissions. Keep tokens out of source control and application logs.

2. Validate the report

Check that the response contains the expected fields and that counts and dates have valid values. Handle missing reports and expired download links explicitly.

3. Aggregate the required metrics

Calculate daily or weekly active-user totals and feature adoption statistics. Avoid copying fields that do not contribute to the reporting objective.

4. Store only the required results

A reporting table might contain a reporting date, organization identifier, assigned-user count, active-user count, and adoption percentage.

There is no need to add columns for prompt text or generated responses when the purpose is simply to measure adoption.

5. Apply retention and access controls

Restrict individual-level reports to authorized personnel. Delete temporary exports after processing unless there is a documented reason to retain them.

Even without conversation text, usernames and detailed usage patterns can identify developers. Aggregated reporting is preferable for routine management reviews when individual-level detail is unnecessary.

Advantages and Disadvantages

Advantages

Reduced data collection: Adoption reporting can focus on activity metrics rather than storing developer conversations.

Consistent reporting: Scheduled reports make it easier to compare usage across reporting periods.

Better rollout decisions: Feature-level trends can help identify where additional training or onboarding may be useful.

Flexible integration: Aggregated results can feed existing dashboards and analytics systems.

Disadvantages

Usage does not equal productivity: Activity counts alone cannot establish that Copilot improves code quality or delivery speed.

API integration requires maintenance: Permissions, response formats, and reporting availability need to be monitored.

Privacy risks remain: User-level activity data can still reveal individual working patterns.

Metric definitions matter: Reports with different scopes or activity definitions cannot always be compared directly.

Common Mistakes and Troubleshooting

Problem	What to check
API request returns `403 Forbidden`	Verify the token permissions and organization reporting policy
Report contains no data	Check the requested date, report availability, and endpoint requirements
Download fails	Confirm that the temporary download link has not expired
JSON parsing fails	Verify whether the response is NDJSON, a JSON array, or a different structure
Adoption rate looks incorrect	Confirm that assigned and active users cover the same population and period
Raw reports accumulate in storage	Review temporary-file cleanup and data-retention rules

A successful API request does not mean the entire pipeline has completed successfully. Downloading, parsing, validation, aggregation, and storage can fail independently.

Log processing status, report dates, and relevant error details. Avoid logging access tokens, temporary download URLs, or complete report contents unless there is a specific and approved operational need.

Best Practices for Production Use

Start with GitHub's built-in dashboard before developing a custom reporting service.
Define the adoption questions before selecting metrics.
Use a dedicated identity with the minimum required permissions.
Validate report fields instead of assuming the schema never changes.
Keep credentials and temporary download links out of logs.
Retain aggregated results rather than raw exports when possible.
Explain to developers what is measured and who can access the reports.
Never use activity counts alone to rank individual developers.

Summary

GitHub Copilot adoption can be measured without building a system that stores the prompts developers enter or the responses Copilot generates. Usage dashboards and APIs provide a starting point for tracking active users, feature adoption, and changes over time.

The key is to design the reporting process carefully. Retrieve the required report, validate its structure, extract only the necessary fields, calculate meaningful metrics, and store the resulting summaries securely.

For most engineering teams, a small reporting pipeline is enough to understand whether Copilot is being adopted and where additional support may help. Collecting more data than the reporting objective requires adds complexity without necessarily providing better insight.