Introduction
A lot of teams are already using AI coding agents to write code, fix bugs, create tests, and prepare pull requests. But there is a bigger question that often gets missed:
Is the overall software delivery process actually getting faster?
Generating code faster does not necessarily mean delivering software faster.
A developer might spend 10 minutes asking an agent to implement a feature, but the pull request could then spend hours waiting for review, fail CI several times, require multiple rounds of rework, or be rewritten by a developer.
That is why measuring the complete chat-to-PR workflow is more useful than measuring how quickly an AI agent generates code.
The workflow can be viewed as:
Team Conversation
|
v
AI Task
|
v
Repository Investigation
|
v
Implementation
|
v
Testing
|
v
Pull Request
|
v
Review
|
v
Merge
GitHub Copilot cloud agent can work asynchronously on repository tasks, create branches, make code changes, and open pull requests. Developers can then review the result and iterate on the same pull request.
The interesting engineering problem is therefore not simply measuring AI coding time. It is measuring how the entire path from a developer's request to a reviewable pull request changes.
What Is Chat-to-PR Delivery Time?
Chat-to-PR delivery time is the elapsed time between the moment a development task is clearly assigned to an AI agent and the moment a pull request containing the proposed implementation is ready for review.
A simple definition is:
Chat-to-PR Time =
PR Ready Timestamp - Task Start Timestamp
For example:
10:00 AM Task assigned to Copilot
10:08 AM Repository investigation completed
10:25 AM Code changes completed
10:31 AM Tests completed
10:35 AM Pull request created
The chat-to-PR time is:
35 minutes
This metric tells us how quickly the task moved from an instruction to a reviewable engineering artifact.
However, it does not tell us whether the implementation was good.
That is why it should be combined with additional quality and review metrics.
Why Measuring Only Code Generation Is Misleading
Imagine two development workflows.
Workflow A
Task started 10:00
Code generated 10:10
PR created 10:15
Review 2 hours
Rework 1 hour
Merge 4 hours later
Workflow B
Task started 10:00
Code generated 10:25
PR created 10:30
Review 15 minutes
Rework 5 minutes
Merge 50 minutes later
Workflow A generated the code faster.
But Workflow B delivered the change faster.
This is why engineering teams should avoid treating generation speed as the primary productivity metric.
A better model is:
AI Efficiency
+
Code Quality
+
Review Efficiency
+
CI Stability
+
Delivery Time
Together, these provide a much better picture.
GitHub Copilot's Agentic Workflow
Copilot cloud agent can be assigned repository tasks and work asynchronously.
A developer can assign an issue to Copilot, provide additional instructions, and let the agent work toward a pull request. GitHub also supports starting an agent task directly from a prompt, allowing developers to review and iterate on a branch before opening the pull request.
A typical workflow looks like:
Issue / Prompt
|
v
Copilot Agent
|
+--> Analyze repository
|
+--> Understand requirements
|
+--> Modify files
|
+--> Run tests
|
+--> Push changes
|
v
Pull Request
|
+--> Automated checks
|
+--> Copilot review
|
+--> Human review
|
v
Merge
GitHub's current agent workflow also allows developers to steer running sessions and iterate on changes before finalizing the pull request.
That makes it possible to measure each stage rather than treating the entire AI interaction as a single black box.
Metrics Worth Measuring
A useful measurement model should contain several timestamps.
| Metric | Definition |
|---|
| Task Start Time | When the agent receives the task |
| First Activity | When the agent begins repository work |
| Code Complete Time | When implementation is finished |
| Test Complete Time | When validation finishes |
| PR Creation Time | When the pull request is opened |
| First Review Time | When review begins |
| Review Complete Time | When review feedback is resolved |
| Merge Time | When the change reaches the target branch |
From these timestamps, teams can calculate several useful measurements.
Chat-to-PR Time
PR Creation - Task Start
This measures how quickly a task becomes a reviewable change.
PR-to-Review Time
First Review - PR Creation
This measures review latency.
Review-to-Merge Time
Merge - First Review
This measures how long the change takes to move through review and final validation.
Total Delivery Time
Merge - Task Start
This is often the most useful metric when evaluating the actual delivery process.
Example Measurement Model
Suppose a team records the following:
Task Started: 09:00
PR Created: 09:40
First Review: 10:15
Review Completed: 10:50
Merged: 11:05
The metrics become:
Chat-to-PR: 40 minutes
PR-to-Review: 35 minutes
Review Duration: 35 minutes
Review-to-Merge: 55 minutes
Total Delivery: 125 minutes
This gives a much more useful picture than saying:
"Copilot wrote the code in 40 minutes."
The actual engineering delivery took 125 minutes.
Measuring Quality Alongside Speed
Speed without quality can create more work.
For example, a fast agent-generated pull request might introduce:
GitHub recommends treating Copilot-generated pull requests with the same thoroughness as other contributions.
Therefore, teams should track quality indicators alongside delivery time.
Useful metrics include:
| Quality Metric | Why It Matters |
|---|
| CI failure rate | Shows whether generated changes pass automated checks |
| Review comments | Indicates review effort |
| Change requests | Measures rework |
| Follow-up commits | Shows how much correction was needed |
| Reverted PRs | Identifies problematic changes |
| Escaped defects | Measures production impact |
| Security findings | Tracks security-related problems |
The goal is not to minimize every number.
For example, fewer review comments are not automatically better. A strong review may identify important architectural issues.
The objective is to understand whether the workflow produces useful, maintainable code with reasonable delivery effort.
Tracking Agent Rework
One particularly useful metric is rework.
Suppose Copilot creates a pull request with 300 lines changed.
A reviewer identifies several problems:
Review 1
- Incorrect validation
- Missing authorization check
- Test coverage incomplete
The agent makes another change:
Review 2
- Retry behavior needs adjustment
The developer then makes a manual correction.
This tells us that the original PR was fast but required significant intervention.
A simple rework ratio can be:
Rework Ratio =
Lines Changed After Initial PR / Initial PR Lines
This should not be treated as a universal quality score, but it can help identify patterns across similar tasks.
Measuring Different Task Types Separately
A common mistake is putting every AI-generated task into one dataset.
A documentation update and a database migration are not comparable.
Consider separating tasks into categories:
Bug Fix
Feature
Refactoring
Test Addition
Documentation
Dependency Update
Infrastructure
Database Change
Security Fix
Then compare metrics within each category.
For example:
| Task Type | Chat-to-PR | Review Time | Rework | CI Failures |
|---|
| Documentation | 15 min | 8 min | Low | Low |
| Test Addition | 25 min | 15 min | Medium | Low |
| Bug Fix | 40 min | 30 min | Medium | Medium |
| Feature | 90 min | 60 min | Medium | Medium |
| Database Change | 75 min | 80 min | High | Medium |
The values above are only an example of how a team could structure its measurement model. They are not benchmark results.
This separation prevents misleading conclusions.
A Simple .NET Measurement Model
For a .NET development team, delivery events can be represented with a small model:
public sealed class AgentDeliveryMetrics
{
public DateTime TaskStarted { get; set; }
public DateTime? PullRequestCreated { get; set; }
public DateTime? FirstReview { get; set; }
public DateTime? ReviewCompleted { get; set; }
public DateTime? Merged { get; set; }
public TimeSpan? ChatToPr =>
PullRequestCreated - TaskStarted;
public TimeSpan? PrToReview =>
FirstReview - PullRequestCreated;
public TimeSpan? ReviewDuration =>
ReviewCompleted - FirstReview;
public TimeSpan? TotalDelivery =>
Merged - TaskStarted;
}
The model itself is simple.
The difficult part is collecting reliable timestamps from the development workflow.
Collecting Data From Pull Requests
A practical system can collect data from GitHub's pull request and repository activity.
For example, a simplified process could be:
GitHub Events
|
v
Data Collector
|
+--> Task Started
+--> PR Created
+--> Review Submitted
+--> Commit Pushed
+--> CI Completed
+--> PR Merged
|
v
Metrics Store
|
v
Dashboard
The important design decision is to keep the raw events.
Instead of storing only:
Chat-to-PR = 42 minutes
store the underlying timestamps as well.
That allows the team to recalculate metrics later when its definitions change.
Building a Simple Dashboard
A development analytics dashboard might show:
AI Development Workflow
Average Chat-to-PR
42 min
Average PR Review
31 min
Average Total Delivery
2h 18m
CI Failure Rate
8%
Average Review Iterations
1.7
More useful is a trend:
Week 1 2h 42m
Week 2 2h 31m
Week 3 2h 20m
Week 4 2h 18m
But again, trends need context.
A reduction in delivery time could happen because:
The metric alone cannot explain the reason.
Using Copilot for the Review Stage
The workflow does not have to stop when the pull request is created.
Copilot can also review pull requests and provide feedback. GitHub's current guidance recommends using code review as part of the pull request lifecycle, including early review of draft pull requests and re-review after substantial changes.
A practical workflow is:
Copilot Creates PR
|
v
Automated CI
|
v
Copilot Review
|
v
Developer Review
|
v
Changes
|
v
Re-review
|
v
Merge
This can reduce some of the mechanical review work while keeping humans responsible for important engineering decisions.
Avoiding the Wrong KPI
One of the biggest mistakes in AI adoption is choosing a metric that encourages the wrong behavior.
For example:
Goal:
Increase AI-generated lines of code.
This can encourage unnecessary code generation.
A better goal is:
Goal:
Reduce time required to deliver
small, correct, reviewable changes.
Similarly, maximizing the number of AI-created pull requests is not necessarily useful.
A team could create many small PRs while increasing review workload.
The measurement system should therefore reward delivery quality, not AI activity.
Common Mistakes
Measuring Only Agent Runtime
The time an agent spends executing is only one part of the workflow.
Ignoring Review Time
A 10-minute implementation followed by a three-hour review is not a 10-minute delivery.
Mixing Task Types
Comparing documentation tasks with complex architecture changes creates misleading averages.
Ignoring Rework
A fast first PR can still be expensive if developers need to rewrite large portions of it.
Using Averages Alone
Averages can hide unusual but important cases.
Track median and percentile values when the dataset becomes large enough.
Treating AI as the Only Variable
Delivery time is affected by CI performance, reviewer availability, task complexity, dependencies, and team processes.
Best Practices
Define the Metric Before Collecting Data
Write down exactly what starts and ends the measurement.
For example:
Start:
Agent receives a clearly defined engineering task.
End:
Pull request is created and passes the initial required checks.
Without a clear definition, different teams may measure different things.
Store Raw Events
Keep timestamps and event metadata rather than only aggregated numbers.
Segment the Data
Separate tasks by complexity and type.
Measure Quality
Always pair speed metrics with review, CI, security, and defect indicators.
Protect Developer Privacy
The goal should be process improvement, not surveillance of individual developers.
Prefer team-level trends over individual productivity scores.
Review the Measurement Model Regularly
As AI tooling changes, the workflow changes too.
A metric that made sense for autocomplete may not make sense for autonomous agents.
Advantages
Makes AI Adoption Measurable
Teams can move beyond opinions and look at actual workflow data.
Identifies Bottlenecks
If chat-to-PR time is low but PR-to-merge time is high, the bottleneck may be review or CI rather than coding.
Helps Compare Workflows
Teams can compare traditional development with agent-assisted workflows without relying only on anecdotal feedback.
Encourages Better Engineering Practices
Tracking rework, CI failures, and review effort naturally encourages teams to improve task definitions and validation.
Disadvantages and Limitations
Metrics Can Be Misinterpreted
A faster PR is not automatically a better PR.
Data Collection Takes Work
Reliable timestamps and event correlation require some engineering effort.
Task Complexity Is Difficult to Normalize
A small bug fix and a large feature cannot be compared directly.
Human Factors Matter
Review availability, team workload, and communication delays can influence delivery time.
Metrics Can Create Bad Incentives
Poorly chosen KPIs can encourage developers to optimize for AI activity instead of actual software quality.
A Practical Measurement Strategy
A team starting this experiment does not need a large analytics platform.
Begin with five metrics:
1. Chat-to-PR Time
2. PR-to-Review Time
3. Review-to-Merge Time
4. CI Failure Rate
5. Rework After Initial PR
Then classify each task by type.
After collecting enough data, look for patterns.
For example:
Question 1:
Are small bug fixes reaching PR faster?
Question 2:
Are AI-generated PRs requiring more review?
Question 3:
Which task categories benefit most?
Question 4:
Where is the biggest remaining bottleneck?
Question 5:
Is total delivery time actually improving?
These questions are much more valuable than simply asking whether developers are using Copilot.
Conclusion
Measuring GitHub Copilot agent workflows should be about more than counting how quickly AI writes code. The more useful question is whether a development task can move from a conversation or request to a clean, reviewable pull request with less overall effort. Chat-to-PR time is a good starting metric, but it becomes much more meaningful when combined with review time, CI failures, rework, and total delivery time. Teams should also be careful not to turn these numbers into individual productivity scores. The goal is to find bottlenecks, understand where agents genuinely help, and improve the software delivery process. In the end, a slightly slower AI-generated PR that is easy to review and merge can be much more valuable than a very fast PR that creates additional work for the team.