AI coding workflows are no longer limited to asking a model to complete a few lines of code. Modern coding agents can inspect repositories, use tools, execute multi-step tasks, and work with reusable skills and plugins.
GitHub Copilot Agent Plugins are designed to package these capabilities into portable units that can be used across supported Copilot environments. GitHub's plugin model can bundle components such as agents, skills, hooks, and MCP server configurations, allowing teams to reuse the same development workflow rather than rebuilding it separately for each client.
That portability creates an interesting engineering question:
Does the same plugin actually behave the same way in VS Code and the Copilot CLI?
The answer is important for teams standardizing AI-assisted development.
A plugin can be structurally portable while its practical behavior differs because the surrounding client has different interaction models, terminal capabilities, user workflows, permissions, or performance characteristics.
This article presents a practical benchmark methodology for comparing GitHub Copilot Agent Plugins across VS Code and the CLI without pretending that one environment is universally better.
What Is Being Benchmarked?
The objective is not to benchmark the underlying language model.
Instead, the benchmark focuses on the plugin execution environment.
Consider this architecture:
Same Plugin
|
┌─────────┴─────────┐
| |
VS Code CLI
| |
Copilot Client Copilot Client
| |
└─────────┬─────────┘
|
AI Model + Tools
The plugin remains constant while the client changes.
This makes it possible to measure differences caused by the development environment rather than by changing the workflow itself.
Why VS Code and CLI Are Different
VS Code provides a graphical development environment where the developer can inspect files, diffs, diagnostics, terminals, and agent interactions in one workspace.
The CLI provides a terminal-oriented workflow.
That distinction matters.
A developer might use the CLI for:
Repository automation
CI-like tasks
Batch changes
Terminal-heavy workflows
Remote development
Scripted workflows
VS Code may be more convenient for:
Interactive debugging
Visual diff inspection
Code navigation
Inline development
Manual review
Iterative editing
The same plugin can therefore have different practical performance characteristics even when its underlying capabilities are the same.
Benchmark Dimensions
A useful benchmark should measure at least six dimensions.
| Dimension | What to Measure |
|---|
| Task success | Did the agent complete the task? |
| Completion time | How long did the workflow take? |
| Tool usage | How many tool calls were required? |
| Interaction count | How much developer intervention was needed? |
| Accuracy | Did the resulting change satisfy the requirements? |
| Resource usage | CPU, memory, tokens, or other measurable consumption |
The benchmark should also capture failures.
A fast workflow that frequently produces incorrect changes is not better than a slower workflow with consistently correct results.
Create a Controlled Test Repository
The first step is to create a repository that can be reproduced across both environments.
For example:
copilot-plugin-benchmark/
│
├── src/
│ ├── Orders/
│ ├── Customers/
│ └── Payments/
│
├── tests/
│ ├── Orders.Tests/
│ └── Payments.Tests/
│
├── docs/
│ └── architecture.md
│
└── README.md
The repository should contain enough realistic complexity to require multiple agent steps.
Avoid using an extremely small project.
A plugin that changes one line of code is unlikely to expose meaningful differences between clients.
Define Repeatable Tasks
Each benchmark task should have a fixed specification.
For example:
Task 1: Add a Validation Rule
Add validation preventing negative order quantities.
Update unit tests.
Do not modify unrelated behavior.
Task 2: Refactor a Service
Extract payment validation into a separate service.
Preserve existing behavior.
Update tests.
Task 3: Diagnose a Failing Test
Identify why the payment test fails.
Fix the underlying issue.
Add a regression test.
Task 4: Repository-Wide Change
Rename the obsolete configuration property
across source code, tests, and documentation.
The tasks should be run independently.
Keep the Plugin Constant
The benchmark becomes less useful if the plugin itself changes between tests.
Use the same:
Plugin version
Skill definitions
Agent configuration
MCP configuration
Repository
Task instructions
Model
for both environments whenever technically possible.
The experimental variable should primarily be:
VS Code
vs
Copilot CLI
Warm-Up Runs Matter
The first run may behave differently from later runs because of:
Therefore, do not compare only the first execution.
A better experiment might use:
Warm-up: 1 run
Measured:
5–10 runs per task
The exact number depends on available resources.
Measure Task Completion Time
The simplest metric is wall-clock duration.
For each run:
Start
|
v
Plugin initialized
|
v
Task submitted
|
v
Agent executes
|
v
Task completed
|
v
Stop
Record:
start_time
end_time
duration_ms
A simple result table could look like:
| Task | VS Code | CLI |
|---|
| Validation | 42 s | 39 s |
| Refactoring | 88 s | 91 s |
| Test diagnosis | 74 s | 68 s |
| Repository change | 121 s | 115 s |
These numbers are illustrative only, not measured GitHub Copilot results.
The benchmark should use actual measurements from your environment.
Measure More Than Time
Time alone can be misleading.
Suppose:
VS Code
Duration: 80 seconds
Correctness: 100%
CLI
Duration: 55 seconds
Correctness: 70%
The CLI is faster, but it is not necessarily the better environment.
Therefore, calculate a broader score.
For example:
Task Success Rate
+
Correctness
+
Developer Intervention
+
Latency
The weighting should be decided before the experiment.
Developer Intervention
One of the most important differences between graphical and CLI workflows is human interaction.
Record how many times the developer had to intervene.
For example:
Agent execution
|
+-- User approves tool
|
+-- User clarifies request
|
+-- User fixes generated change
|
+-- Agent continues
A useful metric is:
Interventions per completed task
Lower is generally better, provided correctness remains high.
Measuring Tool Calls
Agent plugins can expose tools and skills that the agent uses to complete tasks.
Record:
Tool calls
Successful calls
Failed calls
Retries
Average call latency
For example:
| Task | Client | Tool Calls | Failed Calls |
|---|
| Validation | VS Code | 8 | 1 |
| Validation | CLI | 7 | 0 |
| Refactoring | VS Code | 15 | 2 |
| Refactoring | CLI | 14 | 1 |
Again, these are examples of how to structure benchmark results, not actual measurements.
The number of calls is not inherently a quality metric.
A capable agent may use fewer calls because it understands the repository more effectively.
Measuring Accuracy
Accuracy should be evaluated against explicit acceptance criteria.
For a task like:
Add validation for negative quantities.
the acceptance criteria might be:
[ ] Negative quantities rejected
[ ] Positive quantities still accepted
[ ] Existing tests pass
[ ] Regression test added
[ ] No unrelated files changed
Then calculate:
Accuracy =
Passed criteria / Total criteria
This is more useful than asking whether the agent "looked good."
Measuring Unnecessary Changes
AI agents sometimes modify files that were not part of the task.
Track:
Expected changed files
Actual changed files
Unexpected changed files
For example:
Expected:
src/Orders/OrderValidator.cs
tests/Orders.Tests/OrderValidatorTests.cs
Actual:
src/Orders/OrderValidator.cs
tests/Orders.Tests/OrderValidatorTests.cs
README.md
docs/architecture.md
The last two files may indicate unnecessary scope expansion.
A useful metric is:
Unexpected Change Rate
=
Unexpected changed files / Total changed files
Benchmarking Plugin Startup
Portability also includes startup behavior.
Measure the time required to make the plugin ready for use.
Client Start
|
v
Plugin Discovery
|
v
Plugin Initialization
|
v
Tools Available
Record:
Plugin discovery time
Initialization time
Tool availability time
If the plugin is used repeatedly throughout the day, startup overhead can become operationally significant.
Benchmarking Failure Recovery
Do not benchmark only successful workflows.
Introduce controlled failures.
For example:
Test 1:
MCP server unavailable
Test 2:
Tool returns an error
Test 3:
Repository contains invalid configuration
Test 4:
Requested file does not exist
Test 5:
Network-dependent tool unavailable
Then measure whether the agent:
Detects the problem.
Explains the problem.
Attempts an appropriate recovery.
Avoids repeatedly performing the same failed action.
Completes the task when possible.
This produces a much more realistic assessment of plugin behavior.
VS Code Benchmark Considerations
When benchmarking VS Code, control variables such as:
An active development environment can introduce noise.
For example, indexing or compilation occurring in the background can affect measurements.
The benchmark should therefore use a consistent workspace state.
CLI Benchmark Considerations
The CLI environment should also be controlled.
Record:
Shell
Operating system
Working directory
Environment variables
Git state
Terminal configuration
Network state
Avoid running unrelated commands in the same terminal session if they could affect measurements.
For automated testing, a clean process for each run provides better isolation.
Model Selection Must Remain Constant
If the goal is to compare clients, do not accidentally compare different models.
For example:
VS Code
-> Model A
CLI
-> Model B
would produce an invalid experiment.
Instead:
VS Code ──> Model A
CLI ──────> Model A
The same principle applies to reasoning settings and other model parameters.
GitHub has introduced configurable reasoning levels for Copilot cloud agent workflows, and higher reasoning levels can increase token consumption and execution time.
Therefore, model and reasoning configuration should be recorded as part of the benchmark environment.
Benchmark Data Model
A structured benchmark record might look like this:
{
"client": "cli",
"pluginVersion": "1.0.0",
"task": "order-validation",
"model": "same-model-for-all-runs",
"durationMs": 64120,
"toolCalls": 8,
"failedToolCalls": 0,
"interventions": 1,
"changedFiles": 2,
"unexpectedFiles": 0,
"success": true,
"acceptanceScore": 1.0
}
This makes the experiment reproducible and allows results to be analyzed later.
Example Benchmark Runner
A simple .NET benchmark harness can measure execution time around a task:
using System.Diagnostics;
public static async Task<BenchmarkResult> RunAsync(
Func<Task> operation)
{
var stopwatch = Stopwatch.StartNew();
try
{
await operation();
stopwatch.Stop();
return new BenchmarkResult(
true,
stopwatch.Elapsed);
}
catch
{
stopwatch.Stop();
return new BenchmarkResult(
false,
stopwatch.Elapsed);
}
}
public record BenchmarkResult(
bool Success,
TimeSpan Duration);
This does not measure Copilot internals. It provides a framework for recording the surrounding workflow.
For a serious benchmark, collect the agent-specific metrics separately.
Comparing Results
After collecting enough runs, calculate:
Median
Median latency is often more useful than the average because AI workflows can have occasional very slow executions.
p50 = median
p95
The 95th percentile helps expose slow-tail behavior.
p95 = latency below which 95% of executions complete
For developer tooling, tail latency can matter because one unusually slow operation can interrupt an interactive workflow.
Success Rate
Success Rate =
Successful Runs / Total Runs × 100
Intervention Rate
Intervention Rate =
Runs Requiring Intervention / Total Runs × 100
Example Results Dashboard
A final benchmark could look like:
| Metric | VS Code | CLI |
|---|
| Task success | 96% | 94% |
| Median latency | 72 s | 68 s |
| p95 latency | 148 s | 139 s |
| Avg. tool calls | 11.2 | 10.7 |
| Intervention rate | 18% | 21% |
| Unexpected changes | 3% | 4% |
These values are illustrative.
The important point is the structure of the comparison.
A real experiment should report measured values, test environment, plugin version, model configuration, task definitions, and sample size.
Common Benchmarking Mistakes
Changing Multiple Variables
If the plugin, model, repository, and client all change, the experiment cannot tell you what caused the difference.
Using One Run
AI workloads can vary significantly.
One run is not enough to establish a reliable performance difference.
Measuring Only Latency
Fast but incorrect automation is not useful.
Ignoring Human Intervention
A workflow requiring frequent manual correction may not actually save development time.
Comparing Different Models
This invalidates the client comparison.
Ignoring Failure Cases
Production systems encounter failures.
A benchmark that only tests successful requests provides an incomplete picture.
Best Practices
Define the Hypothesis First
For example:
The same Agent Plugin will complete repository maintenance tasks with similar correctness across VS Code and CLI, but interaction overhead will differ.
Then design the experiment to test that hypothesis.
Keep the Plugin Immutable
Use the same plugin version for every run.
Use Repeated Trials
Measure enough runs to identify typical and tail behavior.
Separate Human and Machine Time
Record:
Agent execution time
Developer interaction time
Total elapsed time
This can reveal where the actual productivity cost exists.
Record the Full Environment
Document:
OS
Client version
Plugin version
Model
Reasoning configuration
Repository commit
Network conditions
Task specification
Without this information, benchmark results become difficult to reproduce.
Advantages and Disadvantages
Advantages
Provides objective evidence for client selection.
Identifies workflow bottlenecks.
Helps teams standardize plugin usage.
Makes performance regressions easier to detect.
Separates perceived productivity from measurable behavior.
Disadvantages
AI workloads are inherently variable.
Reproducing identical agent behavior can be difficult.
Human interaction introduces additional variance.
Client updates can change results.
Performance differences may be workload-specific.
Final Thoughts
GitHub Copilot Agent Plugins create an interesting opportunity for standardized AI development workflows because the same packaged capabilities can be used across supported Copilot environments. The important question for engineering teams is not simply whether a plugin works in both VS Code and the CLI, but whether it produces comparable results under realistic development workloads.
A useful benchmark should therefore measure more than execution time. Task correctness, tool usage, developer intervention, unexpected changes, startup overhead, failure recovery, and latency distribution all provide valuable information.
The strongest conclusion will rarely be that one client is universally faster. Instead, the benchmark should identify which environment performs better for which class of development task.
For teams adopting AI agents at scale, that distinction matters. A plugin that performs well for interactive code exploration may not be the best choice for repository-wide automation, while a CLI workflow that performs efficiently in batch operations may be less convenient for developers who need continuous visual inspection.
The goal of benchmarking is not to declare a winner. It is to establish measurable evidence for choosing the right AI development workflow.