AI-assisted software development has changed the economics of writing code. A developer can now describe a feature, generate an implementation, ask for tests, refactor a class, or troubleshoot an error in a fraction of the time required for a traditional workflow.
But generation speed is only one part of developer productivity.
The more important question is what happens after the first version of the code is generated.
Does the code pass tests immediately? How many changes are required before it is production-ready? How often does the developer need to correct architecture, security, performance, or maintainability problems? Does an AI agent require more rework than a developer working manually?
These questions can be studied by measuring code rework.
This article explains how to design a practical benchmark for comparing AI-generated code, agent-assisted workflows, and human-written implementations, with a focus on .NET development.
Introduction
Consider the following three development workflows.
Workflow A
Developer
|
v
Requirements
|
v
Code
|
v
Tests
Workflow B
Developer + AI Assistant
|
v
Prompt
|
v
Generated Code
|
v
Developer Review
|
v
Tests
Workflow C
AI Agent
|
v
Requirements
|
v
Plan
|
v
Code
|
v
Tests
|
v
Fix
|
v
Retest
All three workflows can eventually produce working software.
However, their development effort may be very different.
An AI system might generate 500 lines of code quickly but require several rounds of correction. A human developer might initially write 250 lines but produce a solution that needs less rework.
Therefore, measuring only time to first implementation can produce a misleading conclusion.
A better benchmark measures the complete path from requirement to an accepted implementation.
What Is Code Rework?
Code rework is the amount of additional engineering effort required after an initial implementation because the first version did not fully satisfy the requirements.
Rework can include:
A simple model is:
Total Engineering Effort
=
Initial Implementation
+
Rework
+
Validation
For AI-assisted development, the rework component is particularly important.
Why First-Pass Success Is Not Enough
Suppose two developers complete the same task.
| Metric | Developer A | Developer B |
|---|
| Initial implementation | 30 min | 12 min |
| Fixes | 5 min | 35 min |
| Testing | 10 min | 15 min |
| Total | 45 min | 62 min |
If you measure only initial implementation time, Developer B appears much faster.
If you measure the entire workflow, Developer A is faster.
This is why benchmarking AI development requires a complete workflow rather than a single generation-speed metric.
Human Workflow vs AI-Assisted Workflow
A useful benchmark should compare multiple workflows.
Human-Only Baseline
The developer receives the requirements and implements the solution without AI assistance.
This establishes the baseline.
AI-Assisted Development
The developer uses AI for selected tasks such as:
Code generation
Test generation
Debugging
Refactoring
Documentation
The developer remains responsible for implementation decisions.
Agentic Development
An AI agent receives a higher-level task and performs multiple steps autonomously:
Requirement
|
v
Planning
|
v
Code Generation
|
v
Build
|
v
Test
|
v
Diagnosis
|
v
Fix
|
v
Retest
This workflow is especially interesting because the agent itself may perform multiple rework cycles.
Define the Benchmark Task
The benchmark must start with a fixed software-development task.
For example:
Build an ASP.NET Core API for managing customer orders.
Requirements:
- Create an order
- Retrieve an order
- Update order status
- Validate input
- Persist data
- Return appropriate HTTP responses
- Add automated tests
The task should be detailed enough that every participant or workflow receives the same functional requirements.
Avoid changing requirements during the benchmark.
Otherwise, it becomes difficult to determine whether a code change was genuine rework or a response to a new requirement.
Establish Acceptance Criteria
Before running the benchmark, define what constitutes a successful implementation.
For example:
Functional
- Create order works
- Retrieve order works
- Update status works
Quality
- Tests pass
- Validation exists
- Error handling is correct
Architecture
- Controllers do not access persistence directly
- Business logic remains in the application layer
Security
- Input validation is applied
- Sensitive data is not logged
Performance
- No unnecessary database operations
The acceptance criteria should be written before implementation begins.
Measure Rework Events
A useful benchmark records every meaningful correction.
For example:
Initial Code
|
+--> Test Failure #1
|
+--> Fix
|
+--> Architecture Violation
|
+--> Refactor
|
+--> Test Failure #2
|
+--> Fix
|
+--> Accepted
This produces three rework events.
The benchmark should distinguish between trivial formatting changes and meaningful engineering changes.
Classify Rework
Not all rework has the same importance.
A practical classification is:
| Category | Example |
|---|
| Functional | Incorrect business behavior |
| Test | Failing or missing test |
| Architecture | Wrong dependency direction |
| Security | Missing authorization or unsafe input |
| Performance | Excessive database calls |
| Maintainability | Significant structural refactoring |
| Integration | Incorrect external service behavior |
| Requirement | Misinterpreted specification |
This allows teams to understand why rework occurred rather than only counting the number of changes.
Measure Rework Ratio
One useful metric is the rework ratio.
Rework Ratio =
Rework Effort / Total Engineering Effort
For example:
Initial implementation = 40 minutes
Rework = 20 minutes
Total = 60 minutes
Rework Ratio = 20 / 60
= 33.3%
A lower ratio generally indicates that less additional work was required after the initial implementation.
However, the ratio should never be treated as the only metric.
Measure Lines of Code Carefully
Lines of code can be useful but are often misunderstood.
Suppose an AI generates:
1,200 lines
and a developer writes:
450 lines
That does not mean the AI implementation produced more value.
The generated implementation may contain:
Therefore, track code volume as a secondary metric.
More useful measurements include:
Initial LOC
Final LOC
Deleted LOC
Modified LOC
Added LOC during rework
This provides a better picture of how much of the initial implementation survived.
Measure Code Churn
Code churn measures how much code changes after the initial implementation.
A simple measurement can be:
Churn =
Lines Added
+
Lines Modified
+
Lines Deleted
For example:
Initial implementation: 600 LOC
Rework:
Added: 120
Modified: 180
Deleted: 100
Churn = 400 LOC
A high churn value suggests that a substantial portion of the original implementation required revision.
Measure First-Pass Acceptance
First-pass acceptance is another useful metric.
First-Pass Acceptance Rate =
Tasks Accepted Without Rework
/
Total Tasks
Suppose:
20 benchmark tasks
6 accepted immediately
Then:
6 / 20 = 30%
This can reveal differences between workflows.
However, the benchmark must define what counts as rework.
A simple formatting change should not cause a first-pass failure.
Measure Defect Escape
A particularly important metric is the number of defects discovered after the initial validation stage.
For example:
Generated Code
|
v
Unit Tests
|
v
Integration Tests
|
v
Code Review
|
v
Security Review
If a workflow passes unit tests but repeatedly fails integration or security validation, that is valuable evidence about its quality.
The benchmark should therefore track defects by discovery stage.
Build a Rework Taxonomy
A detailed benchmark can categorize each change.
Rework
|
+-- Functional
|
+-- Architecture
|
+-- Security
|
+-- Performance
|
+-- Testing
|
+-- Maintainability
|
+-- Integration
This makes comparisons more meaningful.
For example:
Human Workflow
Architecture: 2
Functional: 5
Security: 1
AI Workflow
Architecture: 8
Functional: 4
Security: 3
The result suggests a different quality profile rather than simply saying that one workflow is "better."
Measure Developer Intervention
For AI-assisted workflows, developer intervention should be tracked.
Examples include:
Prompt corrections
Manual code edits
Rejected AI suggestions
Additional instructions
Manual debugging
Manual test creation
Architecture corrections
An agent may generate code autonomously but still require substantial human intervention.
Therefore:
AI Generation Time
is not equivalent to:
Total Human Effort
Measure Prompt Iterations
For conversational AI workflows, count meaningful prompt iterations.
Example:
Prompt 1:
Create the API.
Prompt 2:
Fix the failing validation test.
Prompt 3:
Move database access out of the controller.
Prompt 4:
Add cancellation token support.
Prompt 5:
Fix the integration test.
Five interactions may represent substantial rework.
A benchmark can report:
Average AI interaction count per task
alongside engineering time.
Measure Agent Iterations
Agentic systems provide another useful metric:
Agent Iterations
=
Plan
+
Implementation
+
Test
+
Repair
+
Retest
For example:
Task A: 2 iterations
Task B: 4 iterations
Task C: 7 iterations
A high iteration count does not automatically mean poor performance.
A complex task may naturally require more iterations.
The metric becomes useful when task complexity is controlled.
Control Task Complexity
A benchmark becomes unreliable if one workflow receives easy tasks and another receives difficult tasks.
Create task categories.
Small Tasks
Examples:
Add a DTO
Create a validation rule
Add an endpoint
Medium Tasks
Examples:
Implement a service
Add persistence
Introduce business rules
Large Tasks
Examples:
Compare workflows within the same task category.
Use Multiple Tasks
One task is not enough to evaluate a development workflow.
A better benchmark might contain:
10 small tasks
10 medium tasks
5 large tasks
The exact number depends on available resources.
Multiple tasks reduce the influence of a single unusual problem.
Use Repeated Trials
AI systems can produce different results from similar prompts.
Therefore, a single execution may not represent typical behavior.
For experimental evaluation, repeat tasks where practical and record:
Run 1
Run 2
Run 3
...
Then report distributions rather than only one result.
For example:
Median rework time
p90 rework time
Median iteration count
This is more informative than a single average.
Control the Development Environment
Keep the following consistent:
.NET version
Repository state
Dependencies
Requirements
Test suite
Build configuration
Hardware where relevant
Database state
Benchmark procedure
Otherwise, environmental differences can contaminate the results.
Capture Git History
Git provides a practical mechanism for measuring rework.
A benchmark repository can record:
Initial commit
|
v
Generated implementation
|
v
Fix commit
|
v
Refactor commit
|
v
Final accepted commit
This history can be analyzed to determine:
Number of commits
Lines changed
Files changed
Rework categories
Time between changes
A useful workflow is to create a clear baseline commit before implementation begins.
Separate Functional Rework From Cleanup
Not every post-generation change indicates a defect.
For example:
Rename variable
is different from:
Replace incorrect database transaction logic
The benchmark should distinguish cleanup from substantive rework.
Otherwise, style preferences can distort the results.
Measure Time to Accepted Code
One of the strongest overall metrics is:
Time to Accepted Implementation
This includes:
Implementation
+
Validation
+
Rework
+
Final Verification
For example:
| Workflow | Initial Time | Rework | Validation | Total |
|---|
| Human | 45 min | 15 min | 10 min | 70 min |
| AI-assisted | 20 min | 30 min | 12 min | 62 min |
| Agentic | 12 min | 42 min | 15 min | 69 min |
This example is illustrative rather than a benchmark result.
The important point is that faster generation does not necessarily mean faster completion.
Measure Quality Alongside Rework
A benchmark should never optimize only for speed.
A workflow that produces code quickly but introduces more defects may not be preferable.
Track quality metrics such as:
Test pass rate
Defect count
Architecture violations
Security findings
Performance regressions
Code review findings
A useful benchmark therefore has multiple dimensions:
Quality
^
|
|
|
+--------------> Speed
/
/
Rework
A Practical Benchmark Scorecard
A benchmark report can use a scorecard like this:
| Metric | Human | AI-Assisted | Agentic |
|---|
| Initial implementation time | Measure | Measure | Measure |
| Total completion time | Measure | Measure | Measure |
| Rework time | Measure | Measure | Measure |
| Rework ratio | Measure | Measure | Measure |
| First-pass acceptance | Measure | Measure | Measure |
| Defects | Measure | Measure | Measure |
| Architecture violations | Measure | Measure | Measure |
| Test failures | Measure | Measure | Measure |
| Developer interventions | Measure | Measure | Measure |
| Code churn | Measure | Measure | Measure |
Do not fill this table with assumed values. The benchmark should generate the measurements.
Analyze Rework Patterns
The most interesting result may not be the average rework time.
Look for patterns.
For example:
AI-assisted workflow:
High functional accuracy
Medium architecture rework
Low implementation time
Agentic workflow:
Very low initial effort
High integration rework
High iteration count
This tells you where the workflow needs improvement.
Perhaps better architecture context would reduce AI rework.
Perhaps better automated tests would reduce agent iterations.
The benchmark should help answer those questions.
Common AI Rework Patterns in .NET
Some recurring categories are worth watching closely.
Incorrect Dependency Injection
Generated services may be registered incorrectly or instantiated directly.
Incorrect Entity Framework Usage
AI-generated code may introduce inefficient queries, incorrect tracking behavior, or unnecessary database calls.
Weak Validation
Generated endpoints may validate basic input but miss business constraints.
Incorrect Async Patterns
Examples include unnecessary blocking calls or missing cancellation propagation.
Architecture Violations
Controllers may access persistence directly instead of using application services.
Over-Abstraction
AI can sometimes introduce interfaces, factories, helpers, and wrappers that provide little value.
Under-Abstraction
The opposite can also happen when generated code places too much logic in a controller or endpoint.
The benchmark should record these as distinct rework categories.
Benchmarking Human and AI Workflows Fairly
A fair comparison requires equivalent information.
If the human developer receives:
Requirements
Architecture documentation
Existing code
Tests
the AI workflow should receive equivalent relevant context.
Otherwise, the benchmark measures information asymmetry rather than development capability.
Similarly, if the AI has access to the complete repository, the human should have equivalent repository access.
Avoid Benchmark Contamination
Do not allow the same implementation to influence later trials.
For example:
Trial 1
|
v
Modified repository
should not become the starting point for Trial 2.
Each trial should start from the same baseline.
Otherwise, later runs may benefit from earlier fixes.
Track Human Review Time
For AI-assisted development, review time is part of the workflow.
Suppose AI generates a feature in ten minutes, but the developer spends twenty-five minutes reviewing and correcting it.
The true engineering effort is not ten minutes.
Track:
Generation
+
Review
+
Correction
+
Validation
This provides a much more realistic measure.
Build an Automated Measurement Harness
A benchmark harness can collect information automatically.
For example:
Benchmark Runner
|
+--> Reset Repository
|
+--> Start Timer
|
+--> Execute Workflow
|
+--> Run Build
|
+--> Run Tests
|
+--> Analyze Git Diff
|
+--> Run Architecture Checks
|
+--> Record Metrics
A simple result object might look like:
public sealed record BenchmarkResult(
string Workflow,
TimeSpan TotalTime,
TimeSpan ReworkTime,
int TestFailures,
int ArchitectureViolations,
int Commits,
int LinesAdded,
int LinesDeleted);
The goal is to make the measurement process reproducible.
Use Median and Percentiles
Development measurements are rarely normally distributed.
One unusually difficult task can significantly increase the average.
For repeated benchmark runs, report:
Median
P75
P90
For example:
Median completion time: 48 min
P90 completion time: 91 min
This provides a better understanding of variability.
Common Benchmarking Mistakes
Measuring Only Generation Speed
Generation speed does not equal completed engineering work.
Using One Task
One task can produce misleading conclusions.
Changing Requirements Midway
This makes rework difficult to classify.
Ignoring Review Time
Human validation is part of the workflow.
Counting Every Edit as Rework
Formatting and substantive engineering corrections are different.
Ignoring Quality
Fast but defective code is not necessarily productive.
Comparing Different Context Levels
One workflow should not receive substantially more information than another.
Reporting Only Averages
Distributions often provide more useful information.
Using Uncontrolled Repositories
Existing changes can contaminate later trials.
Frequently Asked Questions
Is more rework always evidence that AI performed worse?
Not necessarily. A workflow may intentionally generate a broader initial implementation and use automated validation to refine it. Rework must be evaluated alongside total effort, quality, and final output.
What is the most important metric?
For practical software engineering, time to accepted implementation is often more useful than raw generation speed because it includes validation and correction.
Should generated code be compared with human-written code line by line?
No. Code size alone is a poor measure of engineering quality. Focus on effort, correctness, maintainability, and rework.
How many benchmark tasks should be used?
There is no universal number. More varied tasks generally produce more reliable results, particularly when they cover different complexity levels.
Should AI-assisted and autonomous agents be measured separately?
Yes. A developer using AI suggestions and an autonomous coding agent represent different workflows and should not automatically be treated as the same category.
Can Git history measure rework?
Yes. Git provides useful evidence for code churn, commits, changed files, and post-implementation modifications. It should be combined with time and defect data.
Should benchmark results be treated as universal?
No. Results depend on the task, developer experience, AI system, prompts, repository, tools, and evaluation criteria. Benchmarks are most useful when their methodology is clearly documented.
Conclusion
AI-assisted software development should not be evaluated solely by how quickly code appears on the screen.
The more meaningful measurement is what happens between the first generated implementation and the final accepted solution.
A strong benchmark captures:
Initial Generation
|
v
Validation
|
v
Rework
|
v
Review
|
v
Final Acceptance
By measuring rework time, code churn, defect categories, test failures, architecture violations, developer intervention, and total time to accepted code, teams can build a much more realistic picture of AI-assisted development productivity.
The objective is not to prove that human developers or AI agents are universally better. The objective is to understand where each workflow performs well, where rework occurs, and what engineering practices can reduce that rework.
For .NET teams adopting AI coding assistants and agentic development, this type of benchmark turns a broad productivity discussion into measurable engineering evidence.