AI-assisted software development has changed the economics of writing code. A developer can now describe a feature, generate an implementation, ask for tests, refactor a class, or troubleshoot an error in a fraction of the time required for a traditional workflow.
But generation speed is only one part of developer productivity.
The more important question is what happens after the first version of the code is generated.
Does the code pass tests immediately? How many changes are required before it is production-ready? How often does the developer need to correct architecture, security, performance, or maintainability problems? Does an AI agent require more rework than a developer working manually?
These questions can be studied by measuring code rework.
This article explains how to design a practical benchmark for comparing AI-generated code, agent-assisted workflows, and human-written implementations, with a focus on .NET development.
Introduction
Consider the following three development workflows.
Workflow A
Developer
|
v
Requirements
|
v
Code
|
v
Tests
Workflow B
Developer + AI Assistant
|
v
Prompt
|
v
Generated Code
|
v
Developer Review
|
v
Tests
Workflow C
AI Agent
|
v
Requirements
|
v
Plan
|
v
Code
|
v
Tests
|
v
Fix
|
v
Retest
All three workflows can eventually produce working software.
However, their development effort may be very different.
An AI system might generate 500 lines of code quickly but require several rounds of correction. A human developer might initially write 250 lines but produce a solution that needs less rework.
Therefore, measuring only time to first implementation can produce a misleading conclusion.
A better benchmark measures the complete path from requirement to an accepted implementation.
What Is Code Rework?
Code rework is the amount of additional engineering effort required after an initial implementation because the first version did not fully satisfy the requirements.
Rework can include:
Bug fixes
Test failures
Architecture corrections
Security fixes
Performance improvements
Refactoring
Requirement corrections
API contract changes
Dependency changes
Removing unnecessary generated code
Rewriting incorrect implementations
A simple model is:
Total Engineering Effort
=
Initial Implementation
+
Rework
+
Validation
For AI-assisted development, the rework component is particularly important.
Why First-Pass Success Is Not Enough
Suppose two developers complete the same task.
| Metric | Developer A | Developer B |
|---|---|---|
| Initial implementation | 30 min | 12 min |
| Fixes | 5 min | 35 min |
| Testing | 10 min | 15 min |
| Total | 45 min | 62 min |
If you measure only initial implementation time, Developer B appears much faster.
If you measure the entire workflow, Developer A is faster.
This is why benchmarking AI development requires a complete workflow rather than a single generation-speed metric.
Human Workflow vs AI-Assisted Workflow
A useful benchmark should compare multiple workflows.
Human-Only Baseline
The developer receives the requirements and implements the solution without AI assistance.
This establishes the baseline.
AI-Assisted Development
The developer uses AI for selected tasks such as:
Code generation
Test generation
Debugging
Refactoring
Documentation
The developer remains responsible for implementation decisions.
Agentic Development
An AI agent receives a higher-level task and performs multiple steps autonomously:
Requirement
|
v
Planning
|
v
Code Generation
|
v
Build
|
v
Test
|
v
Diagnosis
|
v
Fix
|
v
Retest
This workflow is especially interesting because the agent itself may perform multiple rework cycles.
Define the Benchmark Task
The benchmark must start with a fixed software-development task.
For example:
Build an ASP.NET Core API for managing customer orders.
Requirements:
- Create an order
- Retrieve an order
- Update order status
- Validate input
- Persist data
- Return appropriate HTTP responses
- Add automated tests
The task should be detailed enough that every participant or workflow receives the same functional requirements.
Avoid changing requirements during the benchmark.
Otherwise, it becomes difficult to determine whether a code change was genuine rework or a response to a new requirement.
Establish Acceptance Criteria
Before running the benchmark, define what constitutes a successful implementation.
For example:
Functional
- Create order works
- Retrieve order works
- Update status works
Quality
- Tests pass
- Validation exists
- Error handling is correct
Architecture
- Controllers do not access persistence directly
- Business logic remains in the application layer
Security
- Input validation is applied
- Sensitive data is not logged
Performance
- No unnecessary database operations
The acceptance criteria should be written before implementation begins.
Measure Rework Events
A useful benchmark records every meaningful correction.
For example:
Initial Code
|
+--> Test Failure #1
|
+--> Fix
|
+--> Architecture Violation
|
+--> Refactor
|
+--> Test Failure #2
|
+--> Fix
|
+--> Accepted
This produces three rework events.
The benchmark should distinguish between trivial formatting changes and meaningful engineering changes.
Classify Rework
Not all rework has the same importance.
A practical classification is:
| Category | Example |
|---|---|
| Functional | Incorrect business behavior |
| Test | Failing or missing test |
| Architecture | Wrong dependency direction |
| Security | Missing authorization or unsafe input |
| Performance | Excessive database calls |
| Maintainability | Significant structural refactoring |
| Integration | Incorrect external service behavior |
| Requirement | Misinterpreted specification |
This allows teams to understand why rework occurred rather than only counting the number of changes.
Measure Rework Ratio
One useful metric is the rework ratio.
Rework Ratio =
Rework Effort / Total Engineering Effort
For example:
Initial implementation = 40 minutes
Rework = 20 minutes
Total = 60 minutes
Rework Ratio = 20 / 60
= 33.3%
A lower ratio generally indicates that less additional work was required after the initial implementation.
However, the ratio should never be treated as the only metric.
Measure Lines of Code Carefully
Lines of code can be useful but are often misunderstood.
Suppose an AI generates:
1,200 lines
and a developer writes:
450 lines
That does not mean the AI implementation produced more value.
The generated implementation may contain:
Duplicate abstractions
Unnecessary helper classes
Excessive comments
Redundant validation
Boilerplate
Unused methods
Therefore, track code volume as a secondary metric.
More useful measurements include:
Initial LOC
Final LOC
Deleted LOC
Modified LOC
Added LOC during rework
This provides a better picture of how much of the initial implementation survived.
Measure Code Churn
Code churn measures how much code changes after the initial implementation.
A simple measurement can be:
Churn =
Lines Added
+
Lines Modified
+
Lines Deleted
For example:
Initial implementation: 600 LOC
Rework:
Added: 120
Modified: 180
Deleted: 100
Churn = 400 LOC
A high churn value suggests that a substantial portion of the original implementation required revision.
Measure First-Pass Acceptance
First-pass acceptance is another useful metric.
First-Pass Acceptance Rate =
Tasks Accepted Without Rework
/
Total Tasks
Suppose:
20 benchmark tasks
6 accepted immediately
Then:
6 / 20 = 30%
This can reveal differences between workflows.
However, the benchmark must define what counts as rework.
A simple formatting change should not cause a first-pass failure.
Measure Defect Escape
A particularly important metric is the number of defects discovered after the initial validation stage.
For example:
Generated Code
|
v
Unit Tests
|
v
Integration Tests
|
v
Code Review
|
v
Security Review
If a workflow passes unit tests but repeatedly fails integration or security validation, that is valuable evidence about its quality.
The benchmark should therefore track defects by discovery stage.
Build a Rework Taxonomy
A detailed benchmark can categorize each change.
Rework
|
+-- Functional
|
+-- Architecture
|
+-- Security
|
+-- Performance
|
+-- Testing
|
+-- Maintainability
|
+-- Integration
This makes comparisons more meaningful.
For example:
Human Workflow
Architecture: 2
Functional: 5
Security: 1
AI Workflow
Architecture: 8
Functional: 4
Security: 3
The result suggests a different quality profile rather than simply saying that one workflow is "better."
Measure Developer Intervention
For AI-assisted workflows, developer intervention should be tracked.
Examples include:
Prompt corrections
Manual code edits
Rejected AI suggestions
Additional instructions
Manual debugging
Manual test creation
Architecture corrections
An agent may generate code autonomously but still require substantial human intervention.
Therefore:
AI Generation Time
is not equivalent to:
Total Human Effort
Measure Prompt Iterations
For conversational AI workflows, count meaningful prompt iterations.
Example:
Prompt 1:
Create the API.
Prompt 2:
Fix the failing validation test.
Prompt 3:
Move database access out of the controller.
Prompt 4:
Add cancellation token support.
Prompt 5:
Fix the integration test.
Five interactions may represent substantial rework.
A benchmark can report:
Average AI interaction count per task
alongside engineering time.
Measure Agent Iterations
Agentic systems provide another useful metric:
Agent Iterations
=
Plan
+
Implementation
+
Test
+
Repair
+
Retest
For example:
Task A: 2 iterations
Task B: 4 iterations
Task C: 7 iterations
A high iteration count does not automatically mean poor performance.
A complex task may naturally require more iterations.
The metric becomes useful when task complexity is controlled.
Join the conversation! Your thoughts help the community grow.