AI coding agents can solve development tasks that would previously require a developer to inspect a repository, understand the code, make changes, run tests, and troubleshoot failures.
The problem is that an agent can also repeat the same mistake.
An agent may introduce an incorrect validation rule, modify an unrelated file, use the wrong abstraction, forget a test, or fix one problem while creating another. If the same type of failure appears repeatedly, simply giving the agent more instructions is not always enough.
The better approach is to turn important mistakes into reusable feedback.
A coding agent should not be treated as a system that remembers every previous interaction perfectly. Instead, teams can build a workflow where mistakes are captured, converted into checks or repository guidance, and then tested again during future tasks.
This creates a continuous improvement loop:
Agent makes a mistake
|
v
Identify why it happened
|
v
Create a rule, test, or eval
|
v
Run the agent again
|
v
Verify the mistake is avoided
|
v
Keep the check for future tasks
The goal is not to prevent every possible mistake. The goal is to stop known mistakes from becoming recurring mistakes.
Why AI Agents Repeat Mistakes
AI coding agents work from the information available to them during a task. If an important project convention or requirement is missing from that context, the agent may choose an approach that looks reasonable but does not fit the repository.
For example, a project may already have a centralized authorization service:
public interface IAuthorizationService
{
Task<bool> CanAccessAsync(UserContext context, string resource);
}
If the agent does not understand that this service should be reused, it may create authorization logic directly inside a controller.
The resulting code may compile and even pass basic tests.
However, it creates duplicated business logic.
The mistake is not necessarily caused by the agent being incapable of writing authorization code. The problem is that the repository's architectural rule was not made clear enough or was not enforced through testing.
This distinction is important when improving an AI coding workflow.
Capture the Exact Mistake
When an agent produces a bad change, avoid writing a vague instruction such as:
Do not make mistakes in authorization.
That instruction is difficult for an agent to apply consistently.
Instead, capture the actual failure.
For example:
The agent added authorization checks directly inside the controller.
Project rule:
Authorization decisions must use IAuthorizationService.
Do not duplicate authorization rules inside controllers.
This gives the agent a specific constraint.
The next step is to decide where that constraint should live.
Turn Repeated Mistakes Into Repository Rules
Repository-level instructions are useful when the same architectural or coding rule applies to many tasks.
For example:
## Authorization
Use IAuthorizationService for authorization decisions.
Controllers should not contain duplicated authorization
business rules.
When modifying an endpoint that already uses the service,
follow the existing authorization pattern.
This is much more useful than repeating the same instruction manually in every prompt.
The same approach can be used for:
- Dependency injection
- Error handling
- Logging
- Database access
- API validation
- Testing conventions
- Naming
- Security controls
- Configuration
- External service integration
The key is to keep the rule specific enough to guide implementation.
Convert Important Mistakes Into Tests
Repository instructions explain what the agent should do.
Automated tests verify what the resulting application actually does.
Suppose an agent repeatedly forgets to validate duplicate email addresses.
A test can preserve that requirement:
[Fact]
public async Task Register_ShouldRejectDuplicateEmail()
{
var request = new RegisterRequest
{
Name = "John",
Email = "[email protected]"
};
await repository.AddAsync(new User
{
Name = "Existing User",
Email = request.Email
});
var result = await service.RegisterAsync(request);
Assert.Equal(RegisterResult.DuplicateEmail, result);
}
Now the requirement is no longer dependent on an agent instruction alone.
If the agent removes or breaks the duplicate-email behavior, the test exposes the regression.
This creates two layers of protection:
Repository guidance
+
Automated tests
=
Stronger agent feedback
Use Agent Evals for Repository-Level Mistakes
Some mistakes cannot be adequately captured by a normal unit test.
Consider an agent that repeatedly modifies unrelated files.
The application may still pass every test.
A repository-level evaluation can check the scope of the generated changes.
For example:
Task:
Add pagination to the customer endpoint.
Expected:
- Modify the customer API implementation.
- Add or update relevant tests.
- Do not modify authentication.
- Do not modify deployment configuration.
- Do not change unrelated controllers.
This tests the agent's development behavior rather than only the application's runtime behavior.
A useful evaluation can therefore combine functional requirements with repository constraints.
Build a Mistake-to-Control Workflow
Not every mistake requires the same type of solution.
| Mistake | Recommended Control |
|---|
| Wrong business behavior | Automated test |
| Missing validation | Unit or integration test |
| Wrong architecture | Repository instruction and code review |
| Unrelated file changes | Agent eval or change-scope check |
| Missing tests | Evaluation check |
| Security regression | Automated security test |
| Repeated formatting issue | Repository guidance or formatter |
| Incorrect dependency usage | Repository guidance and build checks |
| Repeated API contract issue | Integration or contract test |
| Same agent failure across tasks | Dedicated regression eval |
This prevents teams from trying to solve every problem with prompt wording.
Use a Regression Evaluation Suite
Once several recurring mistakes have been identified, group them into a small regression suite.
For example:
Customer API Agent Evals
1. Adds validation correctly
2. Reuses existing authorization service
3. Adds tests for new behavior
4. Does not modify unrelated APIs
5. Preserves existing response contracts
6. Handles database errors correctly
The agent can then be evaluated against the same set of scenarios after changes to its instructions or tools.
A conceptual workflow looks like this:
Agent Version A
|
v
Run regression suite
|
v
Record failures
|
v
Improve agent configuration
|
v
Agent Version B
|
v
Run same suite
|
v
Compare results
This gives the team a repeatable way to determine whether an improvement actually helped.
Avoid Overloading the Agent With Rules
A common mistake is responding to every agent failure by adding another instruction.
Eventually, the repository guidance can become a large collection of rules that are difficult to understand and maintain.
Instead, classify the failure first.
Ask:
Is this a code defect?
Is this a missing project rule?
Is this a missing test?
Is this a task ambiguity?
Is this an agent/tool limitation?
Then apply the appropriate control.
For example, if the agent repeatedly creates invalid SQL, a test may be more valuable than another instruction.
If the agent repeatedly ignores the existing repository pattern, repository guidance may be more appropriate.
If the agent cannot reliably complete a task because the requirement is ambiguous, improve the task definition.
Give the Agent a Clear Definition of Done
Agents perform better when completion criteria are explicit.
Instead of:
Implement customer search.
use:
Implement customer search.
Requirements:
- Search by name and email.
- Support case-insensitive matching.
- Return paginated results.
- Use the existing CustomerRepository.
- Preserve the existing response contract.
- Add unit tests for name and email searches.
- Add tests for empty results.
- Do not modify unrelated endpoints.
This gives the agent both the objective and the constraints.
It also makes evaluation easier because each requirement can become a check.
Protect Against the Same Mistake in Different Forms
An agent may stop making one exact mistake while still making the same underlying mistake differently.
For example, suppose the original problem was:
The agent duplicated authorization logic in CustomerController.
After adding a rule, the agent might move the duplicated logic into CustomerService instead.
The exact mistake changed, but the architectural problem remains.
The better requirement is:
Authorization decisions must use the existing authorization abstraction.
Do not duplicate authorization rules in another application layer.
This focuses on the underlying engineering principle rather than one specific file.
That makes the control more durable.
Use Existing Tooling Before Adding More Prompt Rules
Many recurring mistakes are better handled by deterministic tooling.
For example:
- Formatters can enforce code formatting.
- Linters can detect common code problems.
- Compilers can detect invalid types.
- Unit tests can validate business behavior.
- Integration tests can validate API interactions.
- Static analysis can detect certain security and quality problems.
- CI checks can prevent invalid changes from being merged.
An AI agent should operate inside these controls rather than replacing them.
A useful model is:
AI Agent
|
v
Repository rules
|
v
Build + static analysis
|
v
Unit + integration tests
|
v
Agent evaluation
|
v
Human review
Each layer catches different classes of problems.
Common Mistakes When Improving Agent Reliability
Adding instructions without identifying the root cause
If an agent fails, immediately adding another rule may hide the actual problem.
First determine why the failure occurred.
Using vague instructions
Rules such as "follow best practices" provide little actionable guidance.
Describe the expected behavior.
Testing only the happy path
A fix may work for normal input but fail at boundaries.
Include invalid input, missing data, duplicate data, and failure scenarios where appropriate.
Removing a difficult task after failure
A failed evaluation is useful information.
Instead of deleting the scenario, determine why the agent failed and improve either the agent or the evaluation.
Depending entirely on AI instructions
Instructions are helpful, but they are not a substitute for deterministic tests and CI controls.
Important business and security behavior should still be enforced by software tests.
Best Practices
- Keep a record of important agent failures.
- Convert recurring failures into explicit requirements.
- Use automated tests for application behavior.
- Use repository instructions for project-specific conventions.
- Use evaluations for agent-level behavior that normal tests cannot measure.
- Keep evaluation tasks realistic and representative of actual development.
- Review whether a new rule solves the underlying problem rather than one specific symptom.
- Remove obsolete rules when the architecture or workflow changes.
- Run regression evaluations after significant agent configuration changes.
- Keep human code review as the final quality control for important changes.
Advantages
A structured mistake-prevention process provides several benefits:
- Repeated failures become reusable engineering knowledge.
- Agent behavior becomes easier to improve systematically.
- Important project conventions become clearer.
- Regression scenarios are preserved.
- Developers spend less time correcting the same class of agent mistake.
- Automated tests and evaluations provide measurable feedback.
- Agent improvements can be compared instead of judged only by intuition.
The biggest benefit is that every important mistake can become part of the development feedback loop.
Disadvantages and Limitations
This approach also adds maintenance work.
Repository instructions need to remain current. Tests and evaluations need to be updated when application behavior changes. Poorly designed rules can make agents overly restrictive or encourage unnecessary code changes.
There is also a risk of creating too many controls.
If every small agent mistake becomes another repository rule, the development guidance can become difficult to maintain. Deterministic tooling should be preferred when it can enforce the requirement reliably.
The objective is not to create a giant instruction document. It is to create a small set of useful controls around the mistakes that matter.
Troubleshooting Repeated Agent Failures
When the same mistake continues after adding a rule, check these areas:
The instruction is too vague
Replace general language with a specific requirement and example.
The requirement conflicts with existing code
Inspect the repository before creating a new rule. The existing architecture may already provide the correct solution.
The test does not cover the actual failure
Reproduce the original problem and make sure the test fails when the mistake is reintroduced.
The agent lacks required context
The agent may need access to the relevant configuration, service, documentation, or existing implementation.
The problem is nondeterministic
If different runs produce different results, focus the evaluation on observable outcomes and make the test environment as stable as possible.
Summary
AI coding agents will make mistakes. The important engineering question is what happens after those mistakes occur.
A useful development workflow does not simply correct the generated code and move on. It captures important failures, identifies their underlying cause, and converts them into reusable controls.
Repository instructions can communicate architectural rules. Automated tests can protect application behavior. Static analysis and CI can enforce deterministic requirements. Agent evaluations can test whether the agent performs realistic development tasks correctly.
Together, these controls create a feedback loop that helps prevent known mistakes from returning.
The most effective approach is to treat repeated agent failures as engineering signals. When a mistake matters, reproduce it, understand why it happened, add the right control, and run the same scenario again. Over time, the AI coding workflow becomes more predictable without relying on increasingly complicated prompts.
Join the conversation! Your thoughts help the community grow.