An AI coding agent can produce a working answer and still miss important project-specific knowledge.

This becomes obvious in SharePoint Framework (SPFx) development. A developer may ask an agent to create a web part, configure a property pane, use Microsoft Graph, or follow an organization's SharePoint development standards. The model can know the general technology but still lack the exact skills, conventions, or APIs required by the project.

A useful way to measure this gap is to evaluate an agent against a defined set of SharePoint Framework developer skills.

Instead of asking whether the agent is "good at SharePoint," break the evaluation into individual tasks and check where it succeeds, where it needs additional context, and where it produces code that requires developer correction.

What Is a Knowledge Gap in an AI Coding Agent?

A knowledge gap is the difference between what the agent needs to know to complete a task correctly and what it can produce from its available context.

For example, consider this request:

Create an SPFx web part that displays items from a SharePoint list,
supports filtering by category, and follows our existing project structure.

The agent may understand:

But it may not know:

The resulting code may look reasonable while still being wrong for the project.

That is why evaluating individual skills is more useful than judging the final answer alone.

Define a SharePoint Framework Skill Set

Start by creating a fixed set of capabilities that the agent should demonstrate.

A practical evaluation can include:

Skill

Example task

SPFx fundamentals

Create a basic web part

React integration

Build a functional React component

Property pane

Add configurable web part properties

SharePoint data

Read and update list items

Microsoft Graph

Retrieve Microsoft 365 data

Authentication

Use the project's approved authentication approach

TypeScript

Produce type-safe application code

PnP tooling

Use approved PnP patterns where required

Packaging

Prepare the solution for deployment

Testing

Add meaningful unit tests

Debugging

Diagnose a failing web part

Project conventions

Follow an existing repository's structure

The goal is not to test everything at once.

Each skill should have a small task with an expected result.

Build a Repeatable Evaluation

A good evaluation should give the agent the same information every time.

For example:

Task:
Create an SPFx React web part that reads five items
from a SharePoint list and displays the title and status.

Constraints:
- Use TypeScript.
- Follow the existing project structure.
- Do not introduce a new HTTP client.
- Use the existing SharePoint service.
- Include unit tests.

Then evaluate the output against predefined criteria.

A simple scoring model can be:

Score

Meaning

0

Cannot complete the task

1

Produces substantially incorrect output

2

Produces partially usable output

3

Produces working output with minor corrections

4

Produces correct output following project requirements

This makes the evaluation easier to compare across models, prompts, or agent configurations.

Evaluate More Than Generated Code

Generated code is only one part of an agent evaluation.

For development agents, also measure whether the agent:

  1. Understands the requirement.

  2. Identifies missing information.

  3. Uses the correct project files.

  4. Follows repository conventions.

  5. Avoids unnecessary changes.

  6. Explains assumptions.

  7. Runs or proposes appropriate tests.

  8. Correctly diagnoses errors.

  9. Handles security-sensitive operations carefully.

An agent that generates 200 lines of code but changes unrelated files should not receive a high score simply because the main component works.

Use Existing SPFx Projects as Evaluation Material

A synthetic task can measure basic knowledge, but an existing project reveals much more.

For example, give the agent a small SPFx repository containing:

src/
  webparts/
  services/
  models/
  components/
  utilities/

Then ask it to add a feature without changing the existing architecture.

This tests whether the agent can discover and follow the project's conventions.

The evaluation becomes closer to real development because the agent has to understand existing code before modifying it.

What to Check

Look for:

This is often a better test of agent usefulness than asking it to create an application from an empty directory.

Test Retrieval and Context Understanding

An AI agent may have strong general knowledge but weak repository understanding.

Create tasks where the required information already exists in the project.

For example:

Find how this project currently retrieves SharePoint list data.
Use the same service to add filtering to the Events web part.

A good agent should inspect the repository, find the existing implementation, and extend it.

A weak agent may create another API abstraction even though the project already has one.

This identifies a context gap, which is different from a pure knowledge gap.

Test SharePoint-Specific Reasoning

SharePoint Framework development contains many areas where superficially correct code can still be problematic.

Create targeted tests around:

Permissions

Ask the agent to retrieve Microsoft Graph data and specify the required permissions.

Check whether it identifies the permission requirements instead of silently assuming access exists.

SPFx Version Compatibility

Give the agent a project using a specific SPFx version and ask it to add a feature.

Check whether generated APIs and package versions are compatible with the existing project.

SharePoint API Usage

Ask the agent to retrieve list data, handle errors, and support pagination where the scenario requires it.

Property Pane Configuration

Ask for a configurable property and check whether the value flows correctly from the web part properties into the React component.

These focused tests make it easier to find exactly where an agent needs additional guidance.

Evaluate the Agent's Ability to Learn

A useful agent should improve when you provide the missing information.

Run the same task in two stages.

Stage 1

Give only the original repository and task.

Record the result.

Stage 2

Provide a skill document or additional project guidance describing the expected approach.

Run the task again.

Compare the results.

For example:

Initial result:
The agent creates a new SharePoint API service.

After skill guidance:
The agent reuses the existing SharePointDataService.

The second result demonstrates that the problem was not necessarily a fundamental inability to perform the task. The missing piece was context or instruction.

That distinction is important when deciding whether to improve the model prompt, provide an agent skill, or change the development workflow.

Use SharePoint Framework Developer Skills as a Knowledge Layer

A skill can give the agent focused instructions for a recurring development area.

For example:

Skill: SPFx List Data Access

Rules:
1. Search the repository for an existing SharePoint data service.
2. Reuse it when available.
3. Do not introduce a second HTTP abstraction.
4. Preserve existing error handling.
5. Follow the project's TypeScript types.
6. Add tests for new data-processing logic.

This is more useful than a generic instruction such as:

Write high-quality SharePoint Framework code.

The first version tells the agent what to do in the actual repository.

Measure Knowledge Gaps Over Time

Keep an evaluation matrix instead of relying on informal impressions.

SPFx Fundamentals       4/4
React Integration       4/4
Property Pane           3/4
SharePoint Data         3/4
Graph Integration       2/4
Testing                 2/4
Repository Conventions  1/4
Debugging               3/4

Now the development team can see where additional skills or documentation would provide the most value.

The matrix can also be repeated after changing the agent configuration.

For example:

                Before Skills    After Skills

SPFx Data            2                4
Graph                2                3
Testing              2                3
Project Conventions  1                4

The exact numbers are only meaningful if the same evaluation tasks and scoring criteria are used consistently.

Common Evaluation Mistakes

Testing Only Easy Tasks

Creating a simple web part does not tell you much about an agent's ability to work in a real repository.

Include maintenance, debugging, integration, and modification tasks.

Scoring Only Compilation

Code that compiles can still use the wrong architecture, permissions, API, or business logic.

Changing the Task Every Time

If every evaluation uses a different task, comparing scores becomes difficult.

Keep a stable benchmark set and version it when requirements change.

Ignoring Developer Corrections

Track how much human correction is required.

A useful metric is not simply whether the agent completed the task, but how much work remained after its output.

Giving the Agent Too Much Information

If every answer includes a complete solution, you may measure the quality of the prompt instead of the agent.

Provide only the context that a developer would reasonably have available.

Troubleshooting Poor Evaluation Results

If the agent repeatedly fails a task, first identify the type of failure.

Failure

Likely issue

Wrong API

Product knowledge gap

Correct API, wrong project usage

Repository context gap

Correct implementation, wrong version

Version-awareness gap

Good code, missing tests

Skill or instruction gap

Repeated unrelated changes

Agent workflow problem

Correct after documentation is provided

Knowledge/context gap

This prevents teams from trying to solve every problem by changing the model.

Sometimes the model already knows how to perform the task. It simply cannot see the information required to apply that knowledge.

Best Practices for Building the Evaluation

  1. Define skills as concrete development tasks.

  2. Use the same evaluation tasks when comparing configurations.

  3. Test both greenfield and existing-repository work.

  4. Score correctness, project fit, and developer effort.

  5. Include debugging and maintenance tasks.

  6. Test version-specific SPFx behavior.

  7. Check permission and security decisions.

  8. Record common failure patterns.

  9. Create targeted skills for repeated gaps.

  10. Re-run the benchmark after adding new skills.

  11. Keep expected outputs and scoring rules separate from the agent prompt.

  12. Review failed tasks manually before changing the evaluation criteria.

Advantages and Disadvantages

Advantages

Disadvantages

Makes agent quality measurable

Building good test cases takes time

Identifies specific knowledge gaps

Scores can become misleading if tasks are inconsistent

Helps decide where skills are needed

A benchmark cannot represent every real project

Supports model and prompt comparisons

Human review is still required for some tasks

Reveals repository-context problems

Skills need maintenance as projects change

Tracks improvement over time

A high score does not guarantee production safety

A Practical Evaluation Loop

A SharePoint Framework team can keep the process simple:

Define skill
    |
    v
Create small evaluation task
    |
    v
Run AI agent
    |
    v
Score result
    |
    v
Identify failure type
    |
    v
Add or improve skill guidance
    |
    v
Run the same task again
    |
    v
Compare results

The useful question is not simply, "Does this AI agent know SharePoint Framework?"

A better question is:

Which SharePoint Framework development tasks can it complete correctly, which ones require additional project knowledge, and does targeted skill guidance close those gaps?

That approach turns AI agent evaluation from a subjective demo into a repeatable engineering exercise.