Evaluating AI Agent Knowledge Gaps with SharePoint Framework Developer Skills

An AI coding agent can produce a working answer and still miss important project-specific knowledge.

This becomes obvious in SharePoint Framework (SPFx) development. A developer may ask an agent to create a web part, configure a property pane, use Microsoft Graph, or follow an organization's SharePoint development standards. The model can know the general technology but still lack the exact skills, conventions, or APIs required by the project.

A useful way to measure this gap is to evaluate an agent against a defined set of SharePoint Framework developer skills.

Instead of asking whether the agent is "good at SharePoint," break the evaluation into individual tasks and check where it succeeds, where it needs additional context, and where it produces code that requires developer correction.

What Is a Knowledge Gap in an AI Coding Agent?

A knowledge gap is the difference between what the agent needs to know to complete a task correctly and what it can produce from its available context.

For example, consider this request:

Create an SPFx web part that displays items from a SharePoint list,
supports filtering by category, and follows our existing project structure.

The agent may understand:

SPFx web parts
React components
SharePoint REST APIs
TypeScript
Fluent UI

But it may not know:

The organization's preferred project structure
Existing reusable services
Internal coding standards
Which SPFx version the project uses
How authentication is handled
Which Microsoft Graph permissions are already approved
Which internal skills or tools are available to the agent

The resulting code may look reasonable while still being wrong for the project.

That is why evaluating individual skills is more useful than judging the final answer alone.

Define a SharePoint Framework Skill Set

Start by creating a fixed set of capabilities that the agent should demonstrate.

A practical evaluation can include:

Skill	Example task
SPFx fundamentals	Create a basic web part
React integration	Build a functional React component
Property pane	Add configurable web part properties
SharePoint data	Read and update list items
Microsoft Graph	Retrieve Microsoft 365 data
Authentication	Use the project's approved authentication approach
TypeScript	Produce type-safe application code
PnP tooling	Use approved PnP patterns where required
Packaging	Prepare the solution for deployment
Testing	Add meaningful unit tests
Debugging	Diagnose a failing web part
Project conventions	Follow an existing repository's structure

The goal is not to test everything at once.

Each skill should have a small task with an expected result.

Build a Repeatable Evaluation

A good evaluation should give the agent the same information every time.

For example:

Task:
Create an SPFx React web part that reads five items
from a SharePoint list and displays the title and status.

Constraints:
- Use TypeScript.
- Follow the existing project structure.
- Do not introduce a new HTTP client.
- Use the existing SharePoint service.
- Include unit tests.

Then evaluate the output against predefined criteria.

A simple scoring model can be:

Score	Meaning
0	Cannot complete the task
1	Produces substantially incorrect output
2	Produces partially usable output
3	Produces working output with minor corrections
4	Produces correct output following project requirements

This makes the evaluation easier to compare across models, prompts, or agent configurations.

Evaluate More Than Generated Code

Generated code is only one part of an agent evaluation.

For development agents, also measure whether the agent:

Understands the requirement.
Identifies missing information.
Uses the correct project files.
Follows repository conventions.
Avoids unnecessary changes.
Explains assumptions.
Runs or proposes appropriate tests.
Correctly diagnoses errors.
Handles security-sensitive operations carefully.

An agent that generates 200 lines of code but changes unrelated files should not receive a high score simply because the main component works.

Use Existing SPFx Projects as Evaluation Material

A synthetic task can measure basic knowledge, but an existing project reveals much more.

For example, give the agent a small SPFx repository containing:

src/
  webparts/
  services/
  models/
  components/
  utilities/

Then ask it to add a feature without changing the existing architecture.

This tests whether the agent can discover and follow the project's conventions.

The evaluation becomes closer to real development because the agent has to understand existing code before modifying it.

What to Check

Look for:

Reuse of existing services
Correct imports
Existing type definitions being reused
Consistent component patterns
Appropriate error handling
No unnecessary dependency additions
Minimal file changes
Correct SPFx APIs

This is often a better test of agent usefulness than asking it to create an application from an empty directory.

Test Retrieval and Context Understanding

An AI agent may have strong general knowledge but weak repository understanding.

Create tasks where the required information already exists in the project.

For example:

Find how this project currently retrieves SharePoint list data.
Use the same service to add filtering to the Events web part.

A good agent should inspect the repository, find the existing implementation, and extend it.

A weak agent may create another API abstraction even though the project already has one.

This identifies a context gap, which is different from a pure knowledge gap.

Test SharePoint-Specific Reasoning

SharePoint Framework development contains many areas where superficially correct code can still be problematic.

Create targeted tests around:

Permissions

Ask the agent to retrieve Microsoft Graph data and specify the required permissions.

Check whether it identifies the permission requirements instead of silently assuming access exists.

SPFx Version Compatibility

Give the agent a project using a specific SPFx version and ask it to add a feature.

Check whether generated APIs and package versions are compatible with the existing project.

SharePoint API Usage

Ask the agent to retrieve list data, handle errors, and support pagination where the scenario requires it.

Property Pane Configuration

Ask for a configurable property and check whether the value flows correctly from the web part properties into the React component.

These focused tests make it easier to find exactly where an agent needs additional guidance.

Evaluate the Agent's Ability to Learn

A useful agent should improve when you provide the missing information.

Run the same task in two stages.

Stage 1

Give only the original repository and task.

Record the result.

Stage 2

Provide a skill document or additional project guidance describing the expected approach.

Run the task again.

Compare the results.

For example:

Initial result:
The agent creates a new SharePoint API service.

After skill guidance:
The agent reuses the existing SharePointDataService.

The second result demonstrates that the problem was not necessarily a fundamental inability to perform the task. The missing piece was context or instruction.

That distinction is important when deciding whether to improve the model prompt, provide an agent skill, or change the development workflow.

Use SharePoint Framework Developer Skills as a Knowledge Layer

A skill can give the agent focused instructions for a recurring development area.

For example:

Skill: SPFx List Data Access

Rules:
1. Search the repository for an existing SharePoint data service.
2. Reuse it when available.
3. Do not introduce a second HTTP abstraction.
4. Preserve existing error handling.
5. Follow the project's TypeScript types.
6. Add tests for new data-processing logic.

This is more useful than a generic instruction such as:

Write high-quality SharePoint Framework code.

The first version tells the agent what to do in the actual repository.

Measure Knowledge Gaps Over Time

Keep an evaluation matrix instead of relying on informal impressions.

SPFx Fundamentals       4/4
React Integration       4/4
Property Pane           3/4
SharePoint Data         3/4
Graph Integration       2/4
Testing                 2/4
Repository Conventions  1/4
Debugging               3/4

Now the development team can see where additional skills or documentation would provide the most value.

The matrix can also be repeated after changing the agent configuration.

For example:

                Before Skills    After Skills

SPFx Data            2                4
Graph                2                3
Testing              2                3
Project Conventions  1                4

The exact numbers are only meaningful if the same evaluation tasks and scoring criteria are used consistently.

Common Evaluation Mistakes

Testing Only Easy Tasks

Creating a simple web part does not tell you much about an agent's ability to work in a real repository.

Include maintenance, debugging, integration, and modification tasks.

Scoring Only Compilation

Code that compiles can still use the wrong architecture, permissions, API, or business logic.

Changing the Task Every Time

If every evaluation uses a different task, comparing scores becomes difficult.

Keep a stable benchmark set and version it when requirements change.

Ignoring Developer Corrections

Track how much human correction is required.

A useful metric is not simply whether the agent completed the task, but how much work remained after its output.

Giving the Agent Too Much Information

If every answer includes a complete solution, you may measure the quality of the prompt instead of the agent.

Provide only the context that a developer would reasonably have available.

Troubleshooting Poor Evaluation Results

If the agent repeatedly fails a task, first identify the type of failure.

Failure	Likely issue
Wrong API	Product knowledge gap
Correct API, wrong project usage	Repository context gap
Correct implementation, wrong version	Version-awareness gap
Good code, missing tests	Skill or instruction gap
Repeated unrelated changes	Agent workflow problem
Correct after documentation is provided	Knowledge/context gap

This prevents teams from trying to solve every problem by changing the model.

Sometimes the model already knows how to perform the task. It simply cannot see the information required to apply that knowledge.

Best Practices for Building the Evaluation

Define skills as concrete development tasks.
Use the same evaluation tasks when comparing configurations.
Test both greenfield and existing-repository work.
Score correctness, project fit, and developer effort.
Include debugging and maintenance tasks.
Test version-specific SPFx behavior.
Check permission and security decisions.
Record common failure patterns.
Create targeted skills for repeated gaps.
Re-run the benchmark after adding new skills.
Keep expected outputs and scoring rules separate from the agent prompt.
Review failed tasks manually before changing the evaluation criteria.

Advantages and Disadvantages

Advantages	Disadvantages
Makes agent quality measurable	Building good test cases takes time
Identifies specific knowledge gaps	Scores can become misleading if tasks are inconsistent
Helps decide where skills are needed	A benchmark cannot represent every real project
Supports model and prompt comparisons	Human review is still required for some tasks
Reveals repository-context problems	Skills need maintenance as projects change
Tracks improvement over time	A high score does not guarantee production safety

A Practical Evaluation Loop

A SharePoint Framework team can keep the process simple:

Define skill
    |
    v
Create small evaluation task
    |
    v
Run AI agent
    |
    v
Score result
    |
    v
Identify failure type
    |
    v
Add or improve skill guidance
    |
    v
Run the same task again
    |
    v
Compare results

The useful question is not simply, "Does this AI agent know SharePoint Framework?"

A better question is:

Which SharePoint Framework development tasks can it complete correctly, which ones require additional project knowledge, and does targeted skill guidance close those gaps?

That approach turns AI agent evaluation from a subjective demo into a repeatable engineering exercise.