Foundry Local on Azure Local: Running Private Model Evaluation Without Sending Data to the Cloud

Running an AI model locally is only half the job.

Once a model is deployed, you still need to answer a much more practical question:

Is the model actually giving good answers for our data and our application?

For a normal cloud-based AI service, teams often send test prompts and datasets to an external evaluation service. That can be difficult when the data contains customer information, internal documents, source code, financial records, or other sensitive content.

Foundry Local on Azure Local takes a different approach.

The model can run on your Azure Local infrastructure, and the evaluation can also run inside the same environment. You can upload a test dataset to the cluster, execute evaluation jobs against a deployed model, and download structured results without sending the evaluation data outside the environment.

This became available with the 2607 release of Foundry Local on Azure Local.

The evaluation system supports two broad types of evaluation:

NLP-based metrics such as F1, BLEU, ROUGE, GLEU, and METEOR.
Quality evaluation using another deployed model as a judge for dimensions such as coherence, fluency, relevance, similarity, and response completeness.

That makes local evaluation useful not only for disconnected environments, but also for organizations that want sensitive test data to remain inside their own infrastructure.

Why Private Model Evaluation Matters

Suppose a company wants to evaluate an AI assistant using this dataset:

Customer question
Expected answer

The questions might contain:

Customer names
Account information
Internal policies
Support conversations
Product details
Private documentation

Sending that dataset to an external evaluation platform may not be acceptable.

A local evaluation workflow looks different:

Private Test Dataset
        |
        v
Azure Local Cluster
        |
        +----> Model Under Test
        |
        +----> Evaluation Job
        |
        v
Local Evaluation Results

The data stays inside the environment.

For disconnected deployments, Microsoft explicitly states that evaluation runs entirely on the cluster and no evaluation data leaves the environment.

What Is Model Evaluation?

Model evaluation is the process of measuring how well a model performs against a known test set.

For example:

Question:
What is our refund policy?

Expected:
Customers can request a refund within 30 days.

Model:
Customers can request a refund within 30 days of purchase.

The response looks good.

Now consider:

Expected:
Refunds are available within 30 days.

Model:
Refunds are available within 90 days.

The second answer is clearly problematic.

An evaluation system turns these kinds of observations into measurable results.

Instead of manually reading hundreds of responses, you can run a repeatable evaluation.

How Foundry Local Evaluation Works

The overall workflow is straightforward:

1. Deploy model
       |
       v
2. Prepare evaluation dataset
       |
       v
3. Upload dataset
       |
       v
4. Create evaluation
       |
       v
5. Run evaluators
       |
       v
6. Monitor evaluation
       |
       v
7. Download results

The evaluation APIs are exposed through the Foundry inference platform.

The Datasets API manages uploaded evaluation datasets, while the Evaluations API manages evaluation runs and results.

The Evaluation Dataset

The dataset is the foundation of the evaluation.

For the current API, the dataset must contain:

query
ground_truth

For JSONL, a simple dataset can look like:

{"query":"What atoms make up water?","ground_truth":"Hydrogen and oxygen"}
{"query":"What is the capital of France?","ground_truth":"Paris"}
{"query":"What is 2 + 2?","ground_truth":"4"}

The same type of information can be stored in CSV:

query,ground_truth
"What atoms make up water?","Hydrogen and oxygen"
"What is the capital of France?","Paris"
"What is 2 + 2?","4"

The current Datasets API supports jsonl and csv formats.

The documented maximum dataset size is 1 MB, and the file must contain the query and ground_truth columns.

Why Ground Truth Is Important

A model response is difficult to score if there is nothing to compare it against.

Consider:

Query:
What is the return period?

Model:
You can return the product within 30 days.

Is that correct?

You need some expected answer or another evaluation method to determine that.

The ground_truth value gives the evaluation system a reference.

For example:

query:
What is the return period?

ground_truth:
30 days

Now an evaluator has something to compare against.

Not Every Evaluation Needs Exact Matching

Exact text comparison is often too strict for generative AI.

Suppose the expected answer is:

The return period is 30 days.

The model might respond:

Customers have 30 days to return the product.

The wording is different, but the meaning is similar.

This is why Foundry Local provides multiple evaluation metrics.

NLP Evaluators

NLP evaluators compare generated responses against the expected answer using established text-matching metrics.

The current evaluation API supports:

f1_score
bleu
gleu
rouge
meteor

These metrics are useful when the expected answer provides a meaningful reference.

F1 Score

F1 combines precision and recall.

It is useful when you care about how much of the expected information was captured without adding too much unrelated content.

For example:

Expected:
Azure Kubernetes Service

Model:
Azure Kubernetes Service and Azure Functions

Depending on how the evaluator tokenizes the text, extra content can affect the score.

BLEU

BLEU was originally designed for machine translation evaluation.

It measures overlap between generated text and reference text.

It can still be useful when generated output is expected to resemble a known answer, although it should not be treated as a complete measure of semantic quality.

ROUGE

ROUGE is commonly used for text generation and summarization evaluation.

It measures overlap between generated and reference text.

For summarization workloads, it can provide a useful signal when combined with other measurements.

GLEU

GLEU is another text-overlap metric.

It can be useful for evaluating generated text where the relationship between the prediction and reference is important.

METEOR

METEOR provides another way to compare generated output with reference text, considering more than simple exact token overlap.

The important point is that no single metric tells the complete story.

Quality Evaluators

Text overlap is not enough for many AI applications.

Consider this response:

Question:
Explain how Kubernetes service accounts work.

Expected:
A detailed explanation of service accounts,
tokens, identity, and authorization.

A response can have relatively low word overlap and still be a good answer.

This is where quality evaluators are useful.

Foundry Local supports quality dimensions such as:

coherence
fluency
relevance
similarity
response_completeness

A second deployed model acts as the judge.

The architecture becomes:

                 Test Dataset
                     |
                     v
              Model Under Test
                     |
                     v
              Generated Answers
                     |
                     v
                Judge Model
                     |
                     v
                Quality Scores

The judge model itself runs in the local environment.

That is important when the test data cannot leave the cluster.

Model Under Test vs Judge Model

These two models have different jobs.

Model Under Test

This is the model you want to evaluate.

For example:

phi-4-mini

Judge Model

This model evaluates the generated responses.

For example:

gpt-oss-20b

The judge model is only required when using quality evaluators.

You can therefore have:

Model Under Test
       |
       v
Generated Response
       |
       v
Judge Model
       |
       v
Quality Score

The judge should ideally be capable enough to evaluate the type of responses being produced.

Choosing a Judge Model

The judge model matters.

If the judge is too weak, its evaluation may be unreliable.

For example:

Small Model
     |
     v
Evaluates complex reasoning
     |
     v
Possibly weak evaluation

A stronger judge can provide a more useful quality signal.

However, using a larger judge also consumes more compute.

In a local environment, you need to balance:

Evaluation quality
+
GPU/CPU capacity
+
Evaluation time

If you already have a capable local model available, it can be used as the judge rather than sending the test data to an external service.

Uploading the Dataset

The current Datasets API uses:

POST /api/v1/datasets

The request is multipart form data containing:

name
format
file

For example:

name = support-evaluation
format = jsonl
file = support-test.jsonl

The dataset name must follow the documented naming requirements.

A successful upload creates a dataset resource that can then be referenced by an evaluation job.

Example Dataset Upload

Using PowerShell, the current API can be called like this:

$datasetName = "support-evaluation"
$filePath = ".\support-test.jsonl"
$format = "jsonl"

$fileBytes = [System.IO.File]::ReadAllBytes($filePath)
$fileName = Split-Path $filePath -Leaf
$boundary = [System.Guid]::NewGuid().ToString()

$body = @(
    "--$boundary",
    "Content-Disposition: form-data; name=`"name`"",
    "",
    $datasetName,
    "--$boundary",
    "Content-Disposition: form-data; name=`"format`"",
    "",
    $format,
    "--$boundary",
    "Content-Disposition: form-data; name=`"file`"; filename=`"$fileName`"",
    "Content-Type: application/octet-stream",
    "",
    [System.Text.Encoding]::UTF8.GetString($fileBytes),
    "--$boundary--"
) -join "`r`n"

Invoke-RestMethod `
    -Uri "$baseUrl/api/v1/datasets" `
    -Method POST `
    -Headers $headers `
    -ContentType "multipart/form-data; boundary=$boundary" `
    -Body $body

After uploading, check the dataset:

Invoke-RestMethod `
    -Uri "$baseUrl/api/v1/datasets/$datasetName" `
    -Headers $headers

You want the dataset to reach:

phase: Ready

The response also includes the number of rows and dataset size.

Creating an NLP Evaluation

Once the dataset is ready, create an evaluation.

For example:

{
  "name": "support-nlp-evaluation",
  "datasetRef": "support-evaluation",
  "modelRef": "phi4-cpu-demo",
  "evaluators": [
    "f1_score",
    "bleu",
    "rouge"
  ]
}

The request goes to:

POST /api/v1/evaluations

The important fields are:

name
datasetRef
modelRef
evaluators

For NLP evaluation, you do not need a judge model.

Creating a Quality Evaluation

For quality evaluation, add a judge model.

For example:

{
  "name": "support-quality-evaluation",
  "datasetRef": "support-evaluation",
  "modelRef": "phi4-cpu-demo",
  "judgeModelRef": "gpt-oss-20b",
  "evaluators": [
    "coherence",
    "fluency",
    "relevance",
    "similarity"
  ]
}

Now the flow is:

Dataset
   |
   v
Model Under Test
   |
   v
Generated Response
   |
   v
Judge Model
   |
   v
Quality Metrics

The judgeModelRef is required when at least one quality evaluator is included.

Monitoring an Evaluation

Evaluation jobs are asynchronous.

When you submit one, the initial response can look like:

{
  "name": "support-quality-evaluation",
  "phase": "Pending"
}

You then query:

GET /api/v1/evaluations/support-quality-evaluation

The evaluation progresses through states such as:

Pending
   ↓
Running
   ↓
Succeeded

or:

Pending
   ↓
Running
   ↓
Failed

The API exposes fields such as:

phase
message
metrics
startTime
completionTime
resultsAvailable

This makes it possible to build evaluation workflows into CI/CD or internal tooling.

Reading the Aggregate Metrics

After a successful evaluation, you can receive metrics such as:

{
  "metrics": {
    "coherence": 4.2,
    "fluency": 3.8,
    "f1_score": 0.85
  }
}

These numbers give you an aggregate view.

But do not stop there.

An aggregate score can hide individual failures.

For example:

Average F1: 0.91

sounds good.

But perhaps:

Question 1 -> 0.99
Question 2 -> 0.97
Question 3 -> 0.95
Question 4 -> 0.98
Question 5 -> 0.22

The average hides the problematic case.

This is why per-row results are important.

Downloading Detailed Results

After the evaluation succeeds:

GET /api/v1/evaluations/support-quality-evaluation/results

You can request JSON:

/results

or CSV:

/results?format=csv

The JSON results include aggregate information and per-row evaluation results.

The CSV option is useful when you want to inspect the results in Excel or another analysis tool.

Why Per-Row Results Matter

Suppose you are evaluating a customer support assistant.

Your aggregate result is:

Relevance: 4.4
Fluency: 4.6

That looks good.

But one customer's response might be:

Question:
How do I cancel my subscription?

Answer:
Your subscription cannot be cancelled.

while your actual policy says cancellation is supported.

A high average score does not make that answer acceptable.

For production AI, individual failures often matter more than a single average.

Compare Two Model Versions

One of the most useful applications of local evaluation is comparing model versions.

For example:

Model A
Version 1

versus:

Model B
Version 2

Use the same dataset:

Evaluation Dataset
        |
        +----> Model A
        |
        +----> Model B

Then compare:

Metric	Model A	Model B
F1	0.81	0.86
ROUGE	0.74	0.79
Relevance	3.9	4.3
Fluency	4.1	4.4

The exact numbers above are only an example.

Do not treat them as benchmark results.

The point is that the evaluation framework gives you a repeatable way to generate your own numbers.

Use the Same Dataset for Fair Comparison

If you change both the model and the test dataset, the comparison becomes difficult.

A better process is:

Dataset v1
    |
    +---- Model A
    |
    +---- Model B
    |
    +---- Model C

Keep the dataset stable while comparing model versions.

If you need to change the dataset, create a new dataset version and record that change.

Evaluation Is Not the Same as Benchmarking

A benchmark asks:

How does this model perform on a standardized task?

Application evaluation asks:

How does this model perform on our actual workload?

The second question is often more useful for production decisions.

For example, a model may perform well on a general benchmark but struggle with:

Internal terminology
Company policies
Specific document formats
Domain-specific questions
Structured output requirements

Your own evaluation dataset can expose those problems.

Build a Representative Dataset

Do not create a dataset containing only easy questions.

For a support assistant, include:

Simple questions
Complex questions
Ambiguous questions
Edge cases
Policy questions
Out-of-scope questions
Common mistakes
Long prompts
Short prompts

A realistic dataset gives you a much better picture of production behavior.

Separate Dataset Categories

You can also organize tests into groups.

For example:

support-basic.jsonl
support-policy.jsonl
support-edge-cases.jsonl
support-security.jsonl

Then run separate evaluations.

This can tell you more than one large average.

You might discover:

General questions
Excellent

Policy questions
Good

Security questions
Needs improvement

That is actionable.

Evaluation in a Disconnected Environment

Disconnected environments have stricter requirements.

There may be no internet connection at all.

Foundry Local supports evaluation in this setup.

The documented disconnected workflow still follows the same basic process:

Prepare dataset
      ↓
Upload dataset
      ↓
Run evaluation
      ↓
Monitor
      ↓
Download results

The difference is that the required model artifacts and infrastructure are already available locally.

The evaluation data remains inside the disconnected environment.

Why This Matters for Regulated Data

Consider an organization with:

Medical records
Financial records
Government documents
Private customer data
Internal source code

The team may want to evaluate an AI model using real-world examples.

Sending those examples to an external evaluation system may violate internal policies or regulatory requirements.

A local workflow allows:

Sensitive Dataset
       |
       v
Private Cluster
       |
       +-- Model
       |
       +-- Evaluator
       |
       v
Private Results

The evaluation stays close to the data.

That does not automatically make the system compliant with every regulation, but it gives organizations a stronger infrastructure boundary for keeping evaluation data local.

Local Does Not Mean Automatically Secure

This distinction matters.

Keeping evaluation data inside your cluster is useful, but you still need to secure the cluster.

Consider:

Authentication
Authorization
Encryption
RBAC
Network policies
Storage security
Audit logging
Access controls

A private network is not a replacement for security controls.

If the evaluation dataset contains sensitive information, restrict who can:

Upload datasets
Start evaluations
Download results
Delete datasets
Access model endpoints

Protect Evaluation Results

Evaluation results can contain the original query and generated response.

For example:

{
  "query": "Customer's private question",
  "response": "Model-generated answer"
}

That means the results themselves can be sensitive.

Do not assume that because the model is local, the results are safe to distribute everywhere.

Apply the same access controls to:

Dataset
Evaluation job
Raw results
CSV exports
JSON exports

Judge Models Can Also See the Data

When using a quality evaluator, the judge model processes the generated responses.

That means:

Test dataset
     |
     v
Model Under Test
     |
     v
Generated response
     |
     v
Judge Model

Both models are therefore part of the sensitive-data boundary.

Make sure the judge model is also deployed inside the intended environment.

Do not accidentally configure an external judge service when your requirement is that evaluation data remains local.

Evaluation With a Judge Model Is Not Perfect

A model judging another model is useful, but it is not objective truth.

Consider:

Model A
  |
  v
Judge
  |
  v
Score

The judge has its own biases and limitations.

For important workloads, combine:

NLP metrics
+
Judge scores
+
Human review
+
Application-specific tests

This gives you a stronger evaluation process.

Human Review Still Matters

Suppose an AI assistant generates a response that is technically correct but difficult for users to understand.

A text-overlap metric may not catch the problem.

A judge model may give it a good score.

A human reviewer may immediately notice:

Too much jargon
Missing context
Poor explanation
Unhelpful structure
Incorrect assumption

For high-impact applications, keep humans involved in the evaluation process.

Build a Regression Dataset

Once you find a production failure, add it to your test dataset.

For example:

Production bug
     |
     v
Create test case
     |
     v
Add to regression dataset
     |
     v
Run on every new model

Over time, your evaluation dataset becomes more valuable.

It represents actual problems your application has experienced.

Example Regression Dataset

{"query":"How can I cancel after the trial ends?","ground_truth":"Customers can cancel before the next billing cycle."}
{"query":"Can I export my account data?","ground_truth":"Yes, users can request an account data export."}
{"query":"What happens after three failed payments?","ground_truth":"The account may be suspended according to the billing policy."}

Each new incident can add another row.

This is much more useful than repeatedly testing the same five demo prompts.

Use Evaluation Before a Model Upgrade

Suppose you currently use:

Model A

and want to move to:

Model B

Do not switch immediately.

Run:

Production-like dataset
        |
        +---- Model A
        |
        +---- Model B

Then compare:

Quality
Accuracy
Relevance
Completeness
Latency
Resource usage

If Model B performs worse on an important category, you have evidence before deployment.

Use Evaluation After Prompt Changes

Model evaluation is not only about changing the model.

A prompt change can also affect output quality.

For example:

Prompt v1
   |
   v
Model
   |
   v
Results

versus:

Prompt v2
   |
   v
Same Model
   |
   v
Results

Use the same evaluation dataset.

This gives you a way to determine whether the prompt change actually improved the application.

Evaluation and Performance Are Different

A model can produce better answers while becoming slower.

For example:

Model A
Quality: Good
Latency: Low

Model B
Quality: Better
Latency: High

Whether Model B is better depends on your application.

Evaluation scores should therefore be considered alongside operational metrics such as:

Latency
Throughput
GPU utilization
Memory usage
Concurrency
Cost

Foundry Local's evaluation system focuses on response quality metrics. It should be combined with infrastructure monitoring when making deployment decisions.

Dataset Size and Evaluation Time

A larger dataset generally gives you more confidence, but it also requires more evaluation time and compute.

For example:

50 examples
   |
   v
Fast evaluation

versus:

10,000 examples
   |
   v
Much more compute

Start with a small representative dataset during development.

Use a larger regression dataset before production releases when practical.

A Practical Evaluation Workflow

A useful development process looks like this:

Developer changes model/prompt
            |
            v
Run small evaluation set
            |
            v
Check metrics
            |
            v
Inspect failed cases
            |
            v
Fix prompt/model
            |
            v
Run regression dataset
            |
            v
Approve release

This turns evaluation into part of development rather than something done once at the end.

Automating Evaluation

The evaluation APIs make it possible to automate the workflow.

For example:

Git commit
    |
    v
Build application
    |
    v
Deploy candidate model
    |
    v
Upload evaluation dataset
    |
    v
Create evaluation
    |
    v
Wait for completion
    |
    v
Read metrics
    |
    v
Pass / Fail

A CI/CD system could then use thresholds defined by the team.

For example:

F1 >= agreed threshold
Relevance >= agreed threshold
No critical regression cases

The actual thresholds should come from your application's requirements, not arbitrary values.

Avoid Using One Score as the Gate

A release should not necessarily pass because:

Average score = good

You may also require:

Critical test cases = pass
Security cases = pass
Policy cases = pass
Quality score = acceptable

This is especially important for applications where a small number of dangerous errors matter more than average performance.

Clean Up Old Datasets

Evaluation datasets are stored on persistent storage.

That means they can consume space over time.

If you create:

eval-v1
eval-v2
eval-v3
...
eval-v100

you should have a retention policy.

Delete datasets that are no longer needed:

DELETE /api/v1/datasets/{name}

The Datasets API supports deleting datasets.

Be careful with retention when datasets are part of audit or compliance records.

Keep Dataset Versions Traceable

When an evaluation produces an important result, record:

Dataset version
Model version
Prompt version
Evaluator
Judge model
Evaluation date
Evaluation configuration

For example:

Model: support-model-v4
Dataset: support-regression-v7
Judge: judge-model-v2
Evaluators: relevance, fluency, completeness

Now you can reproduce the evaluation later.

Without this information, an evaluation score becomes difficult to interpret.

Common Mistakes

Using Only Easy Test Cases

A model can look excellent when tested only with simple prompts.

Include edge cases.

Relying on One Metric

F1, BLEU, ROUGE, or a judge score alone does not describe every aspect of an AI system.

Treating Judge Scores as Absolute Truth

A judge model is still a model.

Validate important results with human review.

Sending Sensitive Data Outside the Cluster

If privacy is the reason for local evaluation, make sure the evaluator and judge model are also local.

Ignoring Evaluation Results Per Row

Average metrics can hide serious failures.

Inspect individual cases.

Changing the Dataset Between Model Comparisons

Keep the test set stable when comparing models.

Forgetting the Judge Model Requirement

Quality evaluators require a deployed judge model.

Ignoring Dataset Storage

Uploaded datasets remain on persistent storage until removed.

Using Production Customer Data Without Controls

Local does not mean unrestricted.

Apply appropriate access controls and data governance.

When Local Evaluation Makes the Most Sense

Foundry Local evaluation is particularly useful when:

Evaluation data is sensitive.
The model is already deployed on Azure Local.
The organization has disconnected infrastructure.
Internet access is restricted.
Internal policies require data to remain on-premises.
Teams need repeatable model comparisons.
Developers want evaluation close to the inference environment.
AI applications need regression testing before model or prompt changes.

It is less compelling when the workload is already designed around a cloud evaluation platform and there are no data-residency or connectivity constraints.

Summary

Foundry Local on Azure Local provides a way to evaluate deployed AI models directly inside your own environment. You can upload a JSONL or CSV test dataset, run NLP evaluators such as F1, BLEU, and ROUGE, or use another deployed model as a judge for quality measurements such as coherence, fluency, relevance, similarity, and response completeness.

The biggest advantage is data locality. The evaluation dataset, model under test, and judge model can remain inside the Azure Local environment. Microsoft also supports this evaluation workflow in disconnected environments, where evaluation data does not leave the cluster.

For reliable results, do not depend on a single score. Use representative test cases, keep datasets consistent when comparing models, inspect individual failures, maintain a regression dataset, and combine automated evaluation with human review for important workloads.

In short, local model evaluation turns Foundry Local from simply a place to run an AI model into a practical environment for testing whether that model is actually ready for your application.

Running an AI model locally is only half the job.

Once a model is deployed, you still need to answer a much more practical question:

Is the model actually giving good answers for our data and our application?

Foundry Local on Azure Local takes a different approach.

This became available with the 2607 release of Foundry Local on Azure Local.

The evaluation system supports two broad types of evaluation:

NLP-based metrics such as F1, BLEU, ROUGE, GLEU, and METEOR.
Quality evaluation using another deployed model as a judge for dimensions such as coherence, fluency, relevance, similarity, and response completeness.

That makes local evaluation useful not only for disconnected environments, but also for organizations that want sensitive test data to remain inside their own infrastructure.

Why Private Model Evaluation Matters

Suppose a company wants to evaluate an AI assistant using this dataset:

Customer question
Expected answer

The questions might contain:

Customer names
Account information
Internal policies
Support conversations
Product details
Private documentation

Sending that dataset to an external evaluation platform may not be acceptable.

A local evaluation workflow looks different:

Private Test Dataset
        |
        v
Azure Local Cluster
        |
        +----> Model Under Test
        |
        +----> Evaluation Job
        |
        v
Local Evaluation Results

The data stays inside the environment.

For disconnected deployments, Microsoft explicitly states that evaluation runs entirely on the cluster and no evaluation data leaves the environment.

What Is Model Evaluation?

Model evaluation is the process of measuring how well a model performs against a known test set.

For example:

Question:
What is our refund policy?

Expected:
Customers can request a refund within 30 days.

Model:
Customers can request a refund within 30 days of purchase.

The response looks good.

Now consider:

Expected:
Refunds are available within 30 days.

Model:
Refunds are available within 90 days.

The second answer is clearly problematic.

An evaluation system turns these kinds of observations into measurable results.

Instead of manually reading hundreds of responses, you can run a repeatable evaluation.

How Foundry Local Evaluation Works

The overall workflow is straightforward:

1. Deploy model
       |
       v
2. Prepare evaluation dataset
       |
       v
3. Upload dataset
       |
       v
4. Create evaluation
       |
       v
5. Run evaluators
       |
       v
6. Monitor evaluation
       |
       v
7. Download results

The evaluation APIs are exposed through the Foundry inference platform.

The Datasets API manages uploaded evaluation datasets, while the Evaluations API manages evaluation runs and results.

The Evaluation Dataset

The dataset is the foundation of the evaluation.

For the current API, the dataset must contain:

query
ground_truth

For JSONL, a simple dataset can look like:

{"query":"What atoms make up water?","ground_truth":"Hydrogen and oxygen"}
{"query":"What is the capital of France?","ground_truth":"Paris"}
{"query":"What is 2 + 2?","ground_truth":"4"}

The same type of information can be stored in CSV:

query,ground_truth
"What atoms make up water?","Hydrogen and oxygen"
"What is the capital of France?","Paris"
"What is 2 + 2?","4"

The current Datasets API supports jsonl and csv formats.

The documented maximum dataset size is 1 MB, and the file must contain the query and ground_truth columns.

Why Ground Truth Is Important

A model response is difficult to score if there is nothing to compare it against.

Consider:

Query:
What is the return period?

Model:
You can return the product within 30 days.

Is that correct?

You need some expected answer or another evaluation method to determine that.

The ground_truth value gives the evaluation system a reference.

For example:

query:
What is the return period?

ground_truth:
30 days

Now an evaluator has something to compare against.

Not Every Evaluation Needs Exact Matching

Exact text comparison is often too strict for generative AI.

Suppose the expected answer is:

The return period is 30 days.

The model might respond:

Customers have 30 days to return the product.

The wording is different, but the meaning is similar.

This is why Foundry Local provides multiple evaluation metrics.

NLP Evaluators

NLP evaluators compare generated responses against the expected answer using established text-matching metrics.

The current evaluation API supports:

f1_score
bleu
gleu
rouge
meteor

These metrics are useful when the expected answer provides a meaningful reference.

F1 Score

F1 combines precision and recall.

It is useful when you care about how much of the expected information was captured without adding too much unrelated content.

For example:

Expected:
Azure Kubernetes Service

Model:
Azure Kubernetes Service and Azure Functions

Depending on how the evaluator tokenizes the text, extra content can affect the score.

BLEU

BLEU was originally designed for machine translation evaluation.

It measures overlap between generated text and reference text.

It can still be useful when generated output is expected to resemble a known answer, although it should not be treated as a complete measure of semantic quality.

ROUGE

ROUGE is commonly used for text generation and summarization evaluation.

It measures overlap between generated and reference text.

For summarization workloads, it can provide a useful signal when combined with other measurements.

GLEU

GLEU is another text-overlap metric.

It can be useful for evaluating generated text where the relationship between the prediction and reference is important.

METEOR

METEOR provides another way to compare generated output with reference text, considering more than simple exact token overlap.

The important point is that no single metric tells the complete story.

Quality Evaluators

Text overlap is not enough for many AI applications.

Consider this response:

Question:
Explain how Kubernetes service accounts work.

Expected:
A detailed explanation of service accounts,
tokens, identity, and authorization.

A response can have relatively low word overlap and still be a good answer.

This is where quality evaluators are useful.

Foundry Local supports quality dimensions such as:

coherence
fluency
relevance
similarity
response_completeness

A second deployed model acts as the judge.

The architecture becomes:

                 Test Dataset
                     |
                     v
              Model Under Test
                     |
                     v
              Generated Answers
                     |
                     v
                Judge Model
                     |
                     v
                Quality Scores

The judge model itself runs in the local environment.

That is important when the test data cannot leave the cluster.

Model Under Test vs Judge Model

These two models have different jobs.

Model Under Test

This is the model you want to evaluate.

For example:

phi-4-mini

Judge Model

This model evaluates the generated responses.

For example:

gpt-oss-20b

The judge model is only required when using quality evaluators.

You can therefore have:

Model Under Test
       |
       v
Generated Response
       |
       v
Judge Model
       |
       v
Quality Score

The judge should ideally be capable enough to evaluate the type of responses being produced.

Choosing a Judge Model

The judge model matters.

If the judge is too weak, its evaluation may be unreliable.

For example:

Small Model
     |
     v
Evaluates complex reasoning
     |
     v
Possibly weak evaluation

A stronger judge can provide a more useful quality signal.

However, using a larger judge also consumes more compute.

In a local environment, you need to balance:

Evaluation quality
+
GPU/CPU capacity
+
Evaluation time

If you already have a capable local model available, it can be used as the judge rather than sending the test data to an external service.

Uploading the Dataset

The current Datasets API uses:

POST /api/v1/datasets

The request is multipart form data containing:

name
format
file

For example:

name = support-evaluation
format = jsonl
file = support-test.jsonl

The dataset name must follow the documented naming requirements.

A successful upload creates a dataset resource that can then be referenced by an evaluation job.

Example Dataset Upload

Using PowerShell, the current API can be called like this:

$datasetName = "support-evaluation"
$filePath = ".\support-test.jsonl"
$format = "jsonl"

$fileBytes = [System.IO.File]::ReadAllBytes($filePath)
$fileName = Split-Path $filePath -Leaf
$boundary = [System.Guid]::NewGuid().ToString()

$body = @(
    "--$boundary",
    "Content-Disposition: form-data; name=`"name`"",
    "",
    $datasetName,
    "--$boundary",
    "Content-Disposition: form-data; name=`"format`"",
    "",
    $format,
    "--$boundary",
    "Content-Disposition: form-data; name=`"file`"; filename=`"$fileName`"",
    "Content-Type: application/octet-stream",
    "",
    [System.Text.Encoding]::UTF8.GetString($fileBytes),
    "--$boundary--"
) -join "`r`n"

Invoke-RestMethod `
    -Uri "$baseUrl/api/v1/datasets" `
    -Method POST `
    -Headers $headers `
    -ContentType "multipart/form-data; boundary=$boundary" `
    -Body $body

After uploading, check the dataset:

Invoke-RestMethod `
    -Uri "$baseUrl/api/v1/datasets/$datasetName" `
    -Headers $headers

You want the dataset to reach:

phase: Ready

The response also includes the number of rows and dataset size.

Creating an NLP Evaluation

Once the dataset is ready, create an evaluation.

For example:

{
  "name": "support-nlp-evaluation",
  "datasetRef": "support-evaluation",
  "modelRef": "phi4-cpu-demo",
  "evaluators": [
    "f1_score",
    "bleu",
    "rouge"
  ]
}

The request goes to:

POST /api/v1/evaluations

The important fields are:

name
datasetRef
modelRef
evaluators

For NLP evaluation, you do not need a judge model.

Creating a Quality Evaluation

For quality evaluation, add a judge model.

For example:

{
  "name": "support-quality-evaluation",
  "datasetRef": "support-evaluation",
  "modelRef": "phi4-cpu-demo",
  "judgeModelRef": "gpt-oss-20b",
  "evaluators": [
    "coherence",
    "fluency",
    "relevance",
    "similarity"
  ]
}

Now the flow is:

Dataset
   |
   v
Model Under Test
   |
   v
Generated Response
   |
   v
Judge Model
   |
   v
Quality Metrics

The judgeModelRef is required when at least one quality evaluator is included.

Monitoring an Evaluation

Evaluation jobs are asynchronous.

When you submit one, the initial response can look like:

{
  "name": "support-quality-evaluation",
  "phase": "Pending"
}

You then query:

GET /api/v1/evaluations/support-quality-evaluation

The evaluation progresses through states such as:

Pending
   ↓
Running
   ↓
Succeeded

or:

Pending
   ↓
Running
   ↓
Failed

The API exposes fields such as:

phase
message
metrics
startTime
completionTime
resultsAvailable

This makes it possible to build evaluation workflows into CI/CD or internal tooling.

Reading the Aggregate Metrics

After a successful evaluation, you can receive metrics such as:

{
  "metrics": {
    "coherence": 4.2,
    "fluency": 3.8,
    "f1_score": 0.85
  }
}

These numbers give you an aggregate view.

But do not stop there.

An aggregate score can hide individual failures.

For example:

Average F1: 0.91

sounds good.

But perhaps:

Question 1 -> 0.99
Question 2 -> 0.97
Question 3 -> 0.95
Question 4 -> 0.98
Question 5 -> 0.22

The average hides the problematic case.

This is why per-row results are important.

Downloading Detailed Results

After the evaluation succeeds:

GET /api/v1/evaluations/support-quality-evaluation/results

You can request JSON:

/results

or CSV:

/results?format=csv

The JSON results include aggregate information and per-row evaluation results.

The CSV option is useful when you want to inspect the results in Excel or another analysis tool.

Why Per-Row Results Matter

Suppose you are evaluating a customer support assistant.

Your aggregate result is:

Relevance: 4.4
Fluency: 4.6

That looks good.

But one customer's response might be:

Question:
How do I cancel my subscription?

Answer:
Your subscription cannot be cancelled.

while your actual policy says cancellation is supported.

A high average score does not make that answer acceptable.

For production AI, individual failures often matter more than a single average.

Compare Two Model Versions

One of the most useful applications of local evaluation is comparing model versions.

For example:

Model A
Version 1

versus:

Model B
Version 2

Use the same dataset:

Evaluation Dataset
        |
        +----> Model A
        |
        +----> Model B

Then compare:

Metric	Model A	Model B
F1	0.81	0.86
ROUGE	0.74	0.79
Relevance	3.9	4.3
Fluency	4.1	4.4

The exact numbers above are only an example.

Do not treat them as benchmark results.

The point is that the evaluation framework gives you a repeatable way to generate your own numbers.

Use the Same Dataset for Fair Comparison

If you change both the model and the test dataset, the comparison becomes difficult.

A better process is:

Dataset v1
    |
    +---- Model A
    |
    +---- Model B
    |
    +---- Model C

Keep the dataset stable while comparing model versions.

If you need to change the dataset, create a new dataset version and record that change.

Evaluation Is Not the Same as Benchmarking

A benchmark asks:

How does this model perform on a standardized task?

Application evaluation asks:

How does this model perform on our actual workload?

The second question is often more useful for production decisions.

For example, a model may perform well on a general benchmark but struggle with:

Internal terminology
Company policies
Specific document formats
Domain-specific questions
Structured output requirements

Your own evaluation dataset can expose those problems.

Build a Representative Dataset

Do not create a dataset containing only easy questions.

For a support assistant, include:

Simple questions
Complex questions
Ambiguous questions
Edge cases
Policy questions
Out-of-scope questions
Common mistakes
Long prompts
Short prompts

A realistic dataset gives you a much better picture of production behavior.

Separate Dataset Categories

You can also organize tests into groups.

For example:

support-basic.jsonl
support-policy.jsonl
support-edge-cases.jsonl
support-security.jsonl

Then run separate evaluations.

This can tell you more than one large average.

You might discover:

General questions
Excellent

Policy questions
Good

Security questions
Needs improvement

That is actionable.

Evaluation in a Disconnected Environment

Disconnected environments have stricter requirements.

There may be no internet connection at all.

Foundry Local supports evaluation in this setup.

The documented disconnected workflow still follows the same basic process:

Prepare dataset
      ↓
Upload dataset
      ↓
Run evaluation
      ↓
Monitor
      ↓
Download results

The difference is that the required model artifacts and infrastructure are already available locally.

The evaluation data remains inside the disconnected environment.

Why This Matters for Regulated Data

Consider an organization with:

Medical records
Financial records
Government documents
Private customer data
Internal source code

The team may want to evaluate an AI model using real-world examples.

Sending those examples to an external evaluation system may violate internal policies or regulatory requirements.

A local workflow allows:

Sensitive Dataset
       |
       v
Private Cluster
       |
       +-- Model
       |
       +-- Evaluator
       |
       v
Private Results

The evaluation stays close to the data.

That does not automatically make the system compliant with every regulation, but it gives organizations a stronger infrastructure boundary for keeping evaluation data local.

Local Does Not Mean Automatically Secure

This distinction matters.

Keeping evaluation data inside your cluster is useful, but you still need to secure the cluster.

Consider:

Authentication
Authorization
Encryption
RBAC
Network policies
Storage security
Audit logging
Access controls

A private network is not a replacement for security controls.

If the evaluation dataset contains sensitive information, restrict who can:

Upload datasets
Start evaluations
Download results
Delete datasets
Access model endpoints

Protect Evaluation Results

Evaluation results can contain the original query and generated response.

For example:

{
  "query": "Customer's private question",
  "response": "Model-generated answer"
}

That means the results themselves can be sensitive.

Do not assume that because the model is local, the results are safe to distribute everywhere.

Apply the same access controls to:

Dataset
Evaluation job
Raw results
CSV exports
JSON exports

Judge Models Can Also See the Data

When using a quality evaluator, the judge model processes the generated responses.

That means:

Test dataset
     |
     v
Model Under Test
     |
     v
Generated response
     |
     v
Judge Model

Both models are therefore part of the sensitive-data boundary.

Make sure the judge model is also deployed inside the intended environment.

Do not accidentally configure an external judge service when your requirement is that evaluation data remains local.

Evaluation With a Judge Model Is Not Perfect

A model judging another model is useful, but it is not objective truth.

Consider:

Model A
  |
  v
Judge
  |
  v
Score

The judge has its own biases and limitations.

For important workloads, combine:

NLP metrics
+
Judge scores
+
Human review
+
Application-specific tests

This gives you a stronger evaluation process.

Human Review Still Matters

Suppose an AI assistant generates a response that is technically correct but difficult for users to understand.

A text-overlap metric may not catch the problem.

A judge model may give it a good score.

A human reviewer may immediately notice:

Too much jargon
Missing context
Poor explanation
Unhelpful structure
Incorrect assumption

For high-impact applications, keep humans involved in the evaluation process.

Build a Regression Dataset

Once you find a production failure, add it to your test dataset.

For example:

Production bug
     |
     v
Create test case
     |
     v
Add to regression dataset
     |
     v
Run on every new model

Over time, your evaluation dataset becomes more valuable.

It represents actual problems your application has experienced.

Example Regression Dataset

{"query":"How can I cancel after the trial ends?","ground_truth":"Customers can cancel before the next billing cycle."}
{"query":"Can I export my account data?","ground_truth":"Yes, users can request an account data export."}
{"query":"What happens after three failed payments?","ground_truth":"The account may be suspended according to the billing policy."}

Each new incident can add another row.

This is much more useful than repeatedly testing the same five demo prompts.

Use Evaluation Before a Model Upgrade

Suppose you currently use:

Model A

and want to move to:

Model B

Do not switch immediately.

Run:

Production-like dataset
        |
        +---- Model A
        |
        +---- Model B

Then compare:

Quality
Accuracy
Relevance
Completeness
Latency
Resource usage

If Model B performs worse on an important category, you have evidence before deployment.

Use Evaluation After Prompt Changes

Model evaluation is not only about changing the model.

A prompt change can also affect output quality.

For example:

Prompt v1
   |
   v
Model
   |
   v
Results

versus:

Prompt v2
   |
   v
Same Model
   |
   v
Results

Use the same evaluation dataset.

This gives you a way to determine whether the prompt change actually improved the application.

Evaluation and Performance Are Different

A model can produce better answers while becoming slower.

For example:

Model A
Quality: Good
Latency: Low

Model B
Quality: Better
Latency: High

Whether Model B is better depends on your application.

Evaluation scores should therefore be considered alongside operational metrics such as:

Latency
Throughput
GPU utilization
Memory usage
Concurrency
Cost

Foundry Local's evaluation system focuses on response quality metrics. It should be combined with infrastructure monitoring when making deployment decisions.

Dataset Size and Evaluation Time

A larger dataset generally gives you more confidence, but it also requires more evaluation time and compute.

For example:

50 examples
   |
   v
Fast evaluation

versus:

10,000 examples
   |
   v
Much more compute

Start with a small representative dataset during development.

Use a larger regression dataset before production releases when practical.

A Practical Evaluation Workflow

A useful development process looks like this:

Developer changes model/prompt
            |
            v
Run small evaluation set
            |
            v
Check metrics
            |
            v
Inspect failed cases
            |
            v
Fix prompt/model
            |
            v
Run regression dataset
            |
            v
Approve release

This turns evaluation into part of development rather than something done once at the end.

Automating Evaluation

The evaluation APIs make it possible to automate the workflow.

For example:

Git commit
    |
    v
Build application
    |
    v
Deploy candidate model
    |
    v
Upload evaluation dataset
    |
    v
Create evaluation
    |
    v
Wait for completion
    |
    v
Read metrics
    |
    v
Pass / Fail

A CI/CD system could then use thresholds defined by the team.

For example:

F1 >= agreed threshold
Relevance >= agreed threshold
No critical regression cases

The actual thresholds should come from your application's requirements, not arbitrary values.

Avoid Using One Score as the Gate

A release should not necessarily pass because:

Average score = good

You may also require:

Critical test cases = pass
Security cases = pass
Policy cases = pass
Quality score = acceptable

This is especially important for applications where a small number of dangerous errors matter more than average performance.

Clean Up Old Datasets

Evaluation datasets are stored on persistent storage.

That means they can consume space over time.

If you create:

eval-v1
eval-v2
eval-v3
...
eval-v100

you should have a retention policy.

Delete datasets that are no longer needed:

DELETE /api/v1/datasets/{name}

The Datasets API supports deleting datasets.

Be careful with retention when datasets are part of audit or compliance records.

Keep Dataset Versions Traceable

When an evaluation produces an important result, record:

Dataset version
Model version
Prompt version
Evaluator
Judge model
Evaluation date
Evaluation configuration

For example:

Model: support-model-v4
Dataset: support-regression-v7
Judge: judge-model-v2
Evaluators: relevance, fluency, completeness

Now you can reproduce the evaluation later.

Without this information, an evaluation score becomes difficult to interpret.

Common Mistakes

Using Only Easy Test Cases

A model can look excellent when tested only with simple prompts.

Include edge cases.

Relying on One Metric

F1, BLEU, ROUGE, or a judge score alone does not describe every aspect of an AI system.

Treating Judge Scores as Absolute Truth

A judge model is still a model.

Validate important results with human review.

Sending Sensitive Data Outside the Cluster

If privacy is the reason for local evaluation, make sure the evaluator and judge model are also local.

Ignoring Evaluation Results Per Row

Average metrics can hide serious failures.

Inspect individual cases.

Changing the Dataset Between Model Comparisons

Keep the test set stable when comparing models.

Forgetting the Judge Model Requirement

Quality evaluators require a deployed judge model.

Ignoring Dataset Storage

Uploaded datasets remain on persistent storage until removed.

Using Production Customer Data Without Controls

Local does not mean unrestricted.

Apply appropriate access controls and data governance.

When Local Evaluation Makes the Most Sense

Foundry Local evaluation is particularly useful when:

Evaluation data is sensitive.
The model is already deployed on Azure Local.
The organization has disconnected infrastructure.
Internet access is restricted.
Internal policies require data to remain on-premises.
Teams need repeatable model comparisons.
Developers want evaluation close to the inference environment.
AI applications need regression testing before model or prompt changes.

It is less compelling when the workload is already designed around a cloud evaluation platform and there are no data-residency or connectivity constraints.

Summary

In short, local model evaluation turns Foundry Local from simply a place to run an AI model into a practical environment for testing whether that model is actually ready for your application.