Azure Content Understanding: Measuring Agentic Document Extraction Accuracy and Cost

Introduction

Enterprise applications rarely work with perfectly structured documents.

Invoices, contracts, forms, reports, manuals, and business records can contain paragraphs, tables, images, handwritten information, headers, footnotes, and data spread across multiple pages.

Extracting useful information from these documents is therefore more complicated than simply reading text.

Agentic document extraction takes a different approach. Instead of treating document processing as a single OCR operation, an agentic workflow can reason about document content, determine what information is relevant, and produce structured output for downstream applications.

The important engineering question is not whether an agent can extract information.

It is:

How accurate is the extraction, how much does it cost, and where does the additional reasoning actually provide value?

This article presents a practical benchmark for evaluating agentic document extraction using accuracy, completeness, latency, and cost.

What Is Agentic Document Extraction?

A traditional document extraction workflow might look like this:

Document
   |
   v
OCR
   |
   v
Extracted Text
   |
   v
Application

An agentic workflow can involve additional reasoning:

Document
   |
   v
Content Understanding
   |
   v
Document Analysis
   |
   v
Agentic Reasoning
   |
   v
Structured Data
   |
   v
Application

The difference is important.

A simple OCR system may tell you what text appears on a page.

An agentic workflow can be designed to determine what that information means in the context of the application's requirements.

For example, an invoice may contain:

Invoice Number
Vendor
Invoice Date
Subtotal
Tax
Total
Payment Terms
Purchase Order

The application may only need six of those fields.

An extraction workflow can therefore be evaluated on whether it identifies the required information accurately and consistently.

Why Accuracy Alone Is Not Enough

Suppose an extraction system produces:

{
  "invoiceNumber": "INV-1045",
  "vendor": "Example Ltd",
  "total": 12500
}

The values might be correct.

But what if the system failed to extract the tax amount, payment terms, or purchase-order number that the application requires?

The output is partially correct, but incomplete.

A better evaluation separates:

Field accuracy
Field completeness
Data type correctness
Structural correctness
Confidence
Processing time
Cost

Building a Benchmark Dataset

Start with a representative document collection.

For example:

benchmark/
    invoices/
    purchase-orders/
    contracts/
    forms/
    reports/

Include documents with different characteristics:

Different page counts
Different layouts
Tables
Images
Scanned pages
Missing fields
Repeated fields
Multiple dates
Different number formats

The dataset should represent the documents the production system will actually process.

Avoid creating a benchmark consisting only of clean digital documents.

Creating Ground Truth

Every benchmark document needs a trusted reference.

For example:

{
  "invoiceNumber": "INV-1045",
  "invoiceDate": "2026-08-20",
  "vendor": "Example Ltd",
  "subtotal": 10000,
  "tax": 2500,
  "total": 12500
}

This is the ground truth against which extracted data can be compared.

The reference should be reviewed by a person or generated through a controlled process.

Do not treat the model's own output as ground truth.

Defining an Extraction Schema

A fixed schema makes benchmarking easier.

For example:

{
  "invoiceNumber": "",
  "invoiceDate": "",
  "vendorName": "",
  "purchaseOrderNumber": "",
  "subtotal": 0,
  "tax": 0,
  "total": 0,
  "currency": ""
}

The schema should specify:

Field name
Data type
Required or optional status
Expected format
Validation rules

This prevents evaluation from becoming subjective.

Measuring Field-Level Accuracy

Each extracted field can be compared against the reference value.

For example:

Field	Expected	Extracted	Result
Invoice Number	INV-1045	INV-1045	Correct
Vendor	Example Ltd	Example Ltd	Correct
Tax	2500	2500	Correct
Total	12500	12000	Incorrect

A simple field accuracy calculation is:

Field Accuracy =
Correct Fields
--------------
Evaluated Fields

For example, if 6 out of 7 fields are correct:

6 / 7 = 85.7%

This is only an illustrative calculation.

The benchmark should use actual results from the selected dataset.

Exact Matching vs Normalized Matching

Not every difference means that the extracted value is wrong.

For example:

Expected:
2026-08-20

Extracted:
08/20/2026

These strings differ, but they may represent the same date.

Similarly:

Expected:
12500.00

Extracted:
12,500

The numeric value is equivalent.

Therefore, evaluation should normalize values where appropriate.

For dates:

DateTime.Parse(value);

For numbers:

decimal.Parse(
    value,
    NumberStyles.Number,
    CultureInfo.InvariantCulture);

The parser should match the formats expected in the benchmark.

Measuring Completeness

Accuracy asks whether extracted fields are correct.

Completeness asks whether the required fields were extracted at all.

For example:

Required Fields = 8
Extracted Fields = 7
Correct Fields = 7

Completeness is:

7 / 8 = 87.5%

The system may have perfect accuracy on the fields it returned while still missing important information.

This distinction is especially important for enterprise workflows.

Measuring Structural Accuracy

Some applications require more than flat fields.

Consider a purchase order:

{
  "supplier": "Example Ltd",
  "items": [
    {
      "description": "Laptop",
      "quantity": 5,
      "unitPrice": 900
    }
  ]
}

The extraction system must correctly identify:

Supplier
Item count
Item descriptions
Quantities
Prices

A document can therefore have correct top-level fields while producing an incorrect item list.

Benchmark nested structures separately.

Testing Tables

Tables are particularly important in document extraction.

For example:

Product	Quantity	Unit Price	Total
Laptop	5	900	4500
Monitor	5	250	1250

The benchmark should verify every row and relevant column.

Test:

Row count
Column mapping
Quantity
Unit price
Calculated total

Do not evaluate only whether the table was detected.

The extracted values must also be associated with the correct rows and columns.

Testing Ambiguous Documents

Real documents often contain multiple values that look similar.

For example:

Invoice Date: 08/20/2026
Due Date: 09/19/2026
Purchase Date: 08/18/2026

A weak extraction system may return the first date it finds.

A good benchmark should include these cases.

The question should be:

Did the system identify the correct semantic field?

not:

Did the system find a date?

Testing Missing Fields

Documents do not always contain every expected field.

For example:

Invoice:
Invoice Number
Vendor
Date
Total

Purchase Order:
Not present

The correct output may be:

{
  "purchaseOrderNumber": null
}

rather than:

{
  "purchaseOrderNumber": "UNKNOWN"
}

The expected behavior should be defined before testing.

This prevents the benchmark from rewarding fabricated values.

Measuring False Extractions

A system can also create information that does not exist in the document.

For example:

Document:
No Purchase Order Number

Output:
PO-1042

This is not simply a missing-field problem.

It is a false extraction.

Track these separately because fabricated values can be more dangerous than empty values in business workflows.

Measuring Overall Accuracy

For a larger benchmark, calculate metrics at multiple levels.

Field Accuracy

Measures individual field correctness.

Document Accuracy

Measures whether all required fields for a document were correctly extracted.

Critical Field Accuracy

Measures only fields where errors have a significant business impact.

For example:

Invoice Number
Total Amount
Currency
Vendor

A system could achieve high overall field accuracy while still making unacceptable errors in critical financial fields.

Measuring Processing Time

Accuracy is only one side of the evaluation.

Record processing time for each document.

For example:

Document
   |
   v
Processing Start
   |
   v
Extraction Complete

Then:

Processing Time =
Completion Time - Start Time

Measure different document sizes separately.

A 2-page invoice and a 100-page contract should not necessarily be treated as the same workload.

Measuring Cost

Agentic extraction can involve multiple processing stages.

A conceptual cost model is:

Document Processing
        +
Model Usage
        +
Storage
        +
Indexing
        +
Application Infrastructure

For benchmarking, first calculate the cost of the extraction operation itself.

Then evaluate the broader workflow if the application requires additional processing.

Do not present a single cost number as universal.

Actual cost depends on document size, processing configuration, model usage, region, and other service settings.

Accuracy vs Cost

A useful benchmark compares multiple configurations.

For example:

Configuration	Accuracy	Processing Time	Cost
Configuration A	Measure	Measure	Measure
Configuration B	Measure	Measure	Measure
Configuration C	Measure	Measure	Measure

The goal is to identify a practical operating point.

The most accurate configuration may not be the best production choice if the improvement is small but the processing cost increases substantially.

Likewise, the cheapest configuration may be unsuitable if critical fields are frequently incorrect.

Testing Different Document Classes

Do not calculate only one overall accuracy score.

Break results down by document type.

For example:

Invoices
Contracts
Forms
Purchase Orders
Reports

A system might perform very well on invoices but poorly on complex contracts.

An overall average could hide that problem.

A better report might look like:

Document Type	Accuracy	Completeness	Avg. Processing Time
Invoices	Measure	Measure	Measure
Contracts	Measure	Measure	Measure
Forms	Measure	Measure	Measure
Reports	Measure	Measure	Measure

Confidence and Human Review

For business-critical workflows, automation does not always need to mean zero human involvement.

A practical architecture can use confidence-based review:

Extraction
    |
    v
Validation
    |
    +--> High confidence --> Automated workflow
    |
    +--> Low confidence --> Human review

For example, a financial application might automatically process clearly extracted invoices while sending uncertain documents to an employee.

The threshold should be determined through testing rather than chosen arbitrarily.

Validating Extracted Data

Extraction should be followed by business validation.

For example:

if (invoice.Total < 0)
{
    throw new ValidationException(
        "Invoice total cannot be negative.");
}

For financial documents, you might also validate relationships:

Subtotal + Tax ≈ Total

The exact tolerance depends on the business rules and rounding behavior.

This is important because an extraction system can return syntactically valid data that is logically inconsistent.

Common Mistakes

Measuring Only OCR Accuracy

Recognizing text correctly does not mean extracting the correct business fields.

Using Only Clean Documents

Production documents contain scans, tables, unusual layouts, and missing values.

Ignoring Missing Fields

An extraction system should distinguish between "not found" and "invented."

Using Model Output as Ground Truth

Ground truth must come from a trusted reference.

Reporting Only an Overall Score

Break results down by document type and field category.

Ignoring Cost

Higher accuracy may come with additional processing cost.

Skipping Business Validation

Correctly extracted values can still violate application rules.

Troubleshooting Poor Extraction

If extraction quality is low, inspect the failed document rather than changing the entire system immediately.

Check:

Document quality.
Page layout.
Tables.
Scanned content.
Field ambiguity.
Schema definition.
Validation rules.
Missing-field handling.
Context provided to the extraction process.
Model or processing configuration.

Classify errors:

Wrong Field
Missing Field
False Extraction
Formatting Error
Structural Error
Business Rule Error

This makes improvement work more targeted.

Production Considerations

Before deploying agentic extraction, determine which fields can tolerate errors.

For example:

Low Risk
    |
    +--> Internal document classification

Higher Risk
    |
    +--> Financial totals

Very High Risk
    |
    +--> Regulatory or contractual information

The more serious the business consequence, the stronger the validation and human-review requirements should be.

Sensitive documents also require appropriate access controls, retention policies, and data-handling practices.

Best Practices

Build Ground Truth First

Do not begin benchmarking until expected outputs are defined.

Measure Field-Level Results

Overall document accuracy can hide important failures.

Separate Missing and Incorrect Values

They represent different failure modes.

Test Complex Documents

Include tables, multiple dates, repeated fields, and missing information.

Measure Cost and Latency

Accuracy alone does not determine production suitability.

Add Business Validation

Use deterministic rules to catch logically invalid output.

Review Critical Fields

Automated extraction should receive additional validation when errors have significant consequences.

Re-Test After Configuration Changes

Small changes to extraction configuration can affect accuracy and cost.

Advantages

Can automate structured information extraction from complex documents.
Can reduce manual data-entry work.
Can handle documents containing multiple forms of information.
Supports structured output for downstream applications.
Provides opportunities to combine document understanding with reasoning and validation.

Disadvantages

Accuracy can vary significantly between document types.
Complex documents may still require human review.
Agentic processing can introduce additional processing cost.
Incorrect extraction can create downstream business problems.
Benchmarking requires carefully prepared ground-truth data.
Higher extraction accuracy does not automatically mean lower overall application cost.

Conclusion

Agentic document extraction is most useful when it is evaluated as an engineering system rather than simply demonstrated with a few successful documents.

A serious benchmark should begin with representative documents and trusted ground-truth data. From there, measure field-level accuracy, completeness, structural correctness, false extractions, processing time, and cost.

Different document types should also be evaluated separately. An extraction workflow that performs well on invoices may behave very differently on long contracts or complex reports.

Most importantly, extraction should not be the final validation step. Deterministic business rules can catch impossible values and inconsistent relationships, while confidence-based human review can handle documents that are too uncertain for fully automated processing.

The right production configuration is therefore not necessarily the one with the highest raw accuracy.

It is the configuration that provides reliable extraction, acceptable processing time, manageable cost, and an appropriate level of human validation for the business risk involved.