Introduction
Enterprise applications rarely work with perfectly structured documents.
Invoices, contracts, forms, reports, manuals, and business records can contain paragraphs, tables, images, handwritten information, headers, footnotes, and data spread across multiple pages.
Extracting useful information from these documents is therefore more complicated than simply reading text.
Agentic document extraction takes a different approach. Instead of treating document processing as a single OCR operation, an agentic workflow can reason about document content, determine what information is relevant, and produce structured output for downstream applications.
The important engineering question is not whether an agent can extract information.
It is:
How accurate is the extraction, how much does it cost, and where does the additional reasoning actually provide value?
This article presents a practical benchmark for evaluating agentic document extraction using accuracy, completeness, latency, and cost.
What Is Agentic Document Extraction?
A traditional document extraction workflow might look like this:
Document
|
v
OCR
|
v
Extracted Text
|
v
Application
An agentic workflow can involve additional reasoning:
Document
|
v
Content Understanding
|
v
Document Analysis
|
v
Agentic Reasoning
|
v
Structured Data
|
v
Application
The difference is important.
A simple OCR system may tell you what text appears on a page.
An agentic workflow can be designed to determine what that information means in the context of the application's requirements.
For example, an invoice may contain:
Invoice Number
Vendor
Invoice Date
Subtotal
Tax
Total
Payment Terms
Purchase Order
The application may only need six of those fields.
An extraction workflow can therefore be evaluated on whether it identifies the required information accurately and consistently.
Why Accuracy Alone Is Not Enough
Suppose an extraction system produces:
{
"invoiceNumber": "INV-1045",
"vendor": "Example Ltd",
"total": 12500
}
The values might be correct.
But what if the system failed to extract the tax amount, payment terms, or purchase-order number that the application requires?
The output is partially correct, but incomplete.
A better evaluation separates:
Field accuracy
Field completeness
Data type correctness
Structural correctness
Confidence
Processing time
Cost
Building a Benchmark Dataset
Start with a representative document collection.
For example:
benchmark/
invoices/
purchase-orders/
contracts/
forms/
reports/
Include documents with different characteristics:
Different page counts
Different layouts
Tables
Images
Scanned pages
Missing fields
Repeated fields
Multiple dates
Different number formats
The dataset should represent the documents the production system will actually process.
Avoid creating a benchmark consisting only of clean digital documents.
Creating Ground Truth
Every benchmark document needs a trusted reference.
For example:
{
"invoiceNumber": "INV-1045",
"invoiceDate": "2026-08-20",
"vendor": "Example Ltd",
"subtotal": 10000,
"tax": 2500,
"total": 12500
}
This is the ground truth against which extracted data can be compared.
The reference should be reviewed by a person or generated through a controlled process.
Do not treat the model's own output as ground truth.
Defining an Extraction Schema
A fixed schema makes benchmarking easier.
For example:
{
"invoiceNumber": "",
"invoiceDate": "",
"vendorName": "",
"purchaseOrderNumber": "",
"subtotal": 0,
"tax": 0,
"total": 0,
"currency": ""
}
The schema should specify:
This prevents evaluation from becoming subjective.
Measuring Field-Level Accuracy
Each extracted field can be compared against the reference value.
For example:
| Field | Expected | Extracted | Result |
|---|
| Invoice Number | INV-1045 | INV-1045 | Correct |
| Vendor | Example Ltd | Example Ltd | Correct |
| Tax | 2500 | 2500 | Correct |
| Total | 12500 | 12000 | Incorrect |
A simple field accuracy calculation is:
Field Accuracy =
Correct Fields
--------------
Evaluated Fields
For example, if 6 out of 7 fields are correct:
6 / 7 = 85.7%
This is only an illustrative calculation.
The benchmark should use actual results from the selected dataset.
Exact Matching vs Normalized Matching
Not every difference means that the extracted value is wrong.
For example:
Expected:
2026-08-20
Extracted:
08/20/2026
These strings differ, but they may represent the same date.
Similarly:
Expected:
12500.00
Extracted:
12,500
The numeric value is equivalent.
Therefore, evaluation should normalize values where appropriate.
For dates:
DateTime.Parse(value);
For numbers:
decimal.Parse(
value,
NumberStyles.Number,
CultureInfo.InvariantCulture);
The parser should match the formats expected in the benchmark.
Measuring Completeness
Accuracy asks whether extracted fields are correct.
Completeness asks whether the required fields were extracted at all.
For example:
Required Fields = 8
Extracted Fields = 7
Correct Fields = 7
Completeness is:
7 / 8 = 87.5%
The system may have perfect accuracy on the fields it returned while still missing important information.
This distinction is especially important for enterprise workflows.
Measuring Structural Accuracy
Some applications require more than flat fields.
Consider a purchase order:
{
"supplier": "Example Ltd",
"items": [
{
"description": "Laptop",
"quantity": 5,
"unitPrice": 900
}
]
}
The extraction system must correctly identify:
Supplier
Item count
Item descriptions
Quantities
Prices
A document can therefore have correct top-level fields while producing an incorrect item list.
Benchmark nested structures separately.
Testing Tables
Tables are particularly important in document extraction.
For example:
| Product | Quantity | Unit Price | Total |
|---|
| Laptop | 5 | 900 | 4500 |
| Monitor | 5 | 250 | 1250 |
The benchmark should verify every row and relevant column.
Test:
Row count
Column mapping
Quantity
Unit price
Calculated total
Do not evaluate only whether the table was detected.
The extracted values must also be associated with the correct rows and columns.
Testing Ambiguous Documents
Real documents often contain multiple values that look similar.
For example:
Invoice Date: 08/20/2026
Due Date: 09/19/2026
Purchase Date: 08/18/2026
A weak extraction system may return the first date it finds.
A good benchmark should include these cases.
The question should be:
Did the system identify the correct semantic field?
not:
Did the system find a date?
Testing Missing Fields
Documents do not always contain every expected field.
For example:
Invoice:
Invoice Number
Vendor
Date
Total
Purchase Order:
Not present
The correct output may be:
{
"purchaseOrderNumber": null
}
rather than:
{
"purchaseOrderNumber": "UNKNOWN"
}
The expected behavior should be defined before testing.
This prevents the benchmark from rewarding fabricated values.
Measuring False Extractions
A system can also create information that does not exist in the document.
For example:
Document:
No Purchase Order Number
Output:
PO-1042
This is not simply a missing-field problem.
It is a false extraction.
Track these separately because fabricated values can be more dangerous than empty values in business workflows.
Measuring Overall Accuracy
For a larger benchmark, calculate metrics at multiple levels.
Field Accuracy
Measures individual field correctness.
Document Accuracy
Measures whether all required fields for a document were correctly extracted.
Critical Field Accuracy
Measures only fields where errors have a significant business impact.
For example:
Invoice Number
Total Amount
Currency
Vendor
A system could achieve high overall field accuracy while still making unacceptable errors in critical financial fields.
Measuring Processing Time
Accuracy is only one side of the evaluation.
Record processing time for each document.
For example:
Document
|
v
Processing Start
|
v
Extraction Complete
Then:
Processing Time =
Completion Time - Start Time
Measure different document sizes separately.
A 2-page invoice and a 100-page contract should not necessarily be treated as the same workload.
Measuring Cost
Agentic extraction can involve multiple processing stages.
A conceptual cost model is:
Document Processing
+
Model Usage
+
Storage
+
Indexing
+
Application Infrastructure
For benchmarking, first calculate the cost of the extraction operation itself.
Then evaluate the broader workflow if the application requires additional processing.
Do not present a single cost number as universal.
Actual cost depends on document size, processing configuration, model usage, region, and other service settings.
Accuracy vs Cost
A useful benchmark compares multiple configurations.
For example:
| Configuration | Accuracy | Processing Time | Cost |
|---|
| Configuration A | Measure | Measure | Measure |
| Configuration B | Measure | Measure | Measure |
| Configuration C | Measure | Measure | Measure |
The goal is to identify a practical operating point.
The most accurate configuration may not be the best production choice if the improvement is small but the processing cost increases substantially.
Likewise, the cheapest configuration may be unsuitable if critical fields are frequently incorrect.
Testing Different Document Classes
Do not calculate only one overall accuracy score.
Break results down by document type.
For example:
Invoices
Contracts
Forms
Purchase Orders
Reports
A system might perform very well on invoices but poorly on complex contracts.
An overall average could hide that problem.
A better report might look like:
| Document Type | Accuracy | Completeness | Avg. Processing Time |
|---|
| Invoices | Measure | Measure | Measure |
| Contracts | Measure | Measure | Measure |
| Forms | Measure | Measure | Measure |
| Reports | Measure | Measure | Measure |
Confidence and Human Review
For business-critical workflows, automation does not always need to mean zero human involvement.
A practical architecture can use confidence-based review:
Extraction
|
v
Validation
|
+--> High confidence --> Automated workflow
|
+--> Low confidence --> Human review
For example, a financial application might automatically process clearly extracted invoices while sending uncertain documents to an employee.
The threshold should be determined through testing rather than chosen arbitrarily.
Validating Extracted Data
Extraction should be followed by business validation.
For example:
if (invoice.Total < 0)
{
throw new ValidationException(
"Invoice total cannot be negative.");
}
For financial documents, you might also validate relationships:
Subtotal + Tax ≈ Total
The exact tolerance depends on the business rules and rounding behavior.
This is important because an extraction system can return syntactically valid data that is logically inconsistent.
Common Mistakes
Measuring Only OCR Accuracy
Recognizing text correctly does not mean extracting the correct business fields.
Using Only Clean Documents
Production documents contain scans, tables, unusual layouts, and missing values.
Ignoring Missing Fields
An extraction system should distinguish between "not found" and "invented."
Using Model Output as Ground Truth
Ground truth must come from a trusted reference.
Reporting Only an Overall Score
Break results down by document type and field category.
Ignoring Cost
Higher accuracy may come with additional processing cost.
Skipping Business Validation
Correctly extracted values can still violate application rules.
Troubleshooting Poor Extraction
If extraction quality is low, inspect the failed document rather than changing the entire system immediately.
Check:
Document quality.
Page layout.
Tables.
Scanned content.
Field ambiguity.
Schema definition.
Validation rules.
Missing-field handling.
Context provided to the extraction process.
Model or processing configuration.
Classify errors:
Wrong Field
Missing Field
False Extraction
Formatting Error
Structural Error
Business Rule Error
This makes improvement work more targeted.
Production Considerations
Before deploying agentic extraction, determine which fields can tolerate errors.
For example:
Low Risk
|
+--> Internal document classification
Higher Risk
|
+--> Financial totals
Very High Risk
|
+--> Regulatory or contractual information
The more serious the business consequence, the stronger the validation and human-review requirements should be.
Sensitive documents also require appropriate access controls, retention policies, and data-handling practices.
Best Practices
Build Ground Truth First
Do not begin benchmarking until expected outputs are defined.
Measure Field-Level Results
Overall document accuracy can hide important failures.
Separate Missing and Incorrect Values
They represent different failure modes.
Test Complex Documents
Include tables, multiple dates, repeated fields, and missing information.
Measure Cost and Latency
Accuracy alone does not determine production suitability.
Add Business Validation
Use deterministic rules to catch logically invalid output.
Review Critical Fields
Automated extraction should receive additional validation when errors have significant consequences.
Re-Test After Configuration Changes
Small changes to extraction configuration can affect accuracy and cost.
Advantages
Can automate structured information extraction from complex documents.
Can reduce manual data-entry work.
Can handle documents containing multiple forms of information.
Supports structured output for downstream applications.
Provides opportunities to combine document understanding with reasoning and validation.
Disadvantages
Accuracy can vary significantly between document types.
Complex documents may still require human review.
Agentic processing can introduce additional processing cost.
Incorrect extraction can create downstream business problems.
Benchmarking requires carefully prepared ground-truth data.
Higher extraction accuracy does not automatically mean lower overall application cost.
Conclusion
Agentic document extraction is most useful when it is evaluated as an engineering system rather than simply demonstrated with a few successful documents.
A serious benchmark should begin with representative documents and trusted ground-truth data. From there, measure field-level accuracy, completeness, structural correctness, false extractions, processing time, and cost.
Different document types should also be evaluated separately. An extraction workflow that performs well on invoices may behave very differently on long contracts or complex reports.
Most importantly, extraction should not be the final validation step. Deterministic business rules can catch impossible values and inconsistent relationships, while confidence-based human review can handle documents that are too uncertain for fully automated processing.
The right production configuration is therefore not necessarily the one with the highest raw accuracy.
It is the configuration that provides reliable extraction, acceptable processing time, manageable cost, and an appropriate level of human validation for the business risk involved.