Benchmarking AI Retrieval Latency Across Fabric IQ and Vector Stores

Retrieval latency is one of the most important performance characteristics of an enterprise RAG system.

A response may depend on a large language model, but the model cannot generate a grounded answer until the application retrieves the relevant information. If retrieval takes too long, the entire user experience becomes slower.

This becomes particularly important when comparing a managed knowledge-layer approach such as Microsoft Fabric IQ with a more traditional vector-store architecture.

The comparison should not be reduced to a simple question such as:

"Which one is faster?"

Retrieval performance depends on the complete pipeline, including document ingestion, indexing, query processing, filtering, ranking, network communication, concurrency, and the amount of context returned to the model.

This article presents a practical framework for benchmarking retrieval latency across Fabric IQ and vector-store-based architectures without relying on fabricated performance numbers.

Introduction

A simplified RAG request looks like this:

User Query
    |
    v
Query Processing
    |
    v
Knowledge Retrieval
    |
    v
Ranking / Filtering
    |
    v
Context Assembly
    |
    v
LLM
    |
    v
Answer

When measuring the end-to-end response time, retrieval is only one component.

A useful latency model is:

Total Latency =
Query Processing
+
Retrieval
+
Ranking
+
Context Assembly
+
LLM Generation

If the purpose of an experiment is specifically to compare retrieval technologies, the benchmark must isolate retrieval latency from generation latency.

Otherwise, differences in LLM response time can hide meaningful differences in the retrieval layer.

What Is Retrieval Latency?

Retrieval latency is the time between submitting a search request and receiving the relevant retrieval results.

For example:

Request Started
       |
       v
Query Sent
       |
       v
Knowledge System
       |
       v
Results Returned
       |
       v
Request Completed

The measurement should define precisely where the timer starts and stops.

For example:

var stopwatch = Stopwatch.StartNew();

var results = await retriever.SearchAsync(
    query,
    cancellationToken);

stopwatch.Stop();

var latency = stopwatch.Elapsed;

This measures the application-visible retrieval operation.

It does not necessarily measure all internal work performed by the service.

Why p95 Matters

Average latency is useful but insufficient.

Consider:

Request Latencies

20 ms
22 ms
24 ms
25 ms
27 ms
30 ms
31 ms
200 ms
450 ms
900 ms

The average can hide the slow requests.

For interactive enterprise applications, p95 and p99 are often more informative because they show the experience of slower requests.

Track at least:

p50
p95
p99

Where:

p50 represents typical latency.
p95 represents slower requests experienced by roughly the upper tail.
p99 exposes more extreme latency behavior.

What Are You Actually Comparing?

A fair benchmark needs clearly defined systems.

For example:

System A
Application
   |
   v
Fabric IQ
   |
   v
Retrieved Context

versus:

System B
Application
   |
   v
Vector Store
   |
   v
Retrieved Context

The application, query set, documents, and evaluation conditions should remain as consistent as possible.

Otherwise, the benchmark may compare different architectures rather than different retrieval technologies.

Benchmark Variables

The benchmark should control variables such as:

Variable	Example
Document corpus	Same corpus
Query set	Same queries
Embedding strategy	Same where applicable
Top-K	Same
Metadata filters	Equivalent
Region	Same or documented
Network location	Same
Concurrency	Same
Query preprocessing	Same
Warm/cold state	Separately measured

Not every architecture exposes exactly the same retrieval configuration.

When that happens, document the difference instead of pretending the systems are identical.

Build a Representative Corpus

A benchmark corpus should resemble the production workload.

For example:

1,000 documents
10,000 documents
100,000 documents

Document types might include:

PDF documents
Technical documentation
Policies
Product manuals
Support articles
Structured reports
Tables

Do not benchmark only a tiny collection and assume that the results will scale linearly.

Retrieval behavior can change as corpus size, indexing strategy, metadata filtering, and concurrency change.

Query Dataset

Create a fixed evaluation query set.

For example:

Q001:
What is the default request timeout?

Q002:
How do I configure authentication?

Q003:
What happens when the service returns HTTP 429?

Q004:
Which roles can approve an expense?

Q005:
What are the deployment prerequisites?

The same queries should be executed against both systems.

The dataset should include:

Short queries
Long queries
Exact factual questions
Semantic questions
Multi-condition questions
Metadata-filtered queries
Queries with no relevant result

Warm and Cold Benchmarks

A common benchmarking mistake is measuring only warm requests.

The first request may have different behavior from subsequent requests because of:

Connection establishment
Service initialization
DNS resolution
Authentication
Cache state
Application startup

Measure separately:

Cold Start
Warm Requests

For example:

await retriever.SearchAsync(
    benchmarkQuery,
    cancellationToken);

var stopwatch = Stopwatch.StartNew();

await retriever.SearchAsync(
    benchmarkQuery,
    cancellationToken);

stopwatch.Stop();

The warm-up request should not automatically be included in the steady-state latency calculation.

Sequential Benchmark

Start with sequential execution.

foreach (var query in queries)
{
    var stopwatch = Stopwatch.StartNew();

    await retriever.SearchAsync(
        query,
        cancellationToken);

    stopwatch.Stop();

    results.Add(
        new RetrievalResult(
            query.Id,
            stopwatch.Elapsed));
}

This provides a baseline without introducing concurrent load.

However, real applications rarely process only one query at a time.

Concurrent Benchmark

After the sequential test, introduce controlled concurrency.

await Parallel.ForEachAsync(
    queries,
    new ParallelOptions
    {
        MaxDegreeOfParallelism = 10,
        CancellationToken = cancellationToken
    },
    async (query, token) =>
    {
        await BenchmarkQueryAsync(
            query,
            token);
    });

Do not immediately increase concurrency to hundreds or thousands of requests.

Measure progressively:

depending on the limits of the environment.

The objective is to identify how latency changes under load.

Measuring Throughput

Latency and throughput are related but different.

Throughput measures how many requests the system can process over a period.

For example:

Requests Completed
------------------
Elapsed Time

A system can have:

Low latency + Low throughput

or:

Moderate latency + High throughput

Both characteristics matter for enterprise workloads.

Record:

Requests per second
p50 latency
p95 latency
p99 latency
Error rate

Create a Common Retriever Interface

A benchmark harness should abstract the underlying technology.

public interface IRetriever
{
    string Name { get; }

    Task<IReadOnlyList<RetrievalResult>> SearchAsync(
        string query,
        RetrievalOptions options,
        CancellationToken cancellationToken);
}

Then implement adapters:

FabricIqRetriever
VectorStoreRetriever

The benchmark can execute both through the same interface.

foreach (var retriever in retrievers)
{
    foreach (var query in queries)
    {
        await RunBenchmarkAsync(
            retriever,
            query,
            cancellationToken);
    }
}

This keeps the benchmark logic independent of the underlying implementation.

Retrieval Options

Define common retrieval parameters.

public sealed record RetrievalOptions(
    int TopK,
    string? TenantId,
    string? DocumentType);

This becomes particularly important when comparing filtered retrieval.

For example:

TopK = 5
Tenant = CustomerA
DocumentType = Policy

Both systems should implement an equivalent retrieval requirement.

If one system performs the filtering inside the search engine and another filters results afterward, the benchmark should explicitly document the architectural difference.

Measure More Than Latency

A fast retrieval result is not useful if it retrieves the wrong documents.

Capture both performance and quality:

Metric	Purpose
p50	Typical latency
p95	Tail latency
p99	Extreme tail
Throughput	Capacity
Error rate	Reliability
Recall@K	Retrieval coverage
MRR	Ranking quality
Context size	Downstream cost
No-result rate	Query effectiveness

This prevents the benchmark from optimizing performance at the expense of retrieval quality.

Recall@K

Suppose the evaluation dataset contains the expected source document for each query.

If the correct document appears in the top five results:

Recall@5 = 1

Otherwise:

Recall@5 = 0

Across many queries:

Recall@5 =
Queries with relevant result in top 5
-------------------------------------
Total queries

This can be implemented in a benchmark harness:

bool ContainsExpectedDocument(
    IReadOnlyList<RetrievalResult> results,
    string expectedDocumentId)
{
    return results.Any(
        x => x.DocumentId == expectedDocumentId);
}

For more complex questions, there may be multiple acceptable source documents.

Measuring Ranking Quality

Recall does not tell you where the relevant document appeared.

For example:

Query A:
1. Correct
2. Irrelevant
3. Irrelevant

is preferable to:

Query B:
1. Irrelevant
2. Irrelevant
3. Correct

Mean Reciprocal Rank can capture this difference.

For a relevant result at position r:

MRR Contribution = 1 / r

The average across queries gives the MRR.

Metadata Filtering

Enterprise retrieval frequently requires filtering.

For example:

Tenant = Contoso
Department = Finance
Region = EU
Classification = Internal

Filtering can affect latency because the retrieval engine has additional constraints to evaluate.

Therefore, benchmark at least two categories:

Unfiltered Retrieval
Filtered Retrieval

The difference can reveal whether metadata filtering creates a significant performance cost.

Corpus Size Scaling

Run the same query set across increasing corpus sizes.

For example:

10K documents
100K documents
1M documents

Then compare:

Corpus Size vs p95 Latency

The objective is not to assume a specific scaling curve.

It is to observe how each architecture behaves as the workload grows.

Query Length Scaling

Query size can also affect performance.

Test:

Short:
"authentication timeout"

Medium:
"How is authentication timeout configured?"

Long:
"Explain how authentication timeout is configured
for services deployed behind the enterprise gateway."

This helps determine whether query complexity affects retrieval latency or quality.

Network Latency

Managed knowledge services and external vector stores can introduce network overhead.

The benchmark environment should therefore record:

Application Region
Service Region
Network Path
Protocol
Connection Reuse

If the application and retrieval service are deployed in different regions, the measured latency includes that network distance.

That is valid if it reflects production architecture, but it should be documented.

Connection Reuse

HTTP connection management can influence latency.

A benchmark that creates a new client for every request can measure connection setup rather than the actual retrieval system.

Prefer a long-lived client:

public sealed class RetrievalClient
{
    private readonly HttpClient _httpClient;

    public RetrievalClient(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }
}

Use the application's normal HTTP-client lifecycle when benchmarking an HTTP-based vector store.

Retry Effects

Retries can distort latency.

Suppose a request takes:

Primary attempt = 200 ms
Retry = 500 ms

The final request latency may become:

700+ ms

The benchmark should record:

Attempt Count
Final Latency
Failure Reason

Do not hide retries inside an aggregate latency number.

Timeout Testing

A benchmark should also determine how systems behave near timeout boundaries.

For example:

using var timeout =
    new CancellationTokenSource(
        TimeSpan.FromSeconds(2));

await retriever.SearchAsync(
    query,
    options,
    timeout.Token);

Record:

Timeout Rate
Error Rate
p95 Latency

A system with excellent average latency but frequent timeout behavior under load may not be suitable for an interactive workload.

Benchmark Harness Design

A useful benchmark result model is:

public sealed record BenchmarkResult(
    string Retriever,
    string QueryId,
    TimeSpan Latency,
    int ResultCount,
    bool ContainsExpectedResult,
    int AttemptCount,
    bool Success);

After execution, calculate:

p50
p95
p99
Recall@K
Error Rate
Average Result Count

Keep the raw per-request records rather than storing only aggregate numbers.

Raw measurements make later analysis possible.

Comparing the Results

A final report might look like:

Metric	Fabric IQ	Vector Store
p50 Retrieval	Measure	Measure
p95 Retrieval	Measure	Measure
p99 Retrieval	Measure	Measure
Throughput	Measure	Measure
Error Rate	Measure	Measure
Recall@5	Measure	Measure
MRR	Measure	Measure
Filtered Retrieval	Measure	Measure

The values should come from the benchmark environment.

Avoid statements such as "System A is 40% faster" unless the benchmark actually produced that result under a clearly defined workload.

Interpreting Benchmark Results

Suppose the results show:

System A:
Lower p50
Higher p95
Higher Recall

System B:
Higher p50
Lower p95
Lower Recall

There is no single winner.

The correct choice depends on workload requirements.

For interactive applications, p95 may matter more than p50.

For high-volume batch processing, throughput and cost may matter more.

For enterprise knowledge systems, retrieval quality may outweigh a small latency difference.

Cost Per Retrieval

Latency should also be considered alongside operational cost.

Capture:

Infrastructure Cost
Embedding Cost
Query Cost
Network Cost
Operational Complexity

For managed services, pricing models may differ significantly from self-managed vector stores.

The benchmark should therefore include a cost model rather than assuming that the fastest system is the cheapest system.

Common Benchmarking Mistakes

Using Different Datasets

If the systems receive different documents, the comparison is invalid.

Measuring Only Average Latency

Average values hide tail behavior.

Ignoring Warm-Up

Cold-start behavior can distort steady-state measurements.

Running Only One Query

One query cannot represent an enterprise workload.

Ignoring Retrieval Quality

Fast incorrect results are not useful.

Mixing LLM Latency With Retrieval Latency

If the goal is to compare retrieval systems, measure retrieval independently.

Hiding Retries

Retries can significantly affect observed latency.

Using Production Traffic Without Controls

Production benchmarks can affect users and introduce uncontrolled variables.

Changing Multiple Variables at Once

If chunking, embeddings, Top-K, and retrieval infrastructure all change simultaneously, it becomes difficult to identify the cause of performance differences.

Best Practices

Define the exact benchmark boundary before testing.
Use the same document corpus across systems.
Use the same evaluation queries.
Measure p50, p95, and p99 latency.
Test both cold and warm behavior.
Test controlled concurrency levels.
Measure throughput and error rates.
Evaluate Recall@K and MRR alongside latency.
Test metadata-filtered retrieval separately.
Record retries and timeout behavior.
Keep the network topology consistent or document differences.
Reuse clients and connections as production code would.
Preserve raw benchmark measurements.
Test multiple corpus sizes.
Include cost and operational complexity in the final decision.
Repeat benchmarks after significant architecture or configuration changes.

Frequently Asked Questions

Is Fabric IQ faster than a vector database?

There is no universal answer. Retrieval latency depends on corpus size, configuration, network topology, filtering, query characteristics, concurrency, and the specific vector-store implementation.

Should I benchmark retrieval separately from the LLM?

Yes. If the objective is to compare retrieval technologies, isolate retrieval latency from LLM generation latency. End-to-end latency can then be measured separately.

Is p95 more important than average latency?

For many interactive applications, p95 is more useful because it exposes the slower requests that affect a meaningful portion of users. Average latency should still be reported.

How many queries should a benchmark contain?

There is no universal minimum. The evaluation set should be large and diverse enough to represent the production workload and cover different query types.

Should I benchmark filtered retrieval?

Yes. Enterprise applications frequently apply tenant, role, department, region, or classification filters. These conditions can change both performance and retrieval quality.

Can a vector store be faster simply because it is closer to the application?

Yes. Network distance can significantly affect observed latency. The benchmark should use deployment topology representative of the intended production architecture.

What is the most important retrieval metric?

There is no single metric. A useful evaluation combines latency, tail latency, throughput, reliability, and retrieval quality such as Recall@K and MRR.

Conclusion

Benchmarking AI retrieval systems requires more discipline than measuring how quickly a single search request returns.

A meaningful comparison between Fabric IQ and vector-store-based architectures should use the same corpus, equivalent queries, controlled retrieval parameters, consistent network conditions, and clearly defined measurements. It should separately evaluate cold and warm behavior, sequential and concurrent workloads, filtered retrieval, corpus scaling, and failure conditions.

Most importantly, latency must be evaluated together with retrieval quality. A system that returns results quickly but consistently misses the correct information is not necessarily better for an enterprise RAG workload.

The best benchmark therefore produces a multidimensional view:

Latency
   +
Throughput
   +
Reliability
   +
Retrieval Quality
   +
Cost
   +
Operational Complexity

That approach turns a technology comparison into an engineering decision based on measurable workload characteristics rather than assumptions about which retrieval architecture should be faster.