Amazon S3 Vectors Pre-Filtering: How Metadata Filters Improve Vector Search

Introduction

Vector search is useful when an application needs to find information by meaning rather than exact keywords. It is commonly used in semantic search, retrieval-augmented generation (RAG), recommendation systems, document search, and AI agents.

But real applications rarely search every vector in an index.

A customer-support application may need to search only one customer's documents. A multi-tenant application may need to restrict results to a particular organization. A document system may need to search only active documents or files belonging to a specific category.

This is where metadata filtering becomes important.

Amazon S3 Vectors now supports metadata pre-filtering, which applies metadata filters before similarity search on indexes using the enhanced index mode. This changes how a filtered vector query is evaluated and can improve recall when the filter is highly selective. AWS states that pre-filtering can return up to five times more matching vectors than the previous behavior on highly selective filters.

This article explains how S3 Vectors metadata pre-filtering works, why it matters for vector search, how to configure it, and what developers should consider when building RAG and AI applications.

What Is Metadata Filtering in Vector Search?

A vector represents data as a numerical embedding.

For example, a document about database performance might be converted into an embedding such as:

[0.12, -0.31, 0.77, 0.05, ...]

The vector captures semantic information that can be used for similarity search.

Metadata adds structured information to that vector.

For example:

{
  "tenant_id": "customer-100",
  "category": "technical",
  "status": "active",
  "created_year": 2026
}

The vector represents what the content means, while metadata describes where the content belongs or what attributes it has.

This allows an application to ask a query such as:

Find documents similar to "How do I troubleshoot database timeouts?"
but only where:

tenant_id = customer-100
category = technical
status = active

That combination is much more useful than performing a semantic search across the entire vector index.

Why Pre-Filtering Matters

There are two important stages in a filtered vector search:

Query Vector
     |
     v
Metadata Filter
     |
     v
Eligible Vectors
     |
     v
Similarity Search
     |
     v
Top-K Results

With pre-filtering, S3 Vectors first identifies the vectors that satisfy the metadata conditions and then performs the similarity search against that matching set.

AWS describes this as the ENHANCED index mode. Existing indexes can remain in the CLASSIC mode until they are updated. In CLASSIC mode, metadata filtering and similarity search are performed together during candidate evaluation.

This difference becomes important when the filter matches only a small part of a large index.

Imagine an index containing eight million support tickets.

One customer may have only 400 tickets.

Without effectively narrowing the search space first, the similarity search can draw candidates from the much larger collection. With pre-filtering, the search can focus on the 400 vectors belonging to that customer before selecting the most similar results.

That is the central idea behind metadata pre-filtering.

S3 Vectors Metadata Model

S3 Vectors allows metadata to be attached to vectors as key-value pairs.

For example:

{
  "key": "document-001",
  "data": {
    "float32": [0.12, 0.31, 0.77]
  },
  "metadata": {
    "tenant_id": "tenant-42",
    "category": "documentation",
    "active": true,
    "created_year": 2026
  }
}

Filterable metadata can contain strings, numbers, booleans, and lists. By default, metadata attached to vectors is filterable unless the index configuration explicitly designates particular metadata keys as non-filterable.

This means developers can design metadata around the questions the application needs to ask.

For a multi-tenant RAG application, useful fields might include:

tenant_id
document_type
department
status
created_date
access_level

The important design principle is to store metadata that represents structured filtering requirements, rather than putting the entire document into filterable metadata.

Creating a Vector Index

A vector index must be created with a dimension that matches the embedding model being used.

For example, AWS provides an example using a 1,536-dimensional embedding:

aws s3vectors create-index \
  --index-name product-catalog \
  --vector-bucket-name my-vector-bucket \
  --dimension 1536 \
  --distance-metric cosine

The vector dimension must match the output dimension of the embedding model. AWS also recommends choosing a distance metric appropriate for the embedding model.

For text embeddings, cosine distance is a common choice, but the correct metric should be based on the characteristics and recommendations of the embedding model.

Adding Metadata to Vectors

When storing vectors, attach the metadata that your application will use for filtering.

For example:

aws s3vectors put-vectors \
  --index-name product-catalog \
  --vector-bucket-name my-vector-bucket \
  --vectors '[{
    "key": "doc-001",
    "data": {
      "float32": [0.1, 0.2, 0.3]
    },
    "metadata": {
      "tenant_id": "tenant-42",
      "category": "technical",
      "active": true,
      "created_year": 2026
    }
  }]'

The actual vector must contain the same number of dimensions configured for the index.

The metadata in this example provides several possible filtering dimensions:

tenant_id    → tenant isolation
category     → document classification
active       → current records
created_year → time-based filtering

This structure is especially useful for applications where semantic relevance alone is not enough.

Running a Filtered Vector Query

The QueryVectors operation accepts both the query vector and a metadata filter.

A simple filter can look like this:

aws s3vectors query-vectors \
  --index-name product-catalog \
  --vector-bucket-name my-vector-bucket \
  --query-vector '{"float32": [0.1, 0.2, 0.3]}' \
  --top-k 10 \
  --return-metadata \
  --filter '{"tenant_id": "tenant-42"}'

The query asks S3 Vectors to return the most similar vectors while limiting the search to vectors associated with the specified tenant.

For multiple conditions, use logical operators.

{
  "$and": [
    {
      "tenant_id": "tenant-42"
    },
    {
      "category": "technical"
    },
    {
      "active": true
    }
  ]
}

This is useful for applications where search results must satisfy several constraints at the same time.

Supported Metadata Filters

S3 Vectors supports several metadata filter operations.

These operators are documented by AWS for S3 Vectors metadata filtering.

Using Pre-Filtering for Multi-Tenant RAG

One of the most practical applications is multi-tenant RAG.

Suppose a SaaS application stores documents for several companies:

Vector Index
|
+-- Tenant A
|    +-- Documents
|
+-- Tenant B
|    +-- Documents
|
+-- Tenant C
     +-- Documents

A user from Tenant B should not retrieve information belonging to Tenant A.

The application can include the tenant identifier in the vector metadata:

{
  "tenant_id": "tenant-b",
  "document_type": "policy",
  "status": "active"
}

Then the query can enforce the tenant boundary:

{
  "$and": [
    {
      "tenant_id": "tenant-b"
    },
    {
      "status": "active"
    }
  ]
}

This is an important architectural pattern because the retrieval query itself carries the application's data-scope requirements.

However, developers should not treat metadata filtering as a complete authorization system. Application authorization should still be enforced independently.

Pre-Filtering vs Traditional Filtering

The difference becomes clearer when comparing the two approaches.

AWS reports that on highly selective filters, pre-filtering can return up to five times more matching vectors than the previous behavior on CLASSIC indexes. This is an AWS-reported result for the relevant scenario, not a universal performance guarantee for every workload.

Updating an Existing Index

You do not necessarily need to rebuild your vectors to use pre-filtering.

An existing index can be switched to enhanced index mode:

aws s3vectors update-index-mode \
  --vector-bucket-name my-vector-bucket \
  --index-name product-catalog \
  --index-mode ENHANCED

AWS states that the change takes effect in place and does not require re-ingesting existing vectors. Existing indexes use CLASSIC until the mode is changed.

This makes the feature easier to adopt in an existing vector-search application.

Testing Pre-Filtering Before Changing the Index

AWS also provides a way to test enhanced query behavior on a CLASSIC index by specifying queryMode=ENHANCED for the query.

This is useful when evaluating the behavior before changing the index configuration.

For example, conceptually:

Existing Index
     |
     v
CLASSIC
     |
     +---- Query with ENHANCED mode
     |
     v
Compare Results
     |
     v
Decide Whether to Change Index Mode

AWS documents this testing approach as part of the index-mode configuration.

For production systems, testing the same query patterns against representative data is preferable to assuming that a change will produce the same effect for every workload.

Prefix Filtering With $startsWith

Enhanced query behavior also adds $startsWith.

This can be useful when identifiers contain hierarchical information.

For example:

matter-4417/exhibits/
matter-4417/contracts/
matter-4417/emails/

A query can use:

{
  "$startsWith": {
    "document_id": "matter-4417/exhibits/"
  }
}

This allows the application to restrict retrieval to a specific path or identifier prefix. AWS specifically describes $startsWith for paths, URLs, and hierarchical keys.

Designing Metadata for Production

Good metadata design is important.

Do not add every possible application property simply because S3 Vectors supports metadata.

Start with the filters your application actually needs.

For a document RAG system:

{
  "tenant_id": "tenant-42",
  "document_type": "contract",
  "department": "legal",
  "status": "active",
  "created_year": 2026
}

This is more useful than storing unrelated UI information or large application objects.

Think about metadata as part of the retrieval contract:

Application Requirement
        |
        v
Metadata Field
        |
        v
Query Filter
        |
        v
Scoped Vector Search

Filterable vs Non-Filterable Metadata

Not every piece of metadata needs to be filterable.

S3 Vectors supports non-filterable metadata for information that should be returned with a vector but does not need to participate in filtering. AWS documents this as useful for larger contextual information such as document text or descriptions.

For example:

{
  "metadata": {
    "tenant_id": "tenant-42",
    "category": "technical",
    "description": "Detailed document content..."
  }
}

If description does not need to be used as a query condition, making it non-filterable can be more appropriate.

The distinction is important:

Filterable metadata
→ Used to restrict search

Non-filterable metadata
→ Returned as context

Common Mistakes

Filtering After Retrieving Results

A common application-level pattern is:

Vector Search
     |
     v
Top 10 Results
     |
     v
Application Filter

This can be problematic because relevant vectors that satisfy the metadata condition may never make it into the initial top-K results.

Pre-filtering changes the process to:

Metadata Filter
     |
     v
Eligible Vectors
     |
     v
Similarity Search
     |
     v
Top 10 Results

That difference is particularly important for highly selective filters.

Using Metadata as the Only Security Boundary

Metadata filtering helps scope retrieval, but it should not replace application-level authorization.

A multi-tenant application should validate access independently.

Storing Too Much Filterable Metadata

Filterable metadata is intended for attributes used during queries.

Do not turn it into a general-purpose document store.

Ignoring Filter Selectivity

A filter such as:

{
  "status": "active"
}

may match most of the index.

A filter such as:

{
  "tenant_id": "tenant-42"
}

may match a much smaller subset.

The benefit of pre-filtering depends heavily on the characteristics of the workload.

Troubleshooting

Query Returns Fewer Results Than topK

topK is a maximum number of results, not a guarantee that the requested number will always be returned.

If the filter matches only a small number of vectors, fewer results can be returned. AWS documents this behavior for metadata-filtered queries.

Metadata Filter Returns an Error

Check whether the metadata field is filterable.

If a field was configured as non-filterable, it cannot be used in a query filter.

Query Returns 403 Forbidden

When using metadata filtering or requesting metadata in the response, the caller needs both s3vectors:QueryVectors and s3vectors:GetVectors permissions.

Review the IAM policy before changing application code.

$startsWith Does Not Work

Check that the query is using enhanced behavior. AWS documents $startsWith as requiring ENHANCED query behavior.

Best Practices

1. Design metadata around retrieval requirements. Store fields that your application actually needs for filtering.

2. Use tenant identifiers for multi-tenant retrieval. Scope vector queries to the correct tenant before retrieving context.

3. Choose filterable metadata carefully. Do not treat every metadata field as a search field.

4. Test selective queries. Compare retrieval quality using realistic datasets and filters.

5. Use enhanced mode where it fits the workload. Pre-filtering is particularly relevant when filters narrow the candidate set significantly.

6. Keep authorization separate. Retrieval filtering should complement application security, not replace it.

7. Monitor result quality. A technically valid query can still produce poor RAG results if metadata is incomplete or incorrectly assigned.

8. Validate embedding dimensions. The vector dimension must match the index configuration.

Advantages of S3 Vectors Pre-Filtering

• Narrows the vector search to metadata-matching records before similarity search.

• Can improve recall for highly selective filters.

• Supports common metadata types and logical operators.

• Supports tenant, category, status, and time-based filtering.

• Existing indexes can be updated without re-ingesting vectors.

• Adds $startsWith support for hierarchical filtering.

• Can improve retrieval behavior for RAG and agentic applications.

• AWS states that the feature is available without additional cost in supported S3 Vectors Regions.

Disadvantages and Considerations

• The benefit depends on the selectivity and quality of your metadata.

• Incorrect metadata can produce incomplete retrieval results.

• Pre-filtering does not replace authorization.

• Developers need to understand filterable versus non-filterable metadata.

• Query limits still apply, including limits on filter constraints.

• Vector quality still depends on the embedding model and index configuration.

• Applications should validate retrieval quality rather than assuming pre-filtering automatically improves every query.

When Should You Use Metadata Pre-Filtering?

Pre-filtering is especially relevant when semantic similarity needs to be combined with a strict scope.

Examples include:

• Multi-tenant RAG

• Customer-specific document search

• Enterprise knowledge bases

• Legal document retrieval

• Product catalogs

• User-specific AI agents

• Time-scoped document search

• Region-specific content retrieval

For example, an AI agent should not search an entire enterprise knowledge base when it is supposed to answer questions using only one customer's documents.

The query should encode that scope before similarity ranking occurs.

Summary

Amazon S3 Vectors metadata pre-filtering changes the way developers can approach filtered vector search.

Instead of treating metadata as an additional condition applied around similarity search, the enhanced index mode allows S3 Vectors to resolve the metadata filter first and perform similarity search within the matching set. AWS reports that this can substantially improve recall for highly selective filters, including up to five times more matching vectors in the relevant comparison.

For developers building RAG systems, AI agents, or semantic search applications, the feature is particularly useful when results must satisfy both semantic relevance and structured business constraints.

The key is good metadata design. Store fields such as tenant, category, status, and date when they represent real retrieval requirements. Use filterable metadata for search constraints and non-filterable metadata for additional context.

Pre-filtering does not eliminate the need for application authorization, testing, or retrieval evaluation. It is a retrieval capability that can make filtered vector search more precise and useful when the application's metadata accurately represents the scope of the query.