Elasticsearch can return search results quickly when queries are designed well and the cluster has enough resources. However, as indexes grow and search requirements become more complex, some queries can become noticeably slower.
A search that once took a few milliseconds might begin taking hundreds of milliseconds or several seconds. The cause could be an expensive query clause, a large number of matching documents, nested queries, aggregations, or work performed across multiple shards.
Looking only at the total response time does not always reveal the actual problem. You need to understand which parts of the query consume the most time and how Elasticsearch executes them.
The Elasticsearch Profile API helps developers investigate query execution at a more detailed level. It provides information about query components, collectors, and other execution stages, making it easier to identify expensive operations.
In this article, we will explore how to use the Profile API, interpret its output, identify performance problems, and improve slow Elasticsearch queries.
What Is the Elasticsearch Profile API?
The Elasticsearch Profile API provides detailed timing information about how Elasticsearch executes a search request.
Normally, a search response tells you which documents matched and how long the request took according to the reported timing. It does not explain which query clause consumed the most execution time.
The Profile API adds that information by reporting execution details for individual query components and related operations.
For example, suppose an application searches an index containing millions of product documents. The query filters products by category, checks whether they are available, and sorts the results by price.
The request might be slow because one query clause examines too many documents, or because another part of the query requires expensive processing. Profiling helps you investigate these possibilities.
A profiled search can provide information about:
Query execution time for individual query components.
How often query components execute.
Collector activity associated with collecting matching documents.
Aggregation execution details when profiling is enabled for aggregations.
The execution structure of the query on individual shards.
The Profile API adds overhead to query execution. Its timings should therefore be used for diagnosis rather than treated as exact measurements of normal production latency.
Enable Profiling for a Search Request
You can enable profiling by adding "profile": true to a search request.
Consider an index named products with fields such as category, available, and price.
The following request finds available electronics and sorts them by price:
GET products/_search
{
"profile": true,
"query": {
"bool": {
"filter": [
{
"term": {
"category": "electronics"
}
},
{
"term": {
"available": true
}
}
]
}
},
"sort": [
{
"price": {
"order": "asc"
}
}
]
}
The request returns the normal search results together with a profile section containing execution details.
The profile information is organized by shard. Each shard can include query details, collector information, and other profiling data depending on the operations performed.
A simplified response structure looks like this:
{
"took": 35,
"hits": {
"total": {
"value": 1200,
"relation": "eq"
},
"hits": []
},
"profile": {
"shards": [
{
"id": "[node][products][0]",
"searches": [
{
"query": [
{
"type": "BooleanQuery",
"description": "...",
"time_in_nanos": 2500000,
"breakdown": {
"score": 0,
"match": 500000
},
"children": []
}
],
"collector": {
"name": "SimpleTopScoreDocCollector",
"reason": "search_top_hits",
"time_in_nanos": 300000
}
}
]
}
]
}
}
This is an illustrative example rather than a complete response. Actual profile output varies according to the query, Elasticsearch version, and execution path.
The key fields include type, description, time_in_nanos, breakdown, and children.
The time_in_nanos field reports timing in nanoseconds. The children array describes nested query components, where present. The breakdown object provides more detailed timing categories.
Do not interpret the example's timing values as benchmarks. Real measurements depend on the index, data distribution, hardware, cache state, and cluster workload.
Understand the Main Parts of the Profile Response
The Profile API produces detailed output. Reading every field without a specific question can make the investigation unnecessarily complicated.
Start with the execution structure and look for components that appear expensive relative to the work they perform.
Query Components
The query section describes the query execution tree for each shard.
For example, a Boolean query may contain multiple filter or scoring clauses. Those clauses can appear as child components in the profile output.
Look for query components that have high execution time or are executed frequently.
A query component that takes a relatively small amount of time once may still be expensive when executed many times. Conversely, a component with a larger cumulative time may be reasonable if it processes a substantial amount of data.
Use the query description to connect the profile entry to the corresponding part of your search request.
Timing Breakdown
The breakdown field contains timing information for different parts of query execution. Depending on the query type and Elasticsearch version, the available fields can include activities related to matching, scoring, building scorers, advancing through documents, and counting matches.
These fields help explain where the reported query time is being spent.
However, they should not be treated as completely independent measurements that can always be added together. Some activities are related, and the profiling implementation has its own overhead.
Compare timing patterns rather than attempting to calculate the total request duration by summing every field.
Collector Information
Collectors help gather matching documents and determine which results should be returned.
The collector section can help explain work associated with retrieving top search results or collecting documents for aggregations.
For example, a query might match many documents even though the application requests only a small page of results. Collecting the top results may still require substantial work.
If query execution looks relatively inexpensive but the overall search remains slow, investigate collectors, aggregations, sorting, and other parts of the request.
Shard-Level Information
Elasticsearch distributes data across shards, so a search can involve multiple shard-level executions.
The Profile API reports execution details by shard. This can reveal whether the same query behaves differently across different shards.
However, a slow shard is not automatically evidence of a poorly designed query. Differences in shard size, data distribution, node resources, and concurrent activity may contribute to the result.
Use shard-level profiling alongside cluster monitoring and index statistics.
Find Expensive Query Clauses
One of the most useful applications of the Profile API is identifying which clause contributes most to query execution time.
Consider a search that filters products by category and availability while also checking a text field.
GET products/_search
{
"profile": true,
"query": {
"bool": {
"filter": [
{
"term": {
"category": "electronics"
}
},
{
"term": {
"available": true
}
}
],
"must": [
{
"match": {
"description": "wireless noise cancelling headphones"
}
}
]
}
}
}
The text query may require more work than the exact-match filters, depending on the index and the number of documents involved.
Use the profile output to investigate the actual execution cost instead of assuming which clause is slow.
If the text query dominates, consider whether the search terms, field mapping, analyzer, and query requirements are appropriate.
If the filters dominate, inspect the mapping and data distribution. A field's name alone does not determine whether a query is inexpensive.
Check Whether Scoring Is Necessary
Elasticsearch calculates relevance scores for queries that need them. If your application only needs to filter documents and does not use relevance ranking, scoring may be unnecessary.
For example, a product availability search might use only filters:
GET products/_search
{
"query": {
"bool": {
"filter": [
{
"term": {
"category": "electronics"
}
},
{
"term": {
"available": true
}
}
]
}
}
}
Filter context does not calculate relevance scores for the clauses in that context. It is appropriate when a condition determines whether a document matches rather than how relevant it is.
This does not mean every filter query will be fast. Performance still depends on the field mapping, selectivity, data distribution, and other work required by the request.
Investigate Expensive Aggregations
Aggregations can become a major source of search latency, particularly when they process large numbers of matching documents or create many buckets.
Suppose a product search groups matching products by brand and calculates the average price for each brand.
GET products/_search
{
"size": 0,
"profile": true,
"query": {
"term": {
"category": "electronics"
}
},
"aggs": {
"brands": {
"terms": {
"field": "brand.keyword",
"size": 50
},
"aggs": {
"average_price": {
"avg": {
"field": "price"
}
}
}
}
}
}
Setting size to zero avoids returning individual search hits when the application only needs aggregation results.
The Profile API can include aggregation profiling information when profiling is enabled. Inspect this information when the request spends substantial time calculating buckets or metrics.
If the aggregation is expensive, investigate the following questions:
Does the application need every requested bucket?
Is the query matching more documents than necessary?
Is the aggregation field mapped appropriately?
Is a high-cardinality field creating too many buckets?
Could the application use a more suitable aggregation strategy?
Reducing the requested bucket count can help in some cases, but it does not automatically reduce every part of the aggregation's work. Measure the effect of each change.
Compare Profile Results Before and After a Change
Profiling becomes more useful when you use it to evaluate a specific optimization.
For example, suppose you change a query to reduce unnecessary matching work. Run the same profiled request before and after the change, ideally against comparable data and under similar conditions.
Compare:
The execution time of the relevant query components.
The number of matching documents, where available.
Collector and aggregation behavior.
The execution patterns across shards.
The actual end-to-end search latency outside profiling.
Keep the query, data, result size, and other settings consistent during the comparison.
A profiled request may behave differently from a normal request because profiling adds overhead. Do not use the Profile API's reported time as the sole measure of application performance.
For a more representative comparison, run repeated unprofiled searches under controlled conditions and compare latency distributions, including median and higher-percentile response times.
Common Mistakes When Using the Profile API
Treating Profile Time as Total Request Latency
The Profile API reports execution details for profiled search operations. Its measurements do not represent every part of the request's end-to-end journey.
Network time, coordinating-node work, response serialization, and client-side processing can affect the total latency observed by an application.
Compare profile data with ordinary request timing and cluster monitoring.
Profiling Every Production Request
Profiling introduces overhead and creates additional response data. Enabling it indiscriminately can increase resource consumption and make performance investigations harder.
Use profiling selectively for specific diagnostic requests. Apply appropriate access controls and avoid exposing detailed profiling output unnecessarily.
Assuming the Slowest Clause Is Always the Root Cause
A slow component may reflect the amount of data it processes rather than a defective query.
Before changing the query, understand its purpose, the field mappings, the index structure, and the number of documents involved.
Ignoring Shard Distribution
A query can perform differently across shards because of differences in data volume or distribution.
If one shard consistently behaves differently, investigate shard size, node resources, and index organization rather than changing the query blindly.
Optimizing Without Rechecking Results
A faster query is not useful if it changes the application's required search behavior.
After each optimization, verify that the returned documents, ordering, aggregations, and business rules remain correct.
Best Practices for Elasticsearch Query Profiling
Use the following practices to make investigations more reliable:
Begin with a reproducible slow query and record its normal response latency.
Enable profiling only for requests that need detailed investigation.
Start with the query execution tree before examining individual timing fields.
Investigate collectors and aggregations when query clauses do not explain the observed latency.
Compare results across relevant shards.
Change one important factor at a time.
Validate mapping changes and query rewrites against representative data.
Measure normal request latency before and after the optimization.
Use cluster monitoring to investigate CPU, memory, disk, and workload contention.
Keep a record of the original query, the change, and the measured result.
Advantages and Limitations of the Profile API
The Profile API provides detailed visibility into query execution, helping developers identify expensive clauses, investigate aggregations, and understand differences between shards.
It is particularly useful when a query is complex enough that its performance cannot be explained by the request structure alone.
However, profiling adds overhead, produces verbose output, and does not replace cluster-level monitoring. It also cannot guarantee that a query will behave the same way under every workload.
For these reasons, use it as a diagnostic tool alongside normal latency measurements, query inspection, and representative performance tests.
Summary
The Elasticsearch Profile API helps developers investigate slow searches by exposing query execution details, timing breakdowns, collectors, aggregation activity, and shard-level information.
Start by enabling profiling on a specific search request. Identify expensive query components, investigate sorting and aggregations when necessary, and check whether shard-level differences explain the results.
After making an optimization, verify that the query still returns the correct results and measure normal request latency under comparable conditions.
The most effective approach is to use profiling to identify a likely cause, make a targeted change, and validate the improvement with repeatable measurements. This keeps query optimization focused on actual performance problems instead of assumptions.
Join the conversation! Your thoughts help the community grow.