OpenTelemetry helps developers understand what happens inside an application by collecting traces, metrics, and logs. These signals make it easier to investigate slow requests, identify errors, and monitor application performance across distributed systems.
However, collecting more telemetry does not always produce better observability. An application can generate a large amount of telemetry while providing little additional diagnostic value. One common cause is high-cardinality attributes.
Attributes such as request IDs, user IDs, session IDs, and complete URLs can introduce thousands or millions of unique values into telemetry data. Depending on the signal, instrumentation, and backend, this can increase storage requirements, processing costs, query complexity, and the number of metric time series.
The problem often becomes more noticeable after an application moves into production. A development environment might process a few hundred requests, while a production service handles millions of requests every day.
In this article, we will explore what high-cardinality attributes are, how they affect OpenTelemetry costs, how to identify problematic attributes, and how to reduce unnecessary telemetry without losing important diagnostic information.
Understanding Cardinality in OpenTelemetry
Cardinality describes the number of distinct values associated with a field or attribute.
Consider an application that records an HTTP request metric with the following attributes:
HTTP method
HTTP response status
Application name
Request ID
User ID
The HTTP method may have only a few values, such as GET, POST, PUT, and DELETE. The response status also has a relatively limited set of values.
Request IDs and user IDs are different. They may have thousands or millions of distinct values.
This difference matters because metrics backends commonly represent combinations of metric names and attribute values as separate time series.
For example, consider this metric:
http.server.request.duration
Suppose it includes the following attributes:
http.request.method = GET
http.response.status_code = 200
service.name = order-service
These values are relatively stable. Now imagine adding a unique request ID:
request.id = 8f4d2a91
The next request produces another value:
request.id = b71e9c42
If the backend creates a separate time series for each distinct attribute combination, each request ID can contribute to a new series.
The actual behavior depends on the metrics backend, instrumentation, and aggregation configuration. Nevertheless, unbounded attributes on metrics are a common source of excessive cardinality.
Why High-Cardinality Attributes Increase Costs
High-cardinality attributes can affect several parts of an observability system.
More Metric Time Series
A metric time series is generally identified by its metric name and its associated dimensions or labels.
Suppose an application records request duration for 20 combinations of HTTP method and status code.
Without high-cardinality attributes, the application may produce approximately 20 relevant series for this metric, depending on its aggregation and other dimensions.
If every request adds a unique request ID, the number of series can grow dramatically.
Consider this simplified example:
Configuration | Approximate distinct series |
|---|
4 HTTP methods and 5 status categories | 20 |
Same attributes plus 1,000 distinct request IDs | Up to 20,000 |
Same attributes plus 100,000 distinct request IDs | Up to 2,000,000 |
These figures illustrate the possible scale of the problem. They are not universal predictions because real backends apply different aggregation, filtering, and storage rules.
Higher Ingestion and Storage Costs
More unique metric series can increase memory usage, indexing work, and storage requirements.
Some observability platforms also charge according to active metric series, ingested data volume, retained data, or combinations of these measures.
Traces and logs have different cost models, but excessive attributes can still increase payload sizes, indexing work, and storage consumption.
For example, recording an entire URL containing a large search query in every span may produce substantially more data than recording the route template and a small number of relevant attributes.
Slower Queries and Dashboards
High cardinality can make queries more expensive because the backend must process a larger number of series or indexed values.
A dashboard that groups request duration by request ID is rarely useful. It may produce thousands of groups without helping the developer understand application behavior.
A dashboard grouped by route, status code, and service is usually more actionable because those dimensions help identify meaningful patterns.
More Difficult Troubleshooting
High-cardinality data can also make telemetry harder to interpret.
If a dashboard contains a separate line for every user or request, common trends become difficult to see. Developers may spend more time filtering the data than diagnosing the underlying problem.
The objective should be to preserve useful diagnostic information while controlling the number of distinct metric dimensions.
Identify High-Cardinality Attributes
Before changing instrumentation, determine which attributes are responsible for excessive data growth.
Start by examining the metrics, traces, and logs generated by the application.
Review Metric Dimensions
Look for attributes that contain values expected to change frequently.
Common examples include:
Request IDs and trace IDs
User IDs and session IDs
Email addresses
Complete URLs with query parameters
Raw search terms
File names containing unique identifiers
Unbounded customer or tenant identifiers
Exception messages containing dynamic values
Not every attribute in this list is always inappropriate. A trace ID is useful for correlating a particular request across distributed services. The problem arises when such values are used as metric dimensions or indexed indiscriminately.
For metrics, prefer attributes with a bounded set of meaningful values.
Inspect Attribute Usage
Suppose a service records the following attributes for every request:
http.request.method = GET
http.route = /orders/{orderId}
http.response.status_code = 200
request.id = 7d2c91
customer.id = 59318
The route template is generally more useful than the raw URL because it groups requests by endpoint.
For example, these URLs:
/orders/1001
/orders/1002
/orders/1003
can all be represented by:
http.route = /orders/{orderId}
This reduces unnecessary variation while preserving the endpoint information required for performance analysis.
The request ID can still be retained in an individual trace or log when appropriate. It should not automatically be included in a metric dimension.
Compare Telemetry Volume Before and After Changes
Establish a baseline before modifying instrumentation.
Record relevant measurements, such as:
After making a change, compare the results under a similar workload.
Without a baseline, a reduction in telemetry may simply reflect fewer application requests rather than more efficient instrumentation.
Reduce Cardinality in Metrics
The most effective starting point is often to review the attributes attached to metric measurements.
Consider a simplified .NET example using OpenTelemetry metrics.
The following example deliberately demonstrates a problematic pattern:
using System.Diagnostics.Metrics;
var meter = new Meter("OrderService");
var requestCounter = meter.CreateCounter<long>(
"orders.requests");
void RecordRequest(string requestId, string status)
{
requestCounter.Add(
1,
new KeyValuePair<string, object?>(
"request.id", requestId),
new KeyValuePair<string, object?>(
"status", status));
}
Every unique request ID creates a new attribute value. Depending on the exporter and backend, this can significantly increase metric cardinality.
A better design records bounded attributes:
using System.Diagnostics.Metrics;
var meter = new Meter("OrderService");
var requestCounter = meter.CreateCounter<long>(
"orders.requests");
void RecordRequest(string method, string status)
{
requestCounter.Add(
1,
new KeyValuePair<string, object?>(
"http.request.method", method),
new KeyValuePair<string, object?>(
"http.response.status_code", status));
}
This example uses HTTP method and response status instead of request ID.
In a production application, use established OpenTelemetry HTTP instrumentation where appropriate rather than duplicating its measurements with custom counters. Standard instrumentation provides consistent conventions and reduces the need to maintain custom request-tracking code.
Also ensure that status values are validated and remain within an expected set. A supposedly bounded attribute can still become high-cardinality if arbitrary values are accepted.
Use Route Templates Instead of Raw URLs
Raw URLs can contain identifiers and query parameters that create unnecessary variation.
For example:
/api/orders/1001?include=items
/api/orders/1002?include=items
/api/orders/1003?include=items
If each complete URL becomes a metric attribute, the backend may treat each request as a separate dimension value.
Instead, use the framework's route template when available:
/api/orders/{orderId}
You can retain a small, bounded set of other dimensions, such as HTTP method and response status.
Avoid replacing route templates with arbitrary string truncation. Truncating URLs may still produce many unique values and can make the resulting data misleading.
If a route is unavailable, consider whether the request should be grouped under a bounded fallback value rather than recording its complete URL.
Keep Trace Context Separate from Metric Dimensions
Traces and metrics serve different purposes.
Metrics are useful for understanding aggregate behavior over time. Traces are useful for investigating individual requests and their interactions across services.
A trace ID can help connect spans belonging to one request. However, including trace IDs as metric dimensions is generally inappropriate because they are designed to identify individual traces.
For example, a trace may contain information such as:
service.name = order-service
http.route = /orders/{orderId}
http.response.status_code = 500
The trace can also have its own trace identifier, which allows developers to investigate the specific failed request.
This design preserves request-level diagnostic information without requiring a separate metric series for every trace.
The same principle applies to user IDs and session IDs. If they are necessary for an authorized investigation, store them in an appropriate signal with suitable access controls, retention rules, and privacy protections rather than adding them indiscriminately to metric dimensions.
Filter Unnecessary Attributes with OpenTelemetry
Sometimes attributes are introduced by automatic instrumentation, third-party libraries, or shared application components. Removing them from every instrumentation call may not be practical.
OpenTelemetry provides processing and filtering mechanisms, but their availability and configuration depend on the language SDK, instrumentation library, exporter, and collector.
For example, a Collector configuration can use a transform processor to remove selected attributes from metric datapoints.
A simplified configuration might look like this:
processors:
transform/remove_metric_attributes:
metric_statements:
- context: datapoint
statements:
- delete_key(attributes, "request.id")
- delete_key(attributes, "customer.id")
The relevant processor must be included in the Collector pipeline:
service:
pipelines:
metrics:
receivers: [otlp]
processors: [transform/remove_metric_attributes, batch]
exporters: [otlp]
This example assumes the OpenTelemetry Collector distribution includes the required transform processor and that the OTLP receiver and exporter are configured for your environment.
The exact configuration should be validated against the Collector version you deploy. Attribute removal at the Collector can reduce downstream cardinality and storage, but it does not necessarily reduce the telemetry generated or transmitted by the application before it reaches the Collector.
For the greatest efficiency, avoid generating unnecessary metric dimensions in the application whenever possible. Use Collector-side processing as an additional control.
Choose the Right Place to Control Cardinality
There are several places to manage attributes, and each has different advantages.
Approach | Advantages | Limitations |
|---|
Application instrumentation | Prevents unnecessary attributes from being generated at the source. | May require code changes and redeployment. |
OpenTelemetry SDK configuration | Can apply consistent filtering to supported instrumentation. | Available options vary by SDK and instrumentation package. |
OpenTelemetry Collector | Centralizes filtering across multiple services. | Data may already have been generated and transmitted. |
Observability backend | Can restrict indexing, aggregation, or retention. | Backend-specific controls may not prevent upstream ingestion costs. |
Sampling | Can reduce the volume of exported traces. | Does not solve high-cardinality metric dimensions by itself. |
Sampling deserves particular attention. Trace sampling can reduce the number of traces exported, depending on the sampling strategy. However, sampling is not a general replacement for controlling metric cardinality.
Similarly, deleting attributes from logs or spans may reduce payload size or indexing overhead, but it will not necessarily address high-cardinality metric series if the same attributes remain on metrics.
Apply the control to the signal and processing stage responsible for the cost.
Common Mistakes to Avoid
Removing Every Attribute
Removing all attributes may reduce telemetry costs, but it can also make the data difficult to use.
Without route, service, status, or environment information, developers may struggle to identify which part of an application is failing.
Keep attributes that provide meaningful diagnostic value and have manageable cardinality.
Moving High-Cardinality Values to Another Metric
Renaming a metric does not solve the problem if it still contains the same unbounded dimensions.
Review the full set of attributes rather than focusing only on metric names.
Assuming Every Unique Value Is Harmful
Unique identifiers are often appropriate in traces and logs. The key question is whether the identifier belongs in the signal and whether the backend uses it as a metric dimension or indexed field.
Evaluate the complete telemetry design instead of applying a blanket rule to every attribute.
Filtering Only at the Backend
Backend filtering may reduce storage or indexing, depending on the platform. However, the application may still generate and transmit the data.
Where possible, eliminate unnecessary dimensions at the source and use downstream filtering as a secondary safeguard.
Ignoring Privacy
Attributes such as email addresses, session tokens, customer identifiers, and raw request parameters can expose sensitive information.
Cardinality reduction is an opportunity to review whether each attribute is necessary, whether it contains personal or confidential data, and whether access and retention policies are appropriate.
Best Practices for Production
Use the following practices to keep telemetry useful and cost-effective:
Prefer bounded metric attributes such as HTTP method, status code, service name, and route template.
Avoid request IDs, trace IDs, session IDs, and other unique values as metric dimensions.
Use trace context to investigate individual requests.
Avoid recording complete URLs when route templates provide sufficient information.
Review custom metrics before deploying them to production.
Measure active series and ingestion volume to establish a baseline.
Use SDK and Collector filtering where appropriate.
Validate Collector configuration before deploying changes.
Monitor the impact of filtering on dashboards and alerts.
Apply suitable access controls and retention policies to traces and logs.
Reassess cardinality when adding new instrumentation or application features.
These practices help prevent telemetry costs from growing faster than application traffic while preserving the information developers need to investigate production issues.
Advantages and Limitations of Cardinality Reduction
Aspect | Advantage | Limitation |
|---|
Cost control | Can reduce metric series, ingestion, and storage costs. | Savings depend on backend pricing and implementation. |
Query performance | Fewer dimensions can simplify queries and dashboards. | Other factors, such as retention and query design, also affect performance. |
Troubleshooting | Bounded dimensions make aggregate patterns easier to identify. | Removing useful attributes can make specific incidents harder to investigate. |
Privacy | Reduces unnecessary exposure of identifiers and sensitive values. | Other telemetry fields may still contain sensitive information. |
Operational consistency | Shared instrumentation and Collector rules make policies easier to maintain. | Configuration must be tested across SDKs, exporters, and backend behavior. |
Summary
High-cardinality attributes can increase OpenTelemetry costs by creating excessive metric time series, increasing telemetry volume, and making queries more expensive. Common causes include request IDs, user IDs, session IDs, raw URLs, and other attributes with a large number of unique values.
Start by identifying which attributes contribute to cardinality and measuring the current telemetry volume. Then remove unnecessary metric dimensions, use route templates instead of raw URLs, and keep individual request identifiers in traces or appropriately controlled logs when they are needed.
Application instrumentation is usually the best place to prevent unnecessary dimensions from being generated. SDK configuration, Collector processing, and backend controls can provide additional safeguards.
The goal is not to collect as little telemetry as possible. It is to collect telemetry that answers meaningful operational questions without creating unnecessary cost, complexity, or privacy risk.
Join the conversation! Your thoughts help the community grow.