Kubernetes 1.37 Native Histograms: Improving Prometheus Latency Metrics Without Breaking Existing Dashboards

Observability becomes increasingly expensive as Kubernetes clusters grow.

A small cluster can collect thousands of Prometheus time series without much difficulty. At larger scale, however, histogram metrics can become a significant source of storage, memory, and query overhead. This is particularly relevant for latency metrics, where teams need detailed distributions rather than simple averages.

Kubernetes 1.37 addresses this problem by graduating native histogram support to Beta and enabling it by default. Native histograms were introduced as an Alpha feature in Kubernetes 1.36. The Kubernetes components can now expose histogram data using Prometheus Native Histogram format while retaining compatibility with classic histograms.

This creates an important migration opportunity for teams using Prometheus and Grafana:

You can begin using native histograms without immediately rewriting every existing dashboard and alert.

This article explains how native histograms work, why they improve latency observability, how Kubernetes 1.37 exposes them, how to configure Prometheus, and how to migrate existing dashboards safely.

What Is a Prometheus Histogram?

A histogram records observations such as request durations and groups them into ranges.

For example, an application might measure HTTP request latency using buckets such as:

0.005 seconds
0.010 seconds
0.025 seconds
0.050 seconds
0.100 seconds
0.250 seconds
0.500 seconds
1.000 seconds
2.500 seconds
5.000 seconds
10.000 seconds

A classic Prometheus histogram creates multiple time series for these buckets.

A simplified metric might look like:

http_request_duration_seconds_bucket{le="0.1"} 1200
http_request_duration_seconds_bucket{le="0.5"} 4800
http_request_duration_seconds_bucket{le="1"} 5200
http_request_duration_seconds_bucket{le="+Inf"} 5300

It also exposes count and sum series:

http_request_duration_seconds_count 5300
http_request_duration_seconds_sum 1420.7

This approach is well established, but it has an important limitation.

You have to decide the bucket boundaries before collecting the data.

Why Static Histogram Buckets Are a Problem

Suppose you define these buckets:

10ms
50ms
100ms
500ms
1s
5s

They might work well for an API whose normal latency is between 10ms and 500ms.

But what happens when the workload changes?

Imagine the service later handles requests ranging from 1 microsecond to 20 seconds.

The original buckets may no longer provide useful resolution.

A request completing in 100 microseconds and another completing in 40 milliseconds could end up represented in the same broad bucket.

This makes percentile calculations less precise.

Static buckets also create additional time series because every bucket is represented separately. Kubernetes documentation notes that classic histograms can create substantial storage overhead and that native histograms can reduce the number of time series associated with a histogram metric by roughly an order of magnitude.

What Are Native Histograms?

Prometheus Native Histograms use dynamic exponential bucket boundaries rather than a fixed list of manually selected buckets.

Instead of exporting a separate time series for every bucket, a native histogram stores the distribution using a compact histogram representation containing information about bucket spans and scaling.

Conceptually:

Classic Histogram

Metric
 ├── bucket 0.005
 ├── bucket 0.010
 ├── bucket 0.025
 ├── bucket 0.050
 ├── bucket 0.100
 ├── bucket 0.500
 ├── bucket 1.000
 └── bucket +Inf


Native Histogram

Metric
 └── Dynamic Exponential Distribution
       ├── Sparse Spans
       ├── Zero Threshold
       └── Scaling Schema

This allows the histogram to provide resolution across a much wider range without requiring developers to manually predict every useful latency boundary.

Why Kubernetes 1.37 Matters

Kubernetes 1.37 graduates native histogram support to Beta.

The NativeHistograms feature is enabled by default in Kubernetes 1.37. Kubernetes components can expose histogram metrics in both classic and native formats when the requested scrape protocol supports native histograms.

This affects several core components, including:

kube-apiserver
kube-scheduler
kube-controller-manager
kubelet
kube-proxy

The shared Kubernetes metrics infrastructure allows histogram support to be applied consistently across these components.

This is especially useful for latency metrics such as:

apiserver_request_duration_seconds

and scheduler-related duration metrics.

Kubernetes Uses Dual Exposition

One of the most important parts of the Kubernetes 1.37 implementation is backward compatibility.

Kubernetes can expose both:

Classic Histogram
+
Native Histogram

depending on the scrape protocol and Prometheus configuration.

This means existing monitoring systems do not have to migrate everything immediately.

The architecture looks like this:

                    Kubernetes Component
                           |
                           v
                    Metrics Subsystem
                           |
              +------------+------------+
              |                         |
              v                         v
       Classic Histogram         Native Histogram
              |                         |
              v                         v
       Existing Queries          New PromQL Queries

The Kubernetes documentation describes this as dual exposition, allowing existing dashboards and alerting rules to continue working while teams migrate to native histogram queries.

Why Dual Exposition Is Important

Imagine a production Grafana dashboard containing:

histogram_quantile(
  0.99,
  rate(apiserver_request_duration_seconds_bucket[5m])
)

If the organization immediately removed all classic histogram data, that query could stop returning data.

Dual exposition provides a migration period.

You can collect:

Native Histogram
+
Classic Histogram

and gradually update your dashboards.

This is much safer than requiring a single large migration.

Kubernetes 1.37 Native Histogram Configuration

For Kubernetes 1.37, native histogram support is enabled by default.

The remaining question is whether your Prometheus installation is configured to ingest native histograms.

For Prometheus 3.x, a scrape job can be configured like this:

scrape_configs:
  - job_name: kubernetes-apiservers

    scrape_native_histograms: true

    always_scrape_classic_histograms: true

The first setting enables native histogram ingestion.

The second setting keeps classic histograms during the migration period.

Kubernetes recommends retaining classic histograms while existing dashboards and alerts are being migrated.

Prometheus Version Requirements

Native histogram support depends on the Prometheus version.

A practical compatibility view is:

Prometheus Version	Native Histogram Support
Older than 2.40	Not supported
2.40+	Native histogram support available
Prometheus 3.x	Recommended for current per-job configuration
Prometheus 3.9+	Use per-job native histogram configuration

Kubernetes documentation recommends Prometheus 3.x for current deployments because per-job configuration provides better control over native histogram ingestion.

If a monitoring stack still uses a Prometheus version older than 2.40, upgrading Prometheus should be part of the migration plan.

Prometheus Must Support the Native Exposition Protocol

Native histograms require Prometheus to negotiate an appropriate exposition format.

Prometheus uses its supported protobuf-based protocol when native histogram scraping is enabled.

If you have customized the Prometheus scrape protocol configuration, ensure that the Prometheus native histogram protocol remains available.

For example, a customized configuration should not accidentally exclude the required protocol.

The exact configuration depends on the Prometheus version and deployment method.

Classic Histogram Query

A traditional percentile query might look like:

histogram_quantile(
  0.99,
  rate(apiserver_request_duration_seconds_bucket[5m])
)

Notice the _bucket suffix.

Classic histograms require bucket series because the percentile calculation operates on those predefined boundaries.

Native Histogram Query

With native histograms, the query can operate directly on the histogram metric:

histogram_quantile(
  0.99,
  rate(apiserver_request_duration_seconds[5m])
)

The difference is subtle but important.

Classic:

metric_bucket

Native:

metric

The native histogram contains its distribution information directly.

Kubernetes documents both query styles and recommends migrating dashboard and alert expressions gradually.

Aggregating Across Multiple Kubernetes Components

Classic histogram aggregation typically requires preserving the le label.

For example:

histogram_quantile(
  0.99,
  sum by (le) (
    rate(apiserver_request_duration_seconds_bucket[5m])
  )
)

The le label identifies the bucket boundary.

With native histograms, the aggregation is simpler:

histogram_quantile(
  0.99,
  sum(
    rate(apiserver_request_duration_seconds[5m])
  )
)

There is no need to group by le because the native histogram carries its distribution information directly.

Why This Helps Latency Monitoring

Latency distributions are rarely uniform.

Consider an API where:

95% of requests → < 100ms
4%              → 100–500ms
0.9%            → 500ms–2s
0.1%            → > 2s

An average such as:

Average latency = 120ms

does not tell the whole story.

A percentile such as:

P99 = 1.8s

reveals the long tail.

For SLO-based systems, tail latency can be more important than average latency.

Native histograms provide more flexible resolution across the distribution, making them particularly useful for this kind of analysis.

Native Histograms and SLOs

Suppose your API SLO is:

99% of requests must complete within 500ms.

A classic histogram might have a bucket at:

500ms

That can work well if the bucket boundary was selected intentionally.

But if your SLO later changes to:

99.5% < 350ms

the existing bucket configuration may not provide the desired resolution.

Native histograms dynamically represent the distribution, making queries around different thresholds more flexible.

This can simplify observability for systems where SLO requirements evolve.

Kubernetes Metrics That Benefit

Kubernetes exposes many histogram metrics across its components.

Examples include:

apiserver_request_duration_seconds

and:

scheduler_plugin_execution_duration_seconds

The Kubernetes metrics reference contains the histogram metrics exposed by Kubernetes 1.37 components.

These metrics can help operators investigate:

API server latency
Admission latency
Scheduler performance
Authentication and authorization latency
Validation duration
Controller processing
Node and pod lifecycle operations

Storage Efficiency

One of the major motivations for native histograms is reducing the number of time series required to represent distributions.

With a classic histogram:

1 Histogram
   ↓
10 buckets
   ↓
10+ related time series

With a native histogram:

1 Histogram
   ↓
Native distribution
   ↓
1 primary histogram series

The exact storage characteristics depend on the metric and data distribution, but Kubernetes documents an approximately 10x reduction in time-series count per histogram metric as a key benefit.

This can become significant in large clusters.

For example, a platform might expose hundreds of histogram metrics across:

50 API servers
+
100 scheduler/controller components
+
1,000 nodes

Even relatively small savings per histogram can become substantial at scale.

Native Histograms Do Not Mean Zero Resource Usage

It is important not to interpret storage efficiency as zero overhead.

Native histograms still require memory and computation.

Kubernetes uses bounded histogram configuration to control resource usage. The current implementation applies a maximum bucket count of 160 for histogram metrics.

Operators should therefore monitor Prometheus memory after enabling native histogram ingestion.

The correct question is not:

Do native histograms use resources?

They do.

The better question is:

Do native histograms provide better observability per unit of storage and processing cost?

For many large-scale environments, that is the more useful comparison.

Migrating an Existing Grafana Dashboard

Suppose an existing Grafana dashboard contains:

histogram_quantile(
  0.95,
  sum by (le) (
    rate(apiserver_request_duration_seconds_bucket[5m])
  )
)

Do not immediately delete the classic query.

First enable both formats:

scrape_configs:
  - job_name: kubernetes-apiservers
    scrape_native_histograms: true
    always_scrape_classic_histograms: true

Then create a new dashboard panel using:

histogram_quantile(
  0.95,
  sum(
    rate(apiserver_request_duration_seconds[5m])
  )
)

Compare the new panel against the existing one.

You should validate:

P50
P90
P95
P99
Error thresholds
SLO alerts
Time-range behavior
Missing-data behavior

Only after the results are validated should the team consider removing classic histogram ingestion.

Migrating Alert Rules

The same principle applies to Prometheus alerts.

An existing alert might look like:

histogram_quantile(
  0.99,
  sum by (le) (
    rate(apiserver_request_duration_seconds_bucket[5m])
  )
) > 1

A native histogram version can be:

histogram_quantile(
  0.99,
  sum(
    rate(apiserver_request_duration_seconds[5m])
  )
) > 1

Do not assume that changing the query syntax is enough.

Run both versions in a staging environment and compare their behavior during:

Normal traffic
High traffic
Latency spikes
Deployment events
Control-plane load
Failure scenarios

The alert must represent the same operational condition before replacing the production rule.

Migrating Count and Sum Queries

Classic histograms commonly expose:

_metric_count
_metric_sum

Native histograms can use PromQL functions such as:

histogram_count(...)

and:

histogram_sum(...)

For example:

histogram_count(
  rate(apiserver_request_duration_seconds[5m])
)

and:

histogram_sum(
  rate(apiserver_request_duration_seconds[5m])
)

These functions allow teams to work with the native histogram representation without depending on classic _count and _sum series.

The Safest Migration Strategy

A production migration should happen in stages.

Stage 1: Upgrade Kubernetes

Move to Kubernetes 1.37 or later where native histogram support is enabled by default.

Stage 2: Verify Prometheus Compatibility

Ensure the Prometheus version supports native histograms.

Stage 3: Enable Native Histogram Scraping

Configure the relevant Prometheus jobs:

scrape_native_histograms: true
always_scrape_classic_histograms: true

Stage 4: Keep Existing Dashboards

Do not immediately modify production dashboards.

Let the classic queries continue working.

Stage 5: Build Native Queries

Create parallel panels and alerts using native histogram queries.

Stage 6: Validate

Compare the results and verify SLO behavior.

Stage 7: Migrate

Replace classic queries after validation.

Stage 8: Remove Classic Ingestion

Only after all dependencies have been migrated should you consider:

always_scrape_classic_histograms: false

This staged approach minimizes operational risk. Kubernetes specifically recommends keeping classic histogram ingestion during migration.

A Common Migration Failure

One of the easiest mistakes is configuring:

scrape_native_histograms: true
always_scrape_classic_histograms: false

while leaving an old Grafana query such as:

histogram_quantile(
  0.99,
  rate(apiserver_request_duration_seconds_bucket[5m])
)

The dashboard may suddenly show no data because the Prometheus server is ingesting native histogram data while the query still expects classic _bucket series.

Kubernetes documentation explicitly calls out this migration issue.

The safer configuration during migration is:

scrape_native_histograms: true
always_scrape_classic_histograms: true

Troubleshooting: Dashboard Shows No Data

If a dashboard stops showing latency data after native histogram configuration changes, check the query first.

If it contains:

_bucket

it is probably still using the classic format.

Then inspect the Prometheus scrape configuration.

During migration, verify:

scrape_native_histograms: true
always_scrape_classic_histograms: true

If classic ingestion was disabled accidentally, restore it while migrating the dashboard.

Troubleshooting: Prometheus Does Not Understand Native Histograms

If Prometheus reports an unknown or unsupported histogram format, verify the Prometheus version.

Native histogram support requires Prometheus 2.40 or later.

For older Prometheus deployments:

Upgrade Prometheus
        or
Disable Native Histogram Ingestion

Upgrading is generally preferable if the organization wants to take advantage of the newer observability model.

Troubleshooting: Memory Usage Increases

Native histograms can change the memory profile of the Prometheus server.

Monitor Prometheus resource usage after enabling ingestion.

Useful signals include:

process_resident_memory_bytes

and the Prometheus server's normal TSDB and scrape health metrics.

If memory pressure becomes significant, reduce the scope of native histogram ingestion or temporarily disable it for affected jobs.

Kubernetes documents that its histogram configuration limits the number of buckets to control memory usage.

Troubleshooting: Native Histograms Are Not Being Exposed

First verify the Kubernetes feature gate:

kubernetes_feature_enabled{
  name="NativeHistograms"
}

A value of:

indicates that the feature is enabled for the relevant component.

You can also inspect the component's /metrics endpoint using the appropriate Prometheus protobuf Accept header.

This is mainly useful for troubleshooting rather than routine monitoring.

Classic vs. Native Histograms

Feature	Classic Histogram	Native Histogram
Bucket boundaries	Static	Exponential/dynamic
Bucket configuration	Manual	Automatically structured
Time-series count	Higher	Lower
Quantile resolution	Depends on bucket design	More consistent across ranges
Existing dashboard compatibility	Excellent	Requires native query support
Migration effort	None for existing systems	Requires query migration
Storage efficiency	Lower	Higher
Best use case	Mature existing dashboards	High-scale, high-resolution observability

Neither format needs to be treated as universally superior.

Classic histograms remain useful and are still supported.

Native histograms are an additional representation that addresses several limitations of fixed buckets.

Advantages of Kubernetes 1.37 Native Histograms

Better Latency Resolution

Dynamic exponential buckets provide more useful resolution across a broad range of observations.

Lower Time-Series Overhead

Native histograms can substantially reduce the number of time series required for histogram metrics.

Easier Quantile Queries

Native PromQL queries operate directly on histogram metrics.

Better Long-Tail Visibility

Latency distributions can be represented more effectively without requiring a large manually designed bucket list.

Gradual Migration

Dual exposition allows existing dashboards and alerts to continue working during migration.

Kubernetes-Wide Integration

The feature is implemented in the common metrics infrastructure and is available across major Kubernetes components.

Disadvantages and Considerations

Prometheus Compatibility

Older Prometheus versions cannot ingest native histograms.

Dashboard Migration

Existing _bucket queries need to be migrated if teams want to stop ingesting classic histograms.

Operational Complexity

Running both formats temporarily increases observability configuration complexity.

Resource Consumption

Native histograms still consume memory and processing resources.

Tooling Compatibility

Every component between Kubernetes and the final visualization or storage system must correctly support native histogram data.

Migration Risk

Removing classic histograms too early can break dashboards and alerts.

Best Practices

Keep Classic Histograms During Migration

Use:

scrape_native_histograms: true
always_scrape_classic_histograms: true

until existing queries have been migrated.

Migrate Queries Incrementally

Do not rewrite hundreds of dashboards in one deployment.

Start with a small set of high-value latency dashboards.

Test Alerts Separately

A dashboard can look correct while an alert expression still has an issue.

Validate SLOs

Compare percentile behavior around actual service-level objectives.

Monitor Prometheus Resources

Observe memory, ingestion, query latency, and storage behavior after enabling native histogram ingestion.

Upgrade Prometheus Before Kubernetes Migration

Kubernetes 1.37 can expose native histograms, but your monitoring system must be capable of consuming them.

Maintain a Rollback Plan

If native histogram ingestion causes unexpected issues, Prometheus 3.x can disable native histogram scraping per job without restarting the Kubernetes component.

When Should You Keep Classic Histograms?

Not every organization needs to immediately eliminate classic histograms.

Keeping classic histograms can make sense when:

Existing dashboards are extensive.
Third-party monitoring systems do not yet support native histograms.
The organization is still upgrading Prometheus.
Existing alerts have not been validated against native queries.
Storage is not currently a concern.
The team wants to migrate gradually.

Kubernetes 1.37's dual-exposition design allows this transition.

When Should You Adopt Native Histograms?

Native histograms become especially attractive when:

The cluster has many histogram metrics.
Prometheus storage is growing quickly.
Tail latency matters.
Teams need more flexible percentile analysis.
SLO thresholds change frequently.
Existing bucket definitions are too coarse.
High-cardinality observability is becoming expensive.

The larger the monitoring environment, the more valuable efficient histogram representation can become.

A Production Migration Checklist

Before enabling native histogram ingestion across production, verify:

[ ] Kubernetes is running 1.37+
[ ] Prometheus supports native histograms
[ ] Prometheus scrape configuration is updated
[ ] Native histogram ingestion works in staging
[ ] Classic histogram ingestion remains enabled
[ ] Existing dashboards still return data
[ ] Native queries have been tested
[ ] SLO queries have been validated
[ ] Alert rules have been tested
[ ] Grafana panels have been migrated where appropriate
[ ] Prometheus memory usage is monitored
[ ] Rollback configuration is documented

Only after these checks pass should you consider removing classic histogram ingestion.

Conclusion

Kubernetes 1.37 makes native histograms a practical next step for Prometheus-based observability.

The most important improvement is not simply that Kubernetes can expose another metric format. Native histograms address fundamental limitations of classic histograms by using dynamically structured exponential buckets, reducing time-series overhead and providing more flexible visibility into latency distributions.

The migration does not have to be disruptive.

A safe approach is:

Kubernetes 1.37
      |
      v
Enable Native Histogram Scraping
      |
      +-------------------+
      |                   |
      v                   v
Classic Histograms    Native Histograms
      |                   |
      v                   v
Existing Dashboards    New Queries
      |                   |
      +---------+---------+
                |
                v
          Validate Alerts
                |
                v
       Migrate Dashboards
                |
                v
      Disable Classic Format
        When Ready

For teams operating large Kubernetes environments, native histograms can provide a better balance between latency accuracy, query flexibility, and observability storage efficiency.

The key is to treat Kubernetes 1.37 as the beginning of a controlled observability migration rather than a reason to immediately replace every existing Prometheus query.