Introduction
EC2 Auto Scaling is commonly configured around metrics such as CPU utilization, network traffic, or request counts. These metrics are useful, but they do not always describe what is actually happening inside an application. For example, a background worker may use only 25 percent CPU while hundreds of jobs are waiting in a queue. From the application's point of view, the system is already under pressure even though the CPU graph looks healthy.
Prometheus solves this problem by allowing applications and infrastructure components to expose metrics that describe their real workload. PromQL, the Prometheus Query Language, can then turn those raw metrics into useful measurements. Instead of scaling an EC2 Auto Scaling group simply because CPU crosses a threshold, you can use a PromQL expression that represents the workload your application actually needs to handle.
PromQL is a query language designed for selecting, filtering, aggregating, and calculating time-series metrics stored in Prometheus. A time-series metric is a value recorded over time, such as the number of jobs waiting in a queue, requests processed per second, active workers, or the number of failed operations. PromQL lets you combine those values into a result that is more useful for operational decisions.
The important part is that Auto Scaling does not directly understand PromQL. Prometheus evaluates the PromQL expression, and the resulting value must be made available to the AWS scaling system through a metric that EC2 Auto Scaling can evaluate. This creates a bridge between application-level workload information and infrastructure scaling.
This approach is especially useful when CPU is only an indirect signal. If your application has a queue, job processor, API workload, message consumer, or another measurable workload, scaling from that workload can make the Auto Scaling group respond to actual demand instead of simply responding to resource consumption.
What PromQL Means
PromQL stands for Prometheus Query Language. It is the query language used by Prometheus to work with time-series data.
A simple PromQL expression can select a metric:
http_requests_total
This expression asks Prometheus for the time-series data associated with the http_requests_total metric.
The problem is that raw metrics are often not directly useful for scaling. A counter such as http_requests_total normally continues increasing as requests are processed. Auto Scaling usually needs a value that describes current activity, such as requests per second.
PromQL can calculate that value:
rate(http_requests_total[5m])
The rate function calculates the average per-second increase of a counter over the previous five minutes.
This distinction is important. A metric tells you what was recorded, while a PromQL expression can turn that recorded information into a measurement that is meaningful for a particular operational decision.
For example, you may have thousands of request counters across multiple EC2 instances. Scaling should generally not be based on one instance's value. Instead, you may want the total request rate across the entire Auto Scaling group:
sum(rate(http_requests_total[5m]))
This produces a single value representing the request rate across the matching time series.
Why Use PromQL for Auto Scaling
The main reason to use PromQL for scaling is that applications often know more about workload pressure than infrastructure metrics do.
Consider a .NET background processing application. Each EC2 instance runs a worker that consumes jobs from a queue. The instances may remain around 30 percent CPU while the queue continues growing because the workers are waiting on external services or processing I/O-heavy workloads.
If the scaling policy only looks at CPU, AWS may decide that no additional instances are required.
A workload-based metric can provide a better signal.
For example:
sum(queue_jobs_waiting)
This expression represents the number of waiting jobs across the monitored workers.
If the result becomes large enough, the scaling system can increase the number of EC2 instances. When the queue returns to a normal level, the scaling policy can eventually reduce capacity.
This creates a more direct relationship between application demand and infrastructure capacity.
The Basic Architecture
The complete flow can be understood as a sequence of steps.
First, the application exposes metrics. These metrics may represent HTTP requests, queue depth, active jobs, processing latency, or another workload measurement.
Second, Prometheus collects those metrics and stores them as time-series data.
Third, PromQL evaluates the metrics and produces a calculated value.
Fourth, the calculated value is exposed through the AWS monitoring path used by the scaling policy.
Finally, EC2 Auto Scaling evaluates that metric and decides whether additional capacity is required.
The important architectural point is that PromQL is responsible for calculating the workload signal, while EC2 Auto Scaling remains responsible for changing the number of instances.
This separation is useful because application developers can define the workload metric without changing the basic scaling mechanism.
Choosing the Right Metric
The most important part of PromQL-based scaling is not writing the query. It is choosing a metric that actually represents workload pressure.
A good scaling metric should have a clear relationship with capacity.
For example, these metrics can often be useful:
Metric | What it represents | Scaling usefulness |
|---|
Queue depth | Jobs waiting for processing | High for worker systems |
Requests per second | Incoming application traffic | High for API workloads |
Active jobs | Work currently being processed | High for background workers |
Processing latency | Time required to complete work | Useful when latency indicates overload |
Messages behind | Unprocessed messaging workload | High for message consumers |
CPU utilization | Compute consumption | Useful for CPU-bound workloads |
Memory utilization | Memory pressure | Useful for memory-heavy workloads |
The important question is not whether a metric is available. The question is whether an increase in that metric means the application needs more capacity.
For example, CPU utilization is usually a reasonable metric for a CPU-bound image-processing application. For a database-heavy application, CPU may remain relatively low while database wait time and request latency increase. In that situation, CPU may not be the best scaling signal.
Building a Simple PromQL Scaling Expression
Suppose every worker exposes the number of jobs currently waiting:
worker_queue_depth
If every EC2 instance exposes its own queue depth, Prometheus will have multiple time series.
A basic query can combine them:
sum(worker_queue_depth)
The result is the total queue depth.
Suppose the instances report:
Instance A = 40 jobs
Instance B = 35 jobs
Instance C = 50 jobs
The PromQL result would be:
125
That single value can become the workload signal used by the scaling system.
This is more useful than sending each instance's queue depth separately because Auto Scaling generally needs a metric that describes the capacity requirement of the group rather than the condition of one individual machine.
Filtering Metrics With Labels
Prometheus metrics commonly contain labels. Labels provide additional information about where or how a metric was generated.
For example, a metric might contain labels such as:
worker_queue_depth{environment="production",service="orders"}
This allows PromQL to select only the production order-processing workers.
For example:
sum(worker_queue_depth{environment="production",service="orders"})
This prevents development or testing workloads from affecting production scaling.
Labels become particularly important when the same Prometheus system monitors several applications. Without filtering, a scaling query can accidentally combine unrelated workloads.
A good scaling query should therefore be specific about the environment, service, workload, and other dimensions that matter.
Scaling From Request Rate
For an API service, request rate is often a better signal than total request count.
Suppose the application exposes:
http_requests_total
Because this is a counter, the value increases over time. A raw counter does not tell Auto Scaling how quickly requests are arriving.
PromQL can calculate the rate:
sum(rate(http_requests_total[5m]))
The five-minute window smooths short spikes and provides a more stable measurement.
You can also filter by service:
sum(rate(http_requests_total{service="orders"}[5m]))
This gives the request rate for the order service rather than the entire application environment.
The correct time window depends on the workload. A very short window can react quickly but may produce noisy values. A longer window produces a smoother signal but can make scaling slower to respond to sudden changes.
Scaling From Queue Depth
Queue-based scaling is one of the clearest use cases for PromQL.
Imagine an application processing invoice-generation jobs. The application exposes:
invoice_jobs_waiting
A PromQL expression can calculate the total waiting workload:
sum(invoice_jobs_waiting{environment="production"})
If the value is 20, the system may have enough capacity.
If it grows to 500, workers may be falling behind.
If it reaches several thousand, the application may require additional capacity.
The important advantage is that the metric directly represents unfinished work. This makes it easier to reason about scaling than an indirect metric such as CPU.
Creating a Per-Instance Workload Signal
Sometimes total workload is not the right value.
Suppose a system has 10 EC2 instances and 1,000 jobs waiting. The total queue depth is 1,000, but the scaling decision may need to understand how much work each instance is expected to handle.
PromQL can calculate an average:
avg(worker_queue_depth)
This produces the average queue depth across the matching instances.
Another useful approach is to calculate workload per active worker:
sum(worker_queue_depth) / count(worker_queue_depth)
This can provide a workload-per-worker value.
The correct expression depends on how the application distributes work. A total queue metric may work well when all instances share the same queue. A per-worker metric may be more appropriate when each worker has its own workload.
Avoiding Double Counting
One common problem with PromQL scaling queries is accidentally counting the same workload more than once.
For example, an application might expose both:
jobs_received_total
and:
jobs_processed_total
These are related metrics, but adding them together does not produce a meaningful scaling signal.
Another example occurs when multiple exporters expose the same underlying workload. A PromQL query may accidentally select both copies.
The safest approach is to understand what each metric represents before including it in a scaling expression.
A scaling query should ideally produce one clearly defined measurement with one clear interpretation.
Handling Counters Correctly
Counters require special attention.
A counter represents a value that normally increases over time. Examples include total HTTP requests, total jobs processed, and total errors.
You generally should not use the raw counter as the scaling metric:
http_requests_total
Instead, use functions such as rate or increase when appropriate.
For requests per second:
sum(rate(http_requests_total[5m]))
For the number of requests during a period:
sum(increase(http_requests_total[5m]))
Understanding this difference is important because a raw counter can grow indefinitely even when traffic has already dropped.
Making the Query Stable
A scaling system should not continuously change capacity because the metric moves slightly above and below a threshold.
For example, imagine a scaling threshold of 100 requests per second.
If the observed values move like this:
98
103
97
105
99
102
a poorly designed scaling policy could repeatedly scale out and scale in.
PromQL can help produce a more stable signal by using aggregation and appropriate time windows.
For example:
sum(rate(http_requests_total[5m]))
Using a five-minute rate means the scaling signal is based on a recent period instead of a single instant.
The Auto Scaling policy should also use appropriate cooldown or stabilization behavior so that the group has enough time to react before another scaling decision is made.
Missing Metrics and Application Failures
A production scaling system must also consider what happens when the application stops reporting metrics.
This is different from a metric becoming zero.
A zero value usually means the metric was successfully collected and its value is zero.
A missing value means the monitoring system may not have received a valid sample.
These situations should not automatically be treated as the same thing.
Suppose a worker crashes and stops exporting queue metrics. If the scaling system interprets the missing metric as zero workload, it may incorrectly scale in the environment.
That can make an existing failure worse.
Scaling policies should therefore have a clear strategy for missing, delayed, or invalid metric data.
Testing the PromQL Expression Before Scaling
Do not immediately connect a new PromQL query to production Auto Scaling.
First, run the query manually and inspect its result.
For example:
sum(rate(http_requests_total{service="orders",environment="production"}[5m]))
Check whether the result changes when traffic changes.
Then test several conditions:
Normal traffic
High traffic
Sudden traffic spike
Traffic drop
No traffic
Missing exporter
One instance unavailable
Several instances unavailable
The objective is to confirm that the metric behaves as expected before allowing it to change infrastructure capacity.
This testing step is especially important because a syntactically valid PromQL query can still be logically wrong for scaling.
Example .NET Worker Metric
A .NET background worker can expose a queue-related metric that Prometheus collects.
The application might maintain a metric representing the number of jobs currently waiting.
Conceptually, the application flow looks like this:
Job arrives
|
v
Queue depth increases
|
v
Prometheus collects metric
|
v
PromQL calculates workload
|
v
Scaling metric is evaluated
|
v
EC2 Auto Scaling adjusts capacity
The important part is that the application does not need to know that Auto Scaling is using the metric.
The worker only reports accurate information about its workload. The monitoring and infrastructure layers decide what to do with that information.
This separation keeps the application simpler and allows the same metric to be used for dashboards, alerts, troubleshooting, and scaling.
Target Tracking and PromQL
Target tracking works by attempting to keep a selected metric around a target value.
For example, you may define a workload metric where the desired target is 50 queued jobs per instance.
The idea is:
Workload below target
|
v
Capacity may be reduced
Workload near target
|
v
Capacity remains stable
Workload above target
|
v
Additional capacity may be added
The important part is choosing a target that has operational meaning.
Do not choose 50 simply because the number looks reasonable.
Instead, test the application and determine how much workload one instance can normally process while maintaining acceptable response time, queue delay, or processing latency.
Choosing a Scaling Threshold
The correct threshold should come from application behavior rather than guesswork.
Suppose testing shows that one worker can safely process 100 jobs while keeping the average processing delay below your acceptable limit.
You could use that information to create a scaling target around the workload capacity of an instance.
This is much more useful than choosing an arbitrary threshold such as 70 percent CPU if CPU does not correlate with processing capacity.
The threshold should also leave enough headroom for normal traffic variation. If the application is already at its maximum sustainable capacity when the scaling policy starts adding instances, users may experience delays before the new instances become ready.
Scale-Out and Scale-In Are Different Problems
Scaling out and scaling in should not necessarily use the same behavior.
Scale-out needs to respond quickly enough to protect the application from growing workload.
Scale-in should usually be more conservative because removing capacity too quickly can create another workload spike.
For example:
Queue grows quickly
-> scale out
Queue remains high
-> keep additional capacity
Queue decreases
-> wait and observe
Queue remains low
-> scale in gradually
This approach reduces oscillation.
Oscillation occurs when the Auto Scaling group repeatedly adds and removes instances because the scaling signal changes around the configured threshold.
Common PromQL Mistakes
Using the Raw Counter
Using a raw counter such as http_requests_total as the scaling metric can produce misleading results because the value keeps increasing.
Use rate or another appropriate calculation instead.
Ignoring Labels
If the PromQL query does not filter environment or service labels, unrelated workloads can influence the scaling decision.
Always verify which time series the query actually selects.
Using an Extremely Short Window
A one-minute or even shorter rate may react strongly to temporary traffic spikes.
This can cause unnecessary scaling.
Test different windows and select one that matches the application's traffic pattern.
Scaling From CPU When the Workload Is I/O Bound
An I/O-bound application may have low CPU utilization while requests or jobs continue waiting.
In this situation, CPU may hide the real capacity problem.
Use a metric that represents the actual bottleneck.
Creating a Query Nobody Can Explain
A complicated PromQL expression may be technically correct but operationally difficult to maintain.
Every production scaling query should have a clear explanation.
An engineer responding to an incident should be able to answer:
What does this metric measure?
Why does it cause scale-out?
Why does it cause scale-in?
What happens when the metric disappears?
If those questions cannot be answered easily, the scaling design needs more work.
Troubleshooting PromQL-Based Scaling
When Auto Scaling does not behave as expected, start with the PromQL result rather than immediately changing the Auto Scaling configuration.
First, run the query directly and verify that it returns data.
Next, check whether the query includes the expected labels.
Then verify the time range and aggregation.
After that, compare the PromQL result with the actual application workload.
For example, if users report heavy traffic but the PromQL result remains near zero, the problem is probably in metric collection, labeling, aggregation, or the query itself.
If the PromQL result is correct but Auto Scaling does not react, investigate the AWS metric publishing and scaling policy configuration.
This separation makes troubleshooting much easier because it divides the system into clear layers.
PromQL Scaling vs CPU Scaling
Area | CPU-based scaling | PromQL workload scaling |
|---|
Setup | Usually simpler | Requires metric and query design |
Application awareness | Low | High |
Queue workloads | Often indirect | Very useful |
API workloads | Sometimes useful | Can use request rate |
I/O-heavy workloads | Can be misleading | Can use workload-specific metrics |
Maintenance | Lower | Higher |
Flexibility | Limited to available metrics | Very flexible |
Debugging | Usually straightforward | Requires monitoring the full metric path |
CPU-based scaling remains a good option for many workloads. PromQL-based scaling is not intended to replace it everywhere.
The main advantage of PromQL is that it allows scaling decisions to be based on application behavior when standard infrastructure metrics do not describe the real workload.
Production Checklist
Before using a PromQL-derived metric for production EC2 Auto Scaling, verify the following:
The metric represents real application workload.
The PromQL expression has been tested against normal and peak traffic.
Labels correctly isolate the intended environment and service.
Counters are converted into rates or another meaningful measurement.
The query does not double-count workload.
Missing metrics are handled safely.
The scaling threshold has been validated through load testing.
Scale-out happens early enough to allow new instances to start.
Scale-in is conservative enough to avoid oscillation.
New instances begin reporting metrics correctly.
The scaling metric is monitored independently.
Alerts exist for missing or abnormal metric values.
Engineers can explain the scaling query during an incident.
When PromQL Is a Good Choice
PromQL is a strong choice when the application already exposes useful Prometheus metrics and those metrics describe workload more accurately than standard infrastructure metrics.
It is particularly useful for worker systems, queue consumers, API services, event processing systems, and workloads where business-level activity determines capacity.
It is less attractive when CPU or another standard CloudWatch metric already provides a reliable scaling signal and the additional monitoring pipeline would add unnecessary complexity.
The goal should not be to use PromQL simply because it is available. The goal is to choose the metric that best represents the capacity problem you are trying to solve.
Advantages and Disadvantages
Advantages
Scaling can follow real application workload.
PromQL supports complex aggregation and filtering.
Queue and worker workloads can be measured directly.
Existing Prometheus metrics can serve multiple operational purposes.
Different applications can use different workload definitions.
Scaling logic can evolve without changing the application architecture.
Disadvantages
The setup is more complicated than basic CPU scaling.
The metric pipeline itself becomes part of the scaling system.
Poor PromQL can cause incorrect scaling decisions.
Missing metrics require careful handling.
Teams need enough Prometheus and PromQL knowledge to troubleshoot the system.
Scaling behavior must be tested under realistic workloads.
Summary
Using PromQL to trigger EC2 Auto Scaling allows infrastructure capacity to respond to application workload instead of relying only on CPU, memory, or other generic infrastructure measurements.
The key idea is simple: the application exposes useful metrics, Prometheus collects them, PromQL converts those metrics into a meaningful workload signal, and the resulting metric becomes part of the Auto Scaling decision.
The hardest part is usually not writing the PromQL expression. It is defining what workload actually means for the application. A queue depth metric may be ideal for a background worker, while request rate may be better for an API. A CPU metric may still be the right choice for a compute-heavy workload.
A production scaling query should therefore be understandable, measurable, tested, and directly connected to application capacity. When those conditions are met, PromQL can provide a much more accurate scaling signal than simply watching whether EC2 instances are busy.
Join the conversation! Your thoughts help the community grow.