A new application release can pass automated tests and still introduce problems in production. A database query may become slower under real traffic, an API dependency may fail under load, or a configuration change may increase the error rate.
Deploying the new version to every user at once makes these problems harder to contain. A canary release reduces that risk by sending a small portion of production traffic to the new version before increasing its exposure.
However, a canary release is only useful if the deployment process can detect problems and respond to them.
If the new version starts returning errors but the rollout continues automatically, more users may encounter the same issue. Teams need a way to evaluate production metrics, stop promotion when the canary performs poorly, and roll back when necessary.
Kubernetes provides the deployment and workload primitives for running application versions side by side. A progressive delivery controller, such as Argo Rollouts, can add automated canary analysis and promotion decisions.
This article explains how to build that workflow, configure metric-based checks, and troubleshoot failed canary releases.
What Is a Kubernetes Canary Release?
A canary release introduces a new application version to a limited portion of production traffic before making it available to everyone.
For example, suppose an application currently runs version 1.4. A team wants to release version 1.5.
Instead of replacing every instance immediately, the team deploys version 1.5 alongside version 1.4 and directs a small percentage of traffic to the new version.
The team then evaluates the canary using production signals such as:
- HTTP error rate
- Request latency
- Application availability
- CPU and memory consumption
- Failed requests to dependent services
- Business-specific metrics, such as checkout failures
If the canary performs within acceptable limits, the rollout can proceed to a larger traffic percentage.
If the metrics degrade, the rollout should stop before more users are affected.
Why stopping is different from rolling back
These are related but distinct actions.
Stopping a rollout prevents further promotion. The canary may continue receiving its current share of traffic while the system waits for an analysis result or an operator decision.
Rolling back moves the application away from the problematic version and restores the previous stable release or traffic configuration.
A production-safe deployment strategy should define both behaviors explicitly. Stopping alone may leave users exposed to a faulty canary, while rolling back without confirming the failure conditions can cause unnecessary deployment changes.
How Automated Canary Analysis Works
A typical automated canary process follows this sequence:
- Deploy the new application version.
- Direct a small portion of traffic to the canary.
- Collect metrics from the stable and canary versions.
- Evaluate the metrics against defined thresholds.
- Continue the rollout if the canary meets the acceptance criteria.
- Stop or abort the rollout if the metrics indicate a problem.
- Verify that the stable version is healthy and that traffic has returned to the intended configuration.
The key is to evaluate the new version using metrics that actually describe its behavior.
For example, an increase in total request latency may be caused by a separate database outage affecting both versions. Comparing canary behavior with the stable version can help identify whether the new release is responsible.
However, comparisons alone are not enough. If both versions are unhealthy, the canary should not automatically be considered safe.
A robust analysis combines absolute service-level thresholds with comparisons against the stable version where appropriate.
Choosing a Progressive Delivery Controller
A standard Kubernetes Deployment can manage rolling updates, but it does not natively provide a complete metric-driven canary analysis workflow.
A progressive delivery controller can coordinate traffic changes, metric analysis, promotion, and rollback.
For example, Argo Rollouts provides a Rollout resource and supports analysis through AnalysisTemplate resources. It can evaluate metrics from systems such as Prometheus and use the results to influence rollout progression.
The following example uses Argo Rollouts with Prometheus as the metrics source.
The exact configuration depends on the installed controller version, traffic-routing integration, metric labels, and monitoring setup.
Defining a Canary Rollout
A canary rollout defines the steps used to introduce a new version.
The following example gradually increases the canary's traffic exposure while inserting pauses between stages.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: orders-api
namespace: production
spec:
replicas: 6
selector:
matchLabels:
app: orders-api
template:
metadata:
labels:
app: orders-api
spec:
containers:
- name: orders-api
image: example.com/orders-api:2.1
ports:
- containerPort: 8080
strategy:
canary:
steps:
- setWeight: 10
- pause:
duration: 2m
- setWeight: 25
- pause:
duration: 3m
- setWeight: 50
- pause:
duration: 5m
- setWeight: 100
This configuration defines a six-replica application and a sequence of canary stages.
The image reference is an example and must be replaced with a valid application image.
The traffic percentages shown here represent the intended canary weights. Actual traffic distribution depends on the configured traffic-routing mechanism. Without a traffic-routing integration, canary weights may be approximated through replica counts rather than enforced as exact request percentages.
The pauses provide time for the system to collect metrics before continuing.
However, pauses alone do not evaluate application health. To stop promotion automatically when production metrics degrade, connect the rollout to an analysis template.
Creating a Prometheus Analysis Template
Prometheus collects application metrics that can be used to evaluate the canary.
For an HTTP service, a useful signal is the proportion of requests that return server errors. The following query calculates the error ratio for the canary using an illustrative metric named http_requests_total.
sum(
rate(http_requests_total{
app="orders-api",
version="canary",
status=~"5.."
}[5m])
)
/
clamp_min(
sum(
rate(http_requests_total{
app="orders-api",
version="canary"
}[5m])
),
1e-9
)
The query calculates the rate of server-error responses divided by the total request rate over five minutes.
The metric labels must match the data actually exposed by the application and collected by Prometheus. If your instrumentation uses different names or labels, adapt the query accordingly.
The clamp_min expression prevents division by zero, but it does not make a lack of traffic equivalent to a healthy canary. A production analysis should explicitly account for insufficient traffic or missing metrics.
Configure the analysis
The following AnalysisTemplate evaluates the error ratio and fails the analysis if the result reaches 5% or higher.
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: orders-api-error-rate
namespace: production
spec:
metrics:
- name: canary-error-rate
interval: 1m
count: 5
successCondition: result[0] < 0.05
failureCondition: result[0] >= 0.05
provider:
prometheus:
address: http://prometheus.monitoring.svc:9090
query: |
sum(
rate(http_requests_total{
app="orders-api",
version="canary",
status=~"5.."
}[5m])
)
/
clamp_min(
sum(
rate(http_requests_total{
app="orders-api",
version="canary"
}[5m])
),
1e-9
)
This example evaluates the metric once per minute for five measurements. A result below 0.05 passes the individual measurement, while a result at or above 0.05 fails it.
The threshold is illustrative, not a universal production target. Set it according to the application's service-level objectives, baseline error rate, traffic volume, and acceptable failure budget.
Also note that a single failed measurement can cause an analysis to fail when a failure condition is defined this way. If brief spikes are expected, design the measurement strategy to tolerate noise without hiding sustained degradation.
Connecting the Analysis to the Canary Rollout
The rollout must reference the analysis template so that the controller evaluates the canary before promotion continues.
For example, add an analysis step to the rollout sequence:
strategy:
canary:
steps:
- setWeight: 10
- pause:
duration: 2m
- analysis:
templates:
- templateName: orders-api-error-rate
- setWeight: 25
- pause:
duration: 3m
- analysis:
templates:
- templateName: orders-api-error-rate
- setWeight: 50
- pause:
duration: 5m
- setWeight: 100
This snippet shows the strategy section that belongs in the Rollout resource. It is not a complete standalone manifest; retain the workload definition, selector, and Pod template from the earlier example.
At the 10% stage, the controller evaluates the error-rate analysis. If the analysis succeeds, the rollout can proceed to the next step. The same check is performed again at the 25% stage.
For more demanding production systems, define separate analysis templates for latency, availability, and other critical signals. Avoid making the deployment decision depend on a single metric when the service has several important failure modes.
What Happens When Metrics Degrade?
When an analysis fails, the controller can stop promotion and abort the canary according to the rollout strategy and failure policy.
For a typical Argo Rollouts canary, a failed analysis causes the rollout to enter a degraded or failed state and abort the rollout. The controller then moves traffic away from the canary according to its configured routing integration and rollout behavior.
The exact traffic transition depends on the controller version and the routing configuration. Verify the behavior in a staging environment before relying on automatic rollback in production.
A safe response should include the following actions:
- Stop promotion to higher traffic percentages.
- Remove or reduce traffic to the failing version when the configured abort behavior requires it.
- Preserve diagnostic information, including analysis results and application logs.
- Notify the responsible team.
- Confirm that the stable version can handle the restored traffic.
- Prevent automatic retries from repeatedly deploying the same faulty version without a corrective change.
A failed analysis should be treated as an operational event, not merely a deployment status update.
Selecting the Right Metrics
The quality of automated rollback depends on the metrics being evaluated.
| Metric | What it detects | Important consideration |
|---|
| HTTP error rate | Increased server-side failures | Low traffic can make the ratio unstable |
| Request latency | Slow responses | Evaluate percentiles such as p95 or p99, not only averages |
| Request volume | Changes in traffic | A low request count may make other metrics unreliable |
| CPU usage | Increased processing demand | High CPU does not always mean users are affected |
| Memory usage | Memory growth or resource pressure | A short observation window may miss gradual leaks |
| Dependency failures | Problems calling databases or external services | Distinguish release-related failures from shared dependency outages |
| Business metrics | Failed purchases, registrations, or other critical operations | Ensure events can be attributed to the canary version |
Use metrics that correspond to the application's actual risks. A latency-sensitive API may require strict response-time thresholds, while a batch-processing service may need to monitor job failures and processing delays.
For systems with low traffic, consider synthetic requests or other validation signals. However, synthetic checks should complement real traffic metrics rather than conceal problems affecting actual users.
Common Mistakes in Automated Canary Rollbacks
Using metrics that include both versions
If the analysis query includes stable and canary traffic together, a small canary failure rate may be hidden by the larger volume of healthy stable requests.
Where possible, instrument the application with labels that identify the release or version and evaluate the canary separately.
Setting thresholds without a baseline
A 5% error-rate threshold may be too permissive for one application and too strict for another.
Review historical metrics and service-level objectives before selecting thresholds. Consider the normal error rate, request volume, observation window, and impact of false positives.
Treating missing data as success
A query may return no data when the canary receives no traffic or when monitoring is misconfigured.
Define how the analysis should behave when data is missing, invalid, or insufficient. In critical systems, an inability to verify health should not silently count as a successful analysis.
Using an observation window that is too short
A brief metric spike may trigger a rollback even when the application recovers immediately. Conversely, an excessively long window can allow a faulty release to affect users for too long.
Choose measurement intervals and sample counts based on traffic patterns and the expected time needed to detect a problem.
Assuming rollback fixes every failure
A rollback can restore the previous application version, but it does not necessarily reverse database migrations, undo external side effects, or repair corrupted data.
Design database changes to support safe transitions between application versions. Use backward-compatible migrations and explicit recovery procedures where necessary.
Troubleshooting a Canary That Does Not Stop
If a rollout continues even when the metrics appear unhealthy, investigate the entire analysis pipeline.
First, confirm that Prometheus is collecting the expected metrics. Run the query directly and verify that it returns the intended result for the canary version.
Next, check the label selectors. A query that references version="canary" will not match anything if the application exposes a different label or uses an image tag without a corresponding metric label.
Then inspect the rollout and analysis resources:
kubectl get rollouts -n production
kubectl describe rollout orders-api -n production
kubectl get analysisruns -n production
kubectl get analysisruns -n production -o wide
Check the relevant AnalysisRun for measurement results, errors, and its final phase. Depending on the controller version, the output fields may differ.
If the analysis is failing but traffic is not changing as expected, inspect the configured traffic-routing integration and controller events. A failed analysis and a successful traffic rollback are separate conditions that should both be verified.
Finally, test the full failure path by deploying a deliberately unhealthy canary in a non-production environment. Confirm that the analysis detects the failure, promotion stops, traffic returns to the intended version, and alerts reach the correct team.
Best Practices for Production Canary Releases
- Define measurable acceptance criteria before deploying a new version.
- Evaluate canary-specific metrics where possible.
- Combine error-rate and latency checks with other signals that matter to the application.
- Account for missing data and insufficient traffic.
- Use gradual traffic increases and observation periods appropriate to the service.
- Test failure detection and rollback behavior before enabling automatic promotion.
- Keep database migrations compatible with the stable and canary versions during the rollout.
- Alert on failed analyses and degraded rollouts.
- Record the metrics and events that caused an automated abort.
- Review false positives and missed failures after significant releases.
Automation should reduce the time needed to contain a faulty release, not remove human visibility into production changes.
Advantages and Limitations
Automated canary analysis reduces reliance on manual observation and can prevent a problematic version from receiving additional traffic. It also provides a consistent process for evaluating releases against predefined service-level objectives.
However, the effectiveness of the process depends on instrumentation, threshold selection, traffic routing, and controller configuration. Poor metrics can cause false alarms or allow real problems to pass unnoticed.
Rollback also cannot undo every kind of application change. Database modifications, external API calls, and irreversible operations require separate safeguards.
For applications with meaningful production traffic and measurable health indicators, automated canary analysis can provide an effective layer of release protection.
Summary
A Kubernetes canary release limits the initial exposure of a new application version, but metric-driven decisions are necessary to make that strategy effective.
By combining a progressive delivery controller such as Argo Rollouts with Prometheus analysis, teams can evaluate error rates, latency, and other health indicators before increasing traffic. Failed analysis can stop promotion and trigger the configured abort behavior.
The most important work happens before deployment: define meaningful thresholds, validate the monitoring queries, verify traffic routing, and test the rollback process. With those safeguards in place, automated canary analysis can reduce the impact of production regressions and make releases more predictable.
Join the conversation! Your thoughts help the community grow.