Progressive Delivery with Argo Rollouts: Automated Canary Analysis with Prometheus Metrics
Argo Rollouts replaces the Deployment with a Rollout that opens a new version step by step and advances or rolls back by itself based on metrics read from Prometheus. We convert the Bookstore API to a Rollout, set up a canary with 25% and 50% steps in prod, and see with real output a good version going through, a version returning 30% errors aborted in 92 seconds, and Git brought back in line with a revert.
Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, Argo CD 3.6.1, Argo Rollouts 1.10.2, Prometheus 3.16, .NET 10 and OpenTelemetry .NET 1.21 (OpenTelemetry.Exporter.Prometheus.AspNetCore 1.21.0-beta.1).
Guess First
Version 1.4.0 of the Bookstore API returns 500 for roughly a third of /api requests. It passed dev and staging, because the PostSync smoke test in both environments looks only at /health. The pull request to prod is merged; prod has four pods and the new version first goes to 25%. Who notices, how fast, and what does Git say afterwards? The answer, minute by minute, is in the middle of the post: Argo Rollouts pulled the version back in 92 seconds, but the story did not end there.
A Rollout Instead of a Deployment
A Kubernetes Deployment does a rolling update: old pods stop as new ones become ready, and nobody looks at the error rate of the new version. What we want for the API of Bookstore, the online bookstore used as the shared example across this site's series, is the model from the canary post of the Deployment Strategies series: the new version first goes out to a small share, grows if the metrics are good and goes back if they are bad. That post covered the theory and the decision metrics; here a controller, Argo Rollouts, does the work. Rollouts brings a Rollout CRD that replaces the Deployment and a controller that manages it. It installs from the official manifest, with one twist:
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/download/v1.10.2/install.yamlCustomResourceDefinition.apiextensions.k8s.io "rollouts.argoproj.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytesClient-side apply writes the whole object into the last-applied-configuration annotation, and the schema of the Rollout CRD exceeds the limit. As with Argo CD in the second post, we applied it again with --server-side. Then deployment.yaml in the base became rollout.yaml; the pod template is the same, only the kind and the strategy change:
# A Rollout instead of a Deployment (post 08): same pod template, plus a release strategy.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api
annotations:
argocd.argoproj.io/sync-wave: "1"
labels:
app.kubernetes.io/name: api
spec:
replicas: 1
revisionHistoryLimit: 3
progressDeadlineSeconds: 120
selector:
matchLabels:
app.kubernetes.io/name: api
# Without steps a canary Rollout behaves like a rolling update (dev, staging);
# the prod overlay adds steps and analysis.
strategy:
canary: {}
template:We do not want a canary in dev and staging; every version should go out fully at once there. The prod overlay adds steps, analysis and the replica count with a patch:
# Prod releases go through a canary: 25% -> 50% -> 100%, with Prometheus analysis in the background.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api
spec:
replicas: 4
strategy:
canary:
analysis:
templates:
- templateName: api-canary-health
# Start analysing at step 1 (after the first setWeight), abort the rollout when it fails.
startingStep: 1
args:
- name: namespace
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: canary-hash
valueFrom:
podTemplateHashValue: Latest
steps:
- setWeight: 25
- pause: {duration: 2m}
- setWeight: 50
- pause: {duration: 2m}The prod overlay used to have replicas: [{name: api, count: 2}]. Kustomize's replicas field only changes kinds it knows, and on seeing the Rollout it stopped the build: resource with name api does not match a config with the following GVK [Deployment... StatefulSet... ReplicaSet... ReplicationController...]. That is why the replica count sits in the patch. After the commit Argo CD created the Rollout in all three environments and deleted the Deployment last thanks to PruneLast. When a Rollout is created for the first time it does not run its steps; prod went straight to 100% with 1.2.0.
How Is Traffic Split?
The plan was 10%, 50% and 100%; we started with 25%. The reason is traffic management: if the Rollout has no trafficRouting, the weight is achieved through pod counts. With four replicas, 25% means one canary pod and three stable pods; the Service spreads requests evenly over the pods of both ReplicaSets, so roughly a quarter of the traffic hits the canary. 10% would need ten replicas. For an exact percentage Rollouts has to talk to a traffic router: the Argo project's plugin for the Gateway API (rollouts-plugin-trafficrouter-gatewayapi, currently 0.17.0), Istio, NGINX Ingress, Amazon's Application Load Balancer and others. Then the stable ReplicaSet stays at full capacity and only the weight in the router changes. We did not install an ingress controller in the lab; the replica ratio is enough for Bookstore's four pods.
The second Rollouts strategy is blueGreen: the new version comes up at full capacity, is tried through a preview Service, and after approval the active Service switches to it in one move. prePromotionAnalysis can run an analysis before the switch. We did not set it up in this post; the theory is in the blue-green post of the Deployment Strategies series.
Metrics for the Analysis: the Application, Prometheus and the AnalysisTemplate
Deciding whether the canary is good needs the metrics of the canary pods only. First we exposed HTTP metrics from the API, with OpenTelemetry as in the Observability series:
// HTTP server metrics for Prometheus (GET /metrics): canary analysis in the GitOps series reads them.
builder.Services.AddOpenTelemetry()
.WithMetrics(metrics => metrics
.AddAspNetCoreInstrumentation()
.AddPrometheusExporter());
app.MapPrometheusScrapingEndpoint();The Prometheus running in the lab (lab/prometheus/prometheus.yaml) finds the api pods in the bookstore-* namespaces through Kubernetes service discovery. The important line turns the ReplicaSet hash that Argo Rollouts puts on every pod into a metric label:
# Argo Rollouts labels every pod with the hash of its ReplicaSet: stable vs canary.
- source_labels: [__meta_kubernetes_pod_label_rollouts_pod_template_hash]
target_label: rollouts_pod_template_hashThe Rollout passes the hash of the new ReplicaSet to the analysis as an argument with podTemplateHashValue: Latest. The AnalysisTemplate defines two metrics: the 5xx rate of /api requests and the p95 latency. The details of PromQL are in the Prometheus and PromQL post of the Observability series:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: api-canary-health
spec:
args:
- name: namespace
- name: canary-hash
metrics:
- name: error-rate
# The query uses a 1m window: wait until the canary has a minute of traffic.
initialDelay: 60s
interval: 30s
failureLimit: 1
successCondition: len(result) > 0 && result[0] < 0.05
provider:
prometheus:
address: http://prometheus.monitoring.svc.cluster.local:9090
query: |
sum(rate(http_server_request_duration_seconds_count{namespace="{{args.namespace}}", rollouts_pod_template_hash="{{args.canary-hash}}", http_route=~"/api/.*", http_response_status_code=~"5.."}[1m]) or vector(0))
/
sum(rate(http_server_request_duration_seconds_count{namespace="{{args.namespace}}", rollouts_pod_template_hash="{{args.canary-hash}}", http_route=~"/api/.*"}[1m]))
- name: p95-latency
initialDelay: 60s
interval: 30s
failureLimit: 1
successCondition: len(result) > 0 && result[0] < 0.5
provider:
prometheus:
address: http://prometheus.monitoring.svc.cluster.local:9090
query: |
histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{namespace="{{args.namespace}}", rollouts_pod_template_hash="{{args.canary-hash}}", http_route=~"/api/.*"}[1m])))The thresholds: an error rate below 5% and a p95 below half a second. failureLimit: 1 allows one failed measurement; at the second the analysis is Failed. The len(result) > 0 condition keeps an empty query result from counting as a success; without traffic the analysis does not pass. With startingStep: 1 the analysis starts in the background and runs until the rollout finishes (background analysis). We put a loadgen Deployment in prod that sends about ten requests a second; in real life that traffic comes from users.
A Good Version: 1.3.0
We shipped 1.3.0, which exposes the metrics, through the flow from the sixth post: CI committed to dev, and the promotion branches for staging and prod were merged. At the first step (25%) the Rollout in prod looked like this:
kubectl argo rollouts get rollout api -n bookstore-prod --no-colorStatus: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 1/4
SetWeight: 25
ActualWeight: 25
Images: localhost:52612/bookstore/api:1.2.0@sha256:<digest-1.2.0> (stable)
localhost:52612/bookstore/api:1.3.0@sha256:<digest-1.3.0> (canary)
NAME KIND STATUS AGE INFO
⟳ api Rollout ॥ Paused 4m32s
├──# revision:2
│ ├──⧉ api-6f9cb4c9ff ReplicaSet ✔ Healthy 93s canary
│ │ └──□ api-6f9cb4c9ff-cn64p Pod ✔ Running 93s ready:1/1
│ └──α api-6f9cb4c9ff-2 AnalysisRun ◌ Running 92s ✔ 4
└──# revision:1
└──⧉ api-79f5d5fb59 ReplicaSet ✔ Healthy 4m32s stable
├──□ api-79f5d5fb59-hgxhl Pod ✔ Running 4m32s ready:1/1
├──□ api-79f5d5fb59-l8mh9 Pod ✔ Running 4m32s ready:1/1
└──□ api-79f5d5fb59-xsqjd Pod ✔ Running 4m32s ready:1/1Meanwhile Argo CD shows the application as Suspended; in Argo CD's health model a paused Rollout means "waiting", not an error. The Rollout reached 100% in 4 minutes 6 seconds; the AnalysisRun passed all 14 measurements (an error rate of 0 every time, p95 around 5 milliseconds).
A Bad Version: 1.4.0 and Automatic Rollback
To produce a faulty version we added a lab-only setting to the API: when Bookstore:FaultRate is above zero, that share of /api requests returns 500. The CI script bakes it into the image as a build arg, so 1.4.0 is a "broken release" on its own:
FAULT_RATE=0.3 CONFIG_REPO=../bookstore-config ci/build-and-push.sh api 1.4.01.4.0 passed dev and staging; in both environments the Rollout has no steps, so it went out at once, and the PostSync smoke test stayed green because it checks /health. The timeline after the branch to prod was merged (from the log of a script that read the Rollout every 15 seconds, times in UTC):
04:57:21 Status: ◌ Progressing step 0/4 weight 0
04:57:37 Status: ॥ Paused step 1/4 weight 25 AnalysisRun api-85645b89d-3 ◌ Running
04:58:23 Status: ॥ Paused step 1/4 weight 25 AnalysisRun api-85645b89d-3 ◌ Running ✔ 1,✖ 1
04:58:53 Status: ✖ Degraded step 0/4 weight 0 AnalysisRun api-85645b89d-3 ✖ Failed ✔ 2,✖ 2api-85645b89d-3 Failed Metric "error-rate" assessed Failed due to failed (2) > failureLimit (1)
p95-latency: Successful values=[0.004788722738625373] [0.004792792792792794]
error-rate: Failed values=[0.3097411866014056] [0.2857142857142857]The answer to Guess First: 92 seconds after the canary started, the analysis measured an error rate above 5% for the second time (31% and 29%), aborted the Rollout, scaled the canary ReplicaSet to zero and brought the stable one back to four pods. For a minute and a half a quarter of the traffic went to the canary and about 30% of that failed; the p95 latency was fine, so an analysis that looked only at latency would have let this version through.
Status: ✖ Degraded
Message: RolloutAborted: Rollout aborted update to revision 3: Background analysis phase error/failed: Metric "error-rate" assessed Failed due to failed (2) > failureLimit (1)
Images: localhost:52612/bookstore/api:1.3.0@sha256:<digest-1.3.0> (stable)After the Abort: Git Still Says 1.4.0
The story does not end here. The cluster runs 1.3.0, but Git says 1.4.0 for prod. Argo CD's view:
Sync Status: Synced to main (b1d1de8)
Health Status: Degraded
argoproj.io Rollout bookstore-prod api Synced Degraded RolloutAborted: Rollout aborted update to revision 3: ...Argo CD says Synced because the Rollout's spec matches Git; the drift is not in the spec but in the Rollout's status. Self-heal cannot fix this and should not: trying 1.4.0 again leads to the same result. The right step in GitOps is to make Git match reality, that is, to revert the merge commit of the prod promotion:
git revert -m 1 --no-edit e158a1235f7a8d Revert "Merge promote/prod-api-1.4.0: bookstore-api 1.4.0 to prod"After the revert Argo CD showed the application as OutOfSync but did not sync it. The reason was the sync operation of 1.4.0: Argo CD was waiting for the Rollout in wave 1 to become Healthy, the task counted as failed once the Rollout was Degraded, and automated sync went into retries for the same revision (Retrying attempt #5). It is the situation we saw with the migration failure in the fourth post: a new operation does not start until the old one ends. Stopping the old operation with argocd app terminate-op bookstore-api-prod applied the revert within 15 seconds and the Rollout became Healthy. Since the template equals that of the stable ReplicaSet, Rollouts skips the steps; going back to stable is not a canary.
There is no automatic way to reflect the abort in Git; a person or an automation has to do it. Telling the team through Argo CD Notifications when a Rollout goes Degraded (the last post) and a small job that opens the revert as a pull request are a common pair. The Argo CD UI also offers resource actions such as resume, abort, retry and promote-full for a Rollout; in an emergency they manage the Rollout without touching Git, but the lasting decision still belongs in Git.
Flagger: the Same Job from the Flux Side
Flagger, the progressive delivery tool of the Flux family, solves the same problem with a different model: you do not change your Deployment; Flagger creates a Canary object and copies next to it, shifts traffic through a service mesh or an ingress, and decides with Prometheus queries. It is the natural choice for teams that do not want to turn Deployments into Rollouts and already use Flux and a service mesh. It works with Argo CD as well, but Argo CD's built-in Rollout health check and UI actions make Rollouts the better fit for this series.
Production Notes
Analysis misleads at low traffic: for a canary that gets one request a second, a single 500 pushes the error rate in a one-minute window to 2%, and with no requests the query returns nothing. Pick thresholds that suit the traffic volume, put a minimum request count into the query, and lengthen or stop canaries at night. The HPA works with a Rollout: scaleTargetRef points at the Rollout, and Rollouts splits the total replica count between stable and canary according to the steps; leaving the HPA on the old Deployment when switching to a Rollout is the most common mistake. Migrations do not fit canaries: the migration Job from the fourth post runs as a Sync hook before the canary starts, so the new schema is in front of both stable and canary pods. The expand-contract rule from the migration post of the Deployment Strategies series is mandatory here; a canary does not roll back a broken migration. The Prometheus that the analysis depends on is a dependency too: if Prometheus is unreachable the measurements become Error, and after consecutiveErrorLimit (4 by default) consecutive errors the metric becomes Error and the rollout is aborted; an outage in the monitoring system can stop releases.
Common Pitfalls
Two writers on the same overlay. The Image Updater left over from the sixth post was following 1.2.x in dev. Two to three minutes after CI wrote 1.3.0 and 1.4.0 to dev, Image Updater pulled dev back to 1.2.1 both times: the running image did not match its rule, and the newest version under the rule was 1.2.1.
b1d1de8 argocd-image-updater bookstore-api-dev: update images
bookstore/api: sha256:<digest-1.4.0> -> 1.2.1
bookstore/migrator: sha256:<digest-migrator-1.4.0> -> 1.2.1We removed Image Updater's custom resource from Git. An overlay is written either by CI or by Image Updater; if both write it, the last one wins.
Rushing with git revert HEAD. In our first attempt we reverted HEAD instead of the merge commit; at that moment HEAD was Image Updater's dev commit. In a shared config repo, always revert by the commit's SHA.
Forgetting to fix Git after an abort. The cluster runs the stable version, but Argo CD is Degraded, the sync operation keeps retrying and the next commit is not applied.
Expecting a canary on the first Rollout. A Rollout does not run its steps when it is first created; when moving from a Deployment, the first version goes straight to 100%.
An analysis that looks only at latency or only at health checks. 1.4.0 had a good p95 and a healthy /health.
A Small Experiment
★☆☆ Skip a pause step early with kubectl argo rollouts promote api -n bookstore-prod and confirm that the AnalysisRun keeps running in the background anyway. Then find the resource action menu of the same Rollout in the Argo CD UI.
★★☆ Build 1.4.1 with FAULT_RATE=0.03 and ship it to prod. Does the analysis pass? Set failureLimit to 0 and the threshold to 2%, try again, and observe the balance between false alarms and missed failures at low error rates.
★★★ Install the Gateway API plugin and a Gateway API compatible controller, add trafficRouting.plugins to the Rollout and make the steps 10%, 50% and 100%. Compare the replica count of the stable ReplicaSet in both approaches.
Argo CD now deploys from Git, secrets are safe in Git, and prod releases are guarded by metrics. What remains is to make this a team's daily work: who may sync what, notifications, Argo CD's own metrics and upgrades. The last post of the series looks at running Argo CD in production and at what changed from 2.x to 3.6.