Canary Releases: Shifting Traffic by Percentage, Deciding with Metrics and Automated Analysis

Version note: The examples in this post were run with .NET 10 (SDK 10.0.402), Nginx 1.30, Docker 29, kind 0.33 on Kubernetes 1.37, Envoy Gateway 1.10 and Argo Rollouts 1.10.2.

Guess First

The canary version has a bug and about 5% of its requests return 500. You give the canary 10% of the traffic. After 200 requests in total, you look at the canary's metrics. Do you see the bug? What about after 4,000 requests?

Second question: you set up a canary in Nginx with weight=9 and weight=1. If the same user sends ten requests in a row, how many go to the new version? Do they always see the same version, or does the version change between pages?

The Idea Behind Canary

A canary first runs the new version with a small slice of real traffic, measures the result, and grows the share step by step if things go well. The name comes from canaries in coal mines; Danilo Sato's 2014 "CanaryRelease" article popularized the technique under that name. The "Canarying Releases" chapter of the Google SRE (Site Reliability Engineering) Workbook sharpens the definition: a canary is a partial, time-limited deployment of a change and its evaluation. The part that receives the change is the canary, the rest is the control, and the decision comes from comparing the two.

The calculation in the Workbook shows in one line why canaries work: a version that breaks 20% of requests gives a 20% error rate if it's opened to all traffic. If the same version runs in a 5% canary, the overall error rate stays at 1%. This is the tool that shrinks the first factor of the "blast radius × detection time" product from the first post of the series.

Weights in Nginx: Simple but Not Sticky

The shortest path is giving weights in the upstream:

upstream bookstore_api {
    zone bookstore_api 64k;
    server depwork-stable:8080 resolve weight=9;
    server depwork-canary:8080 resolve weight=1;
}

for i in $(seq 1 1000); do
    curl -s -o /dev/null -w '%header{x-app-version}\n' localhost:18600/api/version
done | sort | uniq -c

 900 1.0.0
 100 1.1.0

Exactly 900 and 100. Nginx's weighted round robin is deterministic: one in every ten requests goes to the canary. Here is the answer to the second opening question: one of the same user's ten requests goes to the new version. The user sees the new version on one page and the old one on the next. If the new version added a JSON (JavaScript Object Notation) field, the Angular app finds the field in one response and misses it in the next. So a weight-based canary only suits stateless changes where the difference between versions isn't visible.

A Canary That Sticks to Users with split_clients

To make the canary stick to users, the decision must be per user rather than per request. Nginx's split_clients directive takes the MurmurHash2 hash of a string and splits it into percentage buckets. The same string always lands in the same bucket:

events {}

http {
    resolver 127.0.0.11 valid=1s ipv6=off;
    log_format canary '$msec $status $backend $request_time $http_x_user_id';
    access_log /var/log/nginx/canary.log canary;

    upstream bookstore_stable {
        zone bookstore_stable 64k;
        server depwork-stable:8080 resolve;
    }

    upstream bookstore_canary {
        zone bookstore_canary 64k;
        server depwork-canary:8080 resolve;
    }

    split_clients "canary-v1.1.0:${http_x_user_id}" $canary_bucket {
        10%     canary;
        *       stable;
    }

    map $http_x_canary $backend {
        always  canary;
        never   stable;
        default $canary_bucket;
    }

    server {
        listen 80;
        location / {
            proxy_pass http://bookstore_$backend;
        }
    }
}

There are three details. The canary-v1.1.0: prefix of the hashed string is a salt; if you change it every release, the users who land in the canary change too, so the same 10% of users aren't the guinea pigs for every new version. The map sends internal users who send an X-Canary: always header (the QA (Quality Assurance) team, employees) to the canary regardless of the percentage, and the never value does the opposite. Finally proxy_pass picks the upstream name through a variable; as long as the value matches a defined upstream group's name, Nginx uses it. The results:

user 42: 1.0.0 1.0.0 1.0.0 1.0.0 1.0.0
user 7: 1.0.0 1.0.0 1.0.0 1.0.0 1.0.0
user 32: 1.1.0 1.1.0 1.1.0 1.1.0 1.1.0
== 1000 different users
 902 1.0.0
  98 1.1.0
== no user id, 200 requests
 200 1.0.0
== X-Canary: always
{"version":"1.1.0","instance":"696fc643689a"}

User 32 is on the canary for every request, 42 and 7 are on stable for every request. 98 of 1000 different users landed on the canary; the target was 10%. But note: all 200 requests without X-User-Id went to stable. The hash of an empty string lands in the same bucket every time. On a site with many anonymous users this silently shrinks the canary's share of traffic; with an unlucky salt the opposite happens and all anonymous traffic goes to the canary. For anonymous users you need to assign a cookie on the first visit and hash that, or use the client IP (Internet Protocol) as a fallback key.

sticky cookie in Nginx 1.30

Since Nginx 1.29.6 the open source version has the sticky directive for upstreams; it used to be commercial only. Combined with weights it gives a canary that sticks via a cookie:

upstream bookstore_api {
    zone bookstore_api 64k;
    server depwork-stable:8080 resolve weight=9 route=stable;
    server depwork-canary:8080 resolve weight=1 route=canary;
    sticky cookie bk_route expires=1h path=/ httponly;
}

I sent four requests from 12 clients, each with its own cookie file. The results for the first six and the cookie Nginx wrote:

client 1: 1.0.0 1.0.0 1.0.0 1.0.0
client 2: 1.0.0 1.0.0 1.0.0 1.0.0
client 3: 1.0.0 1.0.0 1.0.0 1.0.0
client 4: 1.0.0 1.0.0 1.0.0 1.0.0
client 5: 1.1.0 1.1.0 1.1.0 1.1.0
client 6: 1.0.0 1.0.0 1.0.0 1.0.0
Set-Cookie: bk_route=canary; expires=Sun, 08-Nov-26 04:36:29 GMT; max-age=3600; httponly; path=/

On the first request Nginx picks a server by weight and writes its route value into the cookie; later requests go to the same server. Because it doesn't depend on user identity, it works for anonymous traffic too. But a user who clears cookies or comes from another device is redistributed.

Decision Metrics: What Do You Measure?

A canary is worth as much as what you measure. There are three layers:

• Error rate: 5xx count / total requests, separately for canary and control. Keep client errors (4xx) separate; if the new version tightened a validation, 400s may rise, and that can be a deliberate change rather than a bug.
• Latency: p95 or p99, not the average. Slowness in the new version disappears in the average and shows up in the tail.
• Business metrics: add-to-cart and checkout completion rates. A version that's technically flawless but puts the checkout button in the wrong place only shows up here. These metrics accumulate slowly, so they usually become meaningful in the later steps of a canary.

What you compare against matters too. The SRE Workbook recommends comparing a canary and a control running at the same time, rather than "the week before the deploy against the week after". In a before/after comparison, weekday versus weekend traffic, a campaign or another team's deploy skews the result. Two groups running at the same time are exposed to the same outside effects.

Let's write the chosen backend and the duration into the Nginx log and produce a small report:

#!/usr/bin/env bash
# canary-report.sh LOGFILE: request count, 5xx rate and p95 latency per backend
set -euo pipefail
for v in $(awk '$3 != "-" {print $3}' "$1" | sort -u); do
    awk -v v="$v" '$3 == v {print $2, $4}' "$1" | sort -k2 -n | awk -v v="$v" '
        { n++; if ($1 >= 500) e++; t[n] = $2 }
        END { i = int(n * 0.95); if (i < 1) i = 1;
              printf "%-6s requests=%-5d 5xx=%-4d error_rate=%.2f%% p95=%.3fs\n", v, n, e, 100 * e / n, t[i] }'
done

I started the canary version with BUG_FAIL_RATE=0.05 (a test setting in Bookstore that throws an exception on 5% of requests) and sent requests from 4,000 different users:

seq 1 4000 | xargs -P 8 -I{} curl -s -o /dev/null -H "X-User-Id: {}" localhost:18600/api/books
docker cp depwork-nginx:/var/log/nginx/canary.log canary.log
./canary-report.sh canary.log

canary requests=396   5xx=19   error_rate=4.80% p95=0.001s
stable requests=3604  5xx=0    error_rate=0.00% p95=0.001s

The difference is clear: canary 4.80%, stable 0%. Because failed requests don't carry the X-App-Version header (the exception wipes the response together with the headers the middleware added), we log Nginx's routing decision ($backend) instead of the version. Know where your metric comes from; a label that disappears on failed responses pushes the errors out of the count.

The Statistical Trap at Low Traffic

This time I sent only 200 requests to the same canary:

canary requests=19    5xx=1    error_rate=5.26% p95=0.001s
stable requests=181   5xx=0    error_rate=0.00% p95=0.001s
WAIT: canary has 19 requests, need 300
exit=2

One of 19 requests failed. 5.26% looks very close to the true rate, but with 19 requests it's pure luck. The same run could have ended with 0 errors or with 3. If you saw 0 errors you'd declare the canary "clean". That's why the decision script's first question must be "is there enough data?":

#!/usr/bin/env bash
# canary-gate.sh LOGFILE: exit 0 = promote, 1 = abort, 2 = not enough data yet
set -euo pipefail
min_requests=300
max_extra_error_pct=1.0
read -r cn ce sn se < <(awk '
    $3 == "canary" { cn++; if ($2 >= 500) ce++ }
    $3 == "stable" { sn++; if ($2 >= 500) se++ }
    END { print cn + 0, ce + 0, sn + 0, se + 0 }' "$1")
if [ "$cn" -lt "$min_requests" ]; then
    echo "WAIT: canary has $cn requests, need $min_requests"
    exit 2
fi
verdict=$(awk -v cn="$cn" -v ce="$ce" -v sn="$sn" -v se="$se" -v max="$max_extra_error_pct" 'BEGIN {
    c = 100 * ce / cn; s = (sn > 0) ? 100 * se / sn : 0
    printf "%s canary=%.2f%% stable=%.2f%% limit=+%.1f", (c - s > max) ? "ABORT" : "PROMOTE", c, s, max }')
echo "$verdict"
[[ "$verdict" == PROMOTE* ]]

The script has three outcomes: PROMOTE, ABORT and WAIT. The 200-request run returned 2 with WAIT. The broken 4,000-request run and a healthy canary run with the same load ended like this:

ABORT canary=4.80% stable=0.00% limit=+1.0
exit=1
PROMOTE canary=0.00% stable=0.00% limit=+1.0
exit=0

The thresholds (300 requests, +1 point) are example values. In a real system you choose the minimum sample size based on the error rate you expect to catch: a 0.1% error needs thousands of requests, a 5% error a few hundred. If your traffic is low, you lengthen the canary, increase its share, or back it up with synthetic tests instead of technical metrics. In the pipeline in the last post of the series we built this rule as "wait until enough requests arrive" instead of "wait a fixed time", because a fixed time wasn't enough on the first attempt.

A Step Plan

A typical plan is 1%, 10%, 50% and 100%, with a wait and a decision at each step. The gaps between steps are logarithmic, because the first step's goal is to answer "is it completely broken?" with very few users. The later steps are for seeing problems that appear under load (connection pools, cache hit rates, database locks). At 50% both versions take equal load, and comparing canary and control is most meaningful. Step duration depends on your traffic and deploy frequency. As the SRE Workbook points out, a team deploying twenty times a day can't run a week-long canary.

Weights with the Kubernetes Gateway API

In Kubernetes, the Gateway API's (Application Programming Interface) HTTPRoute does the same job. The weights in backendRefs split traffic between Services, and matches routes by header:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookstore-api
spec:
  parentRefs:
    - name: bookstore
  rules:
    - matches:
        - headers:
            - name: X-Canary
              value: always
      backendRefs:
        - name: bookstore-canary
          port: 80
    - backendRefs:
        - name: bookstore-stable
          port: 80
          weight: 90
        - name: bookstore-canary
          port: 80
          weight: 10

I installed Envoy Gateway 1.10 on the kind cluster and sent 1000 requests plus one X-Canary: always request from a curl pod:

    901 1.0.0
     99 1.1.0
{"version":"1.1.0","instance":"bookstore-green-59cffbc476-ncrqn"}

901 to 99. This distribution is Envoy's decision; it isn't deterministic like Nginx's, but it's close to the ratio. HTTPRoute itself provides no stickiness; if you need to stick to users, you write a separate rule with a header or cookie match. The Gateway API details are in the external traffic post of the upcoming Kubernetes series.

Automated Analysis and Promotion with Argo Rollouts

Changing weights by hand and looking at metrics works for a few deploys, then becomes tiring. Argo Rollouts lets you define steps, pauses and analysis with a Rollout resource that replaces the Deployment. Flagger builds the same thing on top of existing Deployments. Using them together with GitOps is the topic of the upcoming GitOps and Argo CD (Continuous Delivery) series; here I only show the idea with a real run:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: bookstore-api
spec:
  replicas: 5
  selector:
    matchLabels:
      app: bookstore-rollout
  template:
    metadata:
      labels:
        app: bookstore-rollout
    spec:
      containers:
        - name: api
          image: depwork-bookstore-api:1.0.0
          env:
            - name: BUG_FAIL_RATE
              value: "0"
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
  strategy:
    canary:
      stableService: bookstore-rollout-stable
      canaryService: bookstore-rollout-canary
      steps:
        - setWeight: 20
        - pause: { duration: 10s }
        - analysis:
            templates:
              - templateName: canary-error-rate
        - setWeight: 60
        - pause: { duration: 10s }

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-error-rate
spec:
  metrics:
    - name: http-5xx
      provider:
        job:
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers:
                  - name: check
                    image: curlimages/curl:8.23.0
                    command:
                      - sh
                      - -c
                      - |
                        fails=0
                        for i in $(seq 1 200); do
                          code=$(curl -s -o /dev/null -w '%{http_code}' http://bookstore-rollout-canary/api/books)
                          [ "$code" -ge 500 ] && fails=$((fails + 1))
                        done
                        echo "5xx: $fails/200"
                        [ "$fails" -le 2 ]

Because I didn't define traffic routing, Argo Rollouts approximates the weight with pod counts: with five replicas, 20% means one new pod. To show analysis without Prometheus I used the Job provider: a Job that sends 200 requests to the canary Service and fails if it sees more than 2 5xx responses. A note on installation as well: the CRDs (Custom Resource Definitions) in Argo Rollouts' install.yaml failed with "metadata.annotations: Too long" on a plain kubectl apply and were applied with --server-side.

The rollout of a healthy version (lines selected from the status taken every four seconds):

06:41:44 Paused step=1 CanaryPauseStep | pods:  4 1.0.0, 1 1.1.0,
06:41:52 Progressing step=2 more replicas need to be updated | pods:  4 1.0.0, 1 1.1.0,
06:41:56 Paused step=4 CanaryPauseStep | pods:  2 1.0.0, 3 1.1.0,
06:42:08 Healthy step=5  | pods:  5 1.1.0,

This time I gave the same Rollout a new revision with BUG_FAIL_RATE=0.1. The numbers in the pods column show the env value:

06:42:50 Paused step=1 CanaryPauseStep | pods:  4 0, 1 0.1,
06:42:58 Progressing step=2 more replicas need to be updated | pods:  4 0, 1 0.1,
06:43:02 Degraded step=0 RolloutAborted: Rollout aborted update to revision 3: Step-based analysis phase error/failed: Metric "http-5xx" assessed Failed due to failed (1) > failureLimit (0) | pods:  5 0,

NAME                           STATUS       AGE
bookstore-api-659d7455f4-2-2   Successful   92s
bookstore-api-d9b6564bc-3-2    Failed       26s

5xx: 20/200

The analysis Job saw 20 errors in 200 requests and failed. Argo Rollouts stopped the rollout and shut down the canary pod. All five pods are on the old version and the status is Degraded. That status is deliberate: the system is saying "I went back, but a human needs to look". It stays that way until a new revision or kubectl argo rollouts retry arrives.

Shadow Traffic and Dark Launch

A canary gives real answers to real users. Sometimes you want to try the new version under real load without ever showing it to users. Nginx's mirror directive sends a copy of every request elsewhere and throws away the copy's response:

server {
    listen 80;

    location /api/books {
        mirror /shadow;
        proxy_pass http://bookstore_stable;
    }

    location = /shadow {
        internal;
        proxy_pass http://bookstore_shadow$request_uri;
    }
}

The shadow side is a version that fails 50% of the time. On the user side there were 20 requests, and the Nginx log (with log_subrequest on) shows:

client sees:
  20 200 1.0.0
nginx log:
  20 200 1.0.0 /api/books
  11 200 1.1.0 /shadow
   9 500 - /shadow

None of the user's requests were affected, yet we learned the new version failed on 9 of 20 copies. By default mirror subrequests aren't written to the log; without log_subrequest on you'd never see the shadow's errors. Use shadowing only on read endpoints. A copied POST /api/orders means a second order, a second payment attempt or a second email. Shadowing writes needs a separate database and fakes of the external services. Dark launch is the feature-flag version of the same idea: the code runs in production and its result is computed, but not shown to the user.

In Production

An automated canary decision doesn't mean the thresholds are right. In the first few canaries, compare the automated decision with a human one and tune the thresholds with real data. Also check whether canary and control share infrastructure. A canary that strains the same database slows the control too, and the difference disappears.

Log which version the canary showed to which user (our $backend field). When a customer says "the checkout page was weird last night", knowing whether they were on the canary explains the incident in minutes.

Common Pitfalls

• Deciding at low traffic. 0 errors in 19 requests says nothing; automated promotion without a minimum sample size is a coin toss.
• Confusing canary with feature flags. A canary restricts the version (the code), a flag restricts behavior. A user on the canary runs the new code but doesn't see the new feature if the flag is off. When chasing a bug, ask about both separately.
• Using a weight-based canary for stateful interactions. A user whose version changes page by page sees the bugs of the mix, not of the new version.
• Looking only at the average. Degradation at p95 doesn't show up in the average.
• Opening shadowing to write endpoints. Copied requests produce real side effects.

A Small Experiment

★☆☆ Change the salt in split_clients to canary-v1.2.0: and send the same 1000 users again. How many users land on the canary in both releases?

★★☆ Add a p95 latency check to canary-gate.sh: ABORT if the canary's p95 is more than 20% higher than the control's. Use Bookstore's /api/slow?ms= endpoint to add artificial latency to the canary.

★★★ Replace the Job analysis in Argo Rollouts with the web metric provider: write a small endpoint that returns the canary's error rate as JSON and evaluate it with successCondition.

A canary shrinks the first factor of risk, the blast radius. But underneath all of them lies the same thing: two versions use the same database at the same time. In the next post we look at how to do that safely: where migrations fit in a deploy, expand-contract and EF (Entity Framework) Core migration bundles.