Rolling Deployments: Updating Servers One by One, maxSurge and maxUnavailable in Kubernetes

Version note: The examples in this post were run with .NET 10 (SDK 10.0.402), Nginx 1.30, Docker 29 and kind 0.33 on Kubernetes 1.37.

Guess First

The Bookstore API (Application Programming Interface) runs as three instances behind Nginx. To roll out a new version you write the simplest possible script: it deletes each instance in turn, starts a new one, and waits three seconds in between. Meanwhile about ten GET requests per second arrive, along with a stream of POST requests. How many requests fail? Zero, a handful, or a one-second outage on every instance swap?

Second question: you do the same thing in Kubernetes with maxSurge: 1 and maxUnavailable: 0, but the new version never becomes ready. When the 60-second progressDeadlineSeconds expires, does Kubernetes roll back to the old version on its own?

What Rolling Deployment Does and What It Doesn't Assume

A rolling deployment updates the fleet piece by piece instead of all at once. At each step one or a few instances leave traffic, switch to the new version, and come back once they are verified healthy. No extra servers are needed, which makes it the cheapest zero-downtime strategy, and it's Kubernetes' default strategy too.

It has two costs. First, the old and new versions take traffic at the same time throughout the deploy, so one request from a user can hit v1 and the next one v2. Second, going back is also a rolling deploy: if half the fleet is on the new version when you spot the bug, reverting them one by one takes the same time. In this post we first build a rolling deploy by hand with three containers that mimic a VM (Virtual Machine) fleet, then look at which decision is yours in Kubernetes.

Starting Point: Three Instances Behind Nginx

We keep the upstream definition in a separate file, because the script will rewrite only this file at every step:

upstream bookstore_api {
    zone bookstore_api 64k;
    server depwork-api-1:8080 resolve;
    server depwork-api-2:8080 resolve;
    server depwork-api-3:8080 resolve;
}

http {
    resolver 127.0.0.11 valid=1s ipv6=off;

    include /etc/nginx/bookstore/upstream.conf;

    server {
        listen 80;
        location / {
            proxy_pass http://bookstore_api;
        }
    }
}

The resolve parameter has been available in open source Nginx since 1.27.3. When a container is recreated and gets a new IP (Internet Protocol) address, Nginx re-resolves the name according to the resolver validity time. Without it, Nginx would keep connecting to the old IP it resolved at startup. A VM fleet doesn't have this problem because addresses are fixed, but in Docker and any dynamic environment rolling doesn't work without this line. The Nginx details are in the load balancing post of the upcoming Web Servers and Hosting series.

Naive Rolling: Delete and Start, One by One

The first script is the one most teams write first:

#!/usr/bin/env bash
# rolling-naive.sh VERSION: restart instances one by one, no drain, no health check
set -euo pipefail
for i in 1 2 3; do
    docker rm -f "depwork-api-$i" >/dev/null
    docker run -d --name "depwork-api-$i" --network depwork-net --stop-timeout 40 \
        "depwork-bookstore-api:$1" >/dev/null
    echo "$(date +%T) api-$i -> $1"
    sleep 3
done

Two probes run alongside it. One sends roughly ten GETs per second, the other calls POST /api/orders, which takes 200 ms, back to back:

(./tools/probe.sh http://localhost:18600/api/books 14 > get.txt) &
(./tools/probe.sh http://localhost:18600/api/orders 14 POST > post.txt) &
sleep 2; ./rolling-naive.sh 1.1.0; wait
awk '{c[$3] += $1} END {for (k in c) print k, c[k]}' get.txt

06:28:04 api-1 -> 1.1.0
06:28:07 api-2 -> 1.1.0
06:28:11 api-3 -> 1.1.0
GET
000 2
200 88
POST
201 41

Two of the 90 GETs got no response within one second (000). Nginx's error log shows why:

[error] 30#30: *36 connect() failed (111: Connection refused) while connecting to upstream, client: 172.66.2.5, server: , request: "GET /api/books HTTP/1.1", upstream: "http://172.19.0.2:8080/api/books", host: "localhost:18600"
[error] 30#30: depwork-api-1 could not be resolved (3: Host not found)

docker rm -f kills the container with SIGKILL (the kill signal). That's no different from a VM suddenly powering off or the process crashing. Kestrel gets no chance to finish its requests. Nginx first gets a connection refusal, then can't resolve the name, and in between connection attempts to the old IP exceed curl's one-second patience. Seeing no POST errors isn't luck; it's Nginx's behavior: proxy_next_upstream defaults to error timeout. When the connection can't be established, meaning the request never reached the upstream, Nginx sends the POST to the next server as well. Things change if the connection drops after the request was sent: non-idempotent methods such as POST, LOCK and PATCH are not retried unless non_idempotent is set explicitly, and the error goes to the user. So what saved the POSTs in this run is that the worst moment didn't land in the middle of a request.

One more thing: the script never checks whether the new container is healthy after starting it. If the new version crashed on startup, the script would happily move to the second instance and turn the whole fleet into the broken version.

Proper Rolling: Drain, Update, Verify, Re-enable

A proper rolling deploy takes four steps per instance. It takes the instance out of the load balancer, then waits for in-flight work to finish. Next it stops the instance gracefully, starts the new one, and doesn't give it traffic until readiness says so:

#!/usr/bin/env bash
# rolling.sh VERSION: drain, update, verify and re-enable one instance at a time
set -euo pipefail
version=$1
conf=nginx/bookstore/upstream.conf

write_upstream() {
    local i flag
    {
        echo "upstream bookstore_api {"
        echo "    zone bookstore_api 64k;"
        for i in 1 2 3; do
            flag=""
            if [ "$i" = "$1" ]; then flag=" down"; fi
            echo "    server depwork-api-$i:8080 resolve$flag;"
        done
        echo "}"
    } > "$conf"
    docker exec depwork-nginx nginx -t -q
    docker exec depwork-nginx nginx -s reload 2>/dev/null
}

wait_ready() {
    local _
    for _ in $(seq 1 30); do
        if docker exec depwork-nginx wget -q -O /dev/null "http://depwork-api-$1:8080/health/ready" 2>/dev/null; then
            return 0
        fi
        sleep 1
    done
    echo "api-$1 did not become ready" >&2
    return 1
}

for i in 1 2 3; do
    write_upstream "$i"
    sleep 2
    docker stop "depwork-api-$i" >/dev/null
    docker rm "depwork-api-$i" >/dev/null
    docker run -d --name "depwork-api-$i" --network depwork-net --stop-timeout 40 \
        -e WARMUP_SECONDS=3 "depwork-bookstore-api:$version" >/dev/null
    wait_ready "$i"
    write_upstream none
    echo "$(date +%T) api-$i -> $version ready"
done

Two details matter here. Nginx sends no new requests to a server marked down. nginx -s reload starts new workers with the new config, while the old workers finish their requests and exit, so the reload itself is a drain. Since Nginx 1.29.6 the open source version also has a drain parameter, but according to the documentation only requests "bound" to that server (stuck to it with a sticky session) go to it in that mode. If you don't use sticky sessions, it behaves the same as down.

The first version of the script had a bug, and its output looked like this:

06:28:51 api-3 -> 1.0.0 ready
06:28:57 api-3 -> 1.0.0 ready
06:29:04 api-3 -> 1.0.0 ready

"api-3" was updated in all three steps, and api-1 and api-2 were never touched. The cause is bash variable scope: the for i in 1 2 3 loop inside write_upstream overwrote the outer loop's i and left it at 3. Adding local i flag to the function was enough. I mention it because deploy scripts are code too. An untested rolling script can update a third of the fleet three times and finish with "success". A final check that verifies each instance's version would have caught this bug immediately.

I ran the fixed script from 1.0.0 to 1.1.0 with the same two probes:

06:29:35 api-1 -> 1.1.0 ready
06:29:41 api-2 -> 1.1.0 ready
06:29:46 api-3 -> 1.1.0 ready
GET
200 237
POST
201 90

237 GETs and 90 POSTs, all successful. The second-by-second breakdown of the GET probe shows the mixed-version window:

   8 06:29:34 200 1.0.0
   6 06:29:35 200 1.0.0
   2 06:29:35 200 1.1.0
   5 06:29:36 200 1.0.0
   3 06:29:36 200 1.1.0
   4 06:29:37 200 1.0.0
   4 06:29:37 200 1.1.0
   3 06:29:38 200 1.0.0
   5 06:29:38 200 1.1.0
   5 06:29:39 200 1.0.0
   4 06:29:39 200 1.1.0
   3 06:29:40 200 1.0.0
   5 06:29:40 200 1.1.0
   8 06:29:41 200 1.1.0

Between 06:29:35 and 06:29:40 the same client got both 1.0.0 and 1.1.0 answers every second. That six-second window is for three instances and an app that starts fast. In a fleet of twenty instances that each take a minute to warm up, the window reaches half an hour.

Consequences of the Mixed-Version Window

The real question with rolling isn't "do errors happen?" but "do the two versions break each other during this window?" Typical breakages:

• The new version renamed a JSON (JavaScript Object Notation) field. The Angular app gets one response from v2, sends the next request to v1, and the field disappears.
• The new version writes to the cache in a new format, and the old version reads the same key and fails to deserialize it.
• The new version drops a new message type into a queue, and a consumer that hasn't been updated yet doesn't recognize it.
• A migration added a new NOT NULL column, and the old version's INSERTs fail.

The cure for all of them is the N and N-1 compatibility from the second post of the series: every change must be correct while the old version is still running. If you can't guarantee that, rolling is the wrong strategy. Either you split the change into backward-compatible steps (for the database, expand-contract in the sixth post), or you move to blue-green, which switches traffic in one move.

Your Decision in Kubernetes: maxSurge, maxUnavailable and Timing

In Kubernetes the Deployment controller does the rolling; how ReplicaSets grow and shrink is covered in the Deployments and ReplicaSets post of the upcoming Kubernetes series. Here we only discuss the strategy decision. A few lines carry it:

spec:
  replicas: 4
  minReadySeconds: 5
  progressDeadlineSeconds: 60
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

maxSurge says how many pods above the desired count you may go during a deploy, maxUnavailable how many below. Both default to 25%. When given as percentages, maxSurge rounds up and maxUnavailable rounds down, so with four replicas the default means one extra and one missing. Capacity drives the choice:

There are two more timing settings. minReadySeconds (default 0) sets how many seconds a new pod must stay healthy after becoming ready before it counts as "available". A value between 5 and 30 seconds is good insurance for catching a version that crashes a few seconds after startup before the rollout continues. progressDeadlineSeconds (default 600) says how long a rollout may go without progress before it is considered stuck.

With these settings I moved four replicas to 1.1.0 on kind:

kubectl set image deploy/bookstore-api api=depwork-bookstore-api:1.1.0
kubectl rollout status deploy/bookstore-api --timeout=120s

deployment.apps/bookstore-api image updated
Waiting for deployment "bookstore-api" rollout to finish: 1 out of 4 new replicas have been updated...
Waiting for deployment "bookstore-api" rollout to finish: 2 out of 4 new replicas have been updated...
Waiting for deployment "bookstore-api" rollout to finish: 3 out of 4 new replicas have been updated...
Waiting for deployment "bookstore-api" rollout to finish: 1 old replicas are pending termination...
deployment "bookstore-api" successfully rolled out

The output is shortened; the rollout took 37 seconds. Each new pod waited for the three-second warm-up, the readiness probe and the five-second minReadySeconds, and then one old pod was terminated. Now let's try a version that never becomes ready:

kubectl set env deploy/bookstore-api WARMUP_SECONDS=100000
kubectl rollout status deploy/bookstore-api --timeout=150s

deployment.apps/bookstore-api env updated
Waiting for deployment "bookstore-api" rollout to finish: 1 out of 4 new replicas have been updated...
error: deployment "bookstore-api" exceeded its progress deadline

NAME                             READY   STATUS    RESTARTS   AGE
bookstore-api-597cdc7fb4-62s9l   0/1     Running   0          61s
bookstore-api-d996bbcc4-6m22j    1/1     Running   0          99s
bookstore-api-d996bbcc4-lvdn4    1/1     Running   0          90s
bookstore-api-d996bbcc4-r6mtw    1/1     Running   0          118s
bookstore-api-d996bbcc4-r7zdm    1/1     Running   0          108s

Available=True MinimumReplicasAvailable: Deployment has minimum availability.
Progressing=False ProgressDeadlineExceeded: ReplicaSet "bookstore-api-597cdc7fb4" has timed out progressing.

Here is the answer to the second opening question: Kubernetes did not roll back. It marked the rollout "ProgressDeadlineExceeded" and stopped. Thanks to maxUnavailable: 0, the four old pods keep carrying traffic and users noticed nothing. But the Deployment sits half-finished. Going back (kubectl rollout undo) or forward is the job of you, your pipeline, or a tool like Argo Rollouts. The non-zero exit code of kubectl rollout status is exactly the right signal for the pipeline to stop here.

When Is Rolling the Wrong Choice?

When there is a single instance. Without a second instance to update in turn, rolling is just recreate under a different name. Either add a second instance or do blue-green.

When the schema change isn't backward compatible. If new code needs the new schema and old code needs the old one, the two versions can't live side by side.

When there is session or in-memory state. Swapping instances in an app that isn't stateless loses users (the second post of the series).

When a fast and complete rollback is a requirement. A rolling deploy's rollback is also rolling; it can't compete with blue-green's one-switch rollback.

When you want to try the new version with a small share of real users first. Rolling's "percentage" depends on the instance count and doesn't stick to users; that's what canary is for.

In Production

Three numbers tell you how healthy a rolling deploy is. First, the 5xx and dropped connection count in the deploy window; with drain and graceful shutdown set up properly it should be zero, and if not, review the settings against the chain in the second post of the series. Second, the length of the mixed-version window: instance count × (drain + startup + readiness + minReadySeconds) / instances changed at once. The longer it gets, the bigger the compatibility risk. Third, the error rate on the first instance of the new version. The first instance is effectively a free canary; don't move on to the second one without looking at its metrics.

As the fleet grows, the script needs two additions: making the number of instances changed at once (the batch size) a parameter, and adding a threshold, for example stopping and reverting if the error rate rose after the first batch. That's the hand-written version of what Kubernetes does with progressDeadlineSeconds.

Common Pitfalls

• Stopping an instance without taking it out of the load balancer. Even with good graceful shutdown, the load balancer keeps sending requests to the stopping instance.
• Skipping the difference between "the container is running" and "ready". Moving to the next instance because docker run returned successfully is the shortest path to spreading a broken version across the fleet.
• Not testing the deploy script. A bug like the local one above leaves the fleet half-updated while the report is green.
• Thinking Kubernetes rolls back automatically when progressDeadlineSeconds is exceeded. It doesn't; it marks the rollout as failed and waits.
• Leaving the maxUnavailable percentage unexamined in small deployments. With three replicas 25% rounds down to 0 and maxSurge 25% rounds up to 1. The behavior may or may not match what you expect; write the number explicitly.

A Small Experiment

★☆☆ Add a check at the end of rolling.sh that reads /api/version from each instance and verifies that all of them are on the target version. Delete the local i flag line to reproduce the bug and watch the check catch it.

★★☆ Change the script to take a BATCH_SIZE parameter (how many instances are updated at once) and measure the length of the mixed-version window with six instances for 1, 2 and 3.

★★★ On kind, try maxSurge: 0, maxUnavailable: 1 and maxSurge: 100%, maxUnavailable: 0 for a Deployment with four replicas. Record how the ReplicaSet counts change in the kubectl get rs -w output during the rollout and compare how many seconds each takes.

Rolling is cheap, but it keeps two versions side by side for a long time and its rollback is slow. In the next post we buy our way out of both weaknesses: a blue-green deployment that sets up two separate environments with Nginx, switches traffic with a single reload, and switches back with the same reload.