Kubernetes Deployments and ReplicaSets: Labels, Rolling Updates, Rollout History and Undo

Version note: The examples in this post were run with kind 0.33 on Kubernetes 1.37, kubectl 1.37 and .NET 10.

Guess First

Four replicas of the Bookstore API (Application Programming Interface) are running version 1.1.0. The Deployment has maxSurge: 1, maxUnavailable: 0 and progressDeadlineSeconds: 120. The CI (Continuous Integration) pipeline mistakenly deploys the 1.2.0 tag, which doesn't exist in the registry. How many pods serve traffic during the rollout: four, three or zero? Two minutes later, what does kubectl rollout status say, and does Kubernetes go back to the old version by itself?

Labels and Selectors: the Thread Between a Deployment and Its Pods

In the previous post we saw that we shouldn't create bare pods. Here is a near-production Deployment for the Bookstore API:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: bookstore
  labels:
    app.kubernetes.io/name: api
    app.kubernetes.io/part-of: bookstore
  annotations:
    kubernetes.io/change-cause: "release 1.0.0"
spec:
  replicas: 4
  revisionHistoryLimit: 5
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
        app.kubernetes.io/part-of: bookstore
    spec:
      containers:
        - name: api
          image: bookstore-api:1.0.0
          ports:
            - name: http
              containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            periodSeconds: 2

A Deployment recognises its pods by label, not by name. spec.selector says "pods carrying these labels are mine", and spec.template.metadata.labels stamps those labels on every pod it creates. The two must match; if they don't, the API server rejects the Deployment. For label names we use the common labels Kubernetes recommends, such as app.kubernetes.io/name and app.kubernetes.io/part-of; tools (dashboards, Helm, Argo CD (Continuous Delivery)) recognise them.

Don't worry about the readiness probe for now; the details are in post 8. Here it has a single job: giving the Deployment the "this pod is ready for traffic" signal. A rolling update is blind without it.

$ kubectl apply -f api-deployment.yaml
deployment.apps/api created
$ kubectl get pods -l app.kubernetes.io/name=api --show-labels
NAME                   READY   STATUS    RESTARTS   AGE   LABELS
api-6ddb5d7947-5zx5r   1/1     Running   0          8s    app.kubernetes.io/name=api,app.kubernetes.io/part-of=bookstore,pod-template-hash=6ddb5d7947
api-6ddb5d7947-bf4pj   1/1     Running   0          8s    app.kubernetes.io/name=api,app.kubernetes.io/part-of=bookstore,pod-template-hash=6ddb5d7947
api-6ddb5d7947-whqpn   1/1     Running   0          8s    app.kubernetes.io/name=api,app.kubernetes.io/part-of=bookstore,pod-template-hash=6ddb5d7947
api-6ddb5d7947-xk2jw   1/1     Running   0          8s    app.kubernetes.io/name=api,app.kubernetes.io/part-of=bookstore,pod-template-hash=6ddb5d7947

There's a label we didn't write: pod-template-hash=6ddb5d7947. The Deployment controller adds it, and it's also the middle part of the pod name. That hash is the key to the next section.

Selectors have one more property: in apps/v1 they're immutable. I tried adding a second label to the selector:

$ kubectl apply -f api-selector.yaml
The Deployment "api" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/name":"api","app.kubernetes.io/part-of":"bookstore"}}: field is immutable

The only way to change a selector is to delete and recreate the Deployment. So put only the application's identity (name, and component if needed) in the selector; add labels that may change, such as version, environment or team, to the template but keep them out of the selector.

The Role of the ReplicaSet

A Deployment doesn't create pods directly. For each distinct pod template it creates a ReplicaSet; the ReplicaSet creates the pods and keeps their count. The template's hash goes into both the ReplicaSet's name and the pods' pod-template-hash label, so pods of different versions of the same Deployment never get mixed up.

$ kubectl get rs -l app.kubernetes.io/name=api
NAME             DESIRED   CURRENT   READY   AGE
api-6ddb5d7947   4         4         4       8s

When you change the image, an environment variable, a probe or resources, in short anything under spec.template, a new hash and a new ReplicaSet are born. Changing replicas doesn't touch the template; the current ReplicaSet's count changes and no rollout starts. Old ReplicaSets are kept with 0 replicas; revisionHistoryLimit (default 10, 5 for us) decides how many are kept. Rollbacks use these stored ReplicaSets. You don't need to manage ReplicaSets by hand; if you scale a ReplicaSet owned by a Deployment, the Deployment controller corrects it right away.

How a Rolling Update Progresses

I changed the image to 1.1.0, updated the change-cause and recorded the ReplicaSet counts every second. Columns: ReplicaSet, desired, ready, available.

$ kubectl apply -f api-deployment-1.1.0.yaml
deployment.apps/api configured
03:46:15 api-6ddb5d7947   4     4        4|api-b995dcccb    1     <none>   <none>|
03:46:17 api-6ddb5d7947   4     4     4|api-b995dcccb    1     1     <none>|
03:46:21 api-6ddb5d7947   3     3     3|api-b995dcccb    2     1     1|
03:46:23 api-6ddb5d7947   3     3     3|api-b995dcccb    2     2     1|
03:46:27 api-6ddb5d7947   2     2     2|api-b995dcccb    3     2     2|
03:46:29 api-6ddb5d7947   2     2     2|api-b995dcccb    3     3     2|
03:46:31 api-6ddb5d7947   2     2     2|api-b995dcccb    3     3     3|
03:46:34 api-6ddb5d7947   1     1     1|api-b995dcccb    4     4     3|
03:46:38 api-6ddb5d7947   0     <none>   <none>|api-b995dcccb    4     4        4|

Let's read the parameters from this table. maxSurge: 1 allows at most 1 above the desired count (4), so 5 pods in total; in the first row 1 pod opened in the new ReplicaSet while the 4 old pods were still up. maxUnavailable: 0 means the number of available pods never drops below 4; so an old pod is only shut down after a new pod becomes available. minReadySeconds: 5 says "count it as available once it has stayed ready for 5 seconds without crashing": the pod that became ready at 03:46:17 counted as available at 03:46:21 and the old ReplicaSet dropped to 3. The four-pod rollout took 23 seconds and capacity never fell below 4 pods.

The defaults are maxSurge: 25% and maxUnavailable: 25%; the percentage is rounded up for surge and down for unavailable. With 4 replicas both come to 1, and with the defaults you may run one pod short during a rollout. For an API with tightly planned capacity maxUnavailable: 0 is safer; if the nodes have no room for an extra pod, use maxSurge: 0, maxUnavailable: 1 to stop first and start after. They can't both be 0.

The second strategy is Recreate: it shuts down all old pods first and then starts the new ones. It causes downtime, but it's the right choice when two versions can't run at the same time (for example a schema the old and new versions can't share, or a file with a single writer). Which strategy to choose when, and how it compares with blue-green and canary, is the subject of the Rolling Deployment post in the upcoming Deployment Strategies series; here we only look at the mechanics.

A Stuck Rollout and progressDeadlineSeconds

Now the opening question. I applied the non-existent 1.2.0 tag:

$ kubectl apply -f api-deployment-1.2.0.yaml
deployment.apps/api configured
$ kubectl rollout status deploy/api --timeout=180s
Waiting for deployment "api" rollout to finish: 0 out of 4 new replicas have been updated...
Waiting for deployment "api" rollout to finish: 1 out of 4 new replicas have been updated...
error: deployment "api" exceeded its progress deadline
$ kubectl get pods -l app.kubernetes.io/name=api
NAME                  READY   STATUS         RESTARTS   AGE
api-8596656d5-kvgvv   0/1     ErrImagePull   0          2m1s
api-b995dcccb-7p526   1/1     Running        0          2m21s
api-b995dcccb-gc6zd   1/1     Running        0          2m33s
api-b995dcccb-r6m9g   1/1     Running        0          2m27s
api-b995dcccb-x9nlk   1/1     Running        0          2m16s
$ kubectl get deploy api -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
Available=True MinimumReplicasAvailable: Deployment has minimum availability.
Progressing=False ProgressDeadlineExceeded: ReplicaSet "api-8596656d5" has timed out progressing.

The four old pods kept serving traffic throughout the rollout; maxSurge: 1 allowed only one new pod, which couldn't pull its image, and because of maxUnavailable: 0 no old pod was shut down. From the user's point of view there was no outage. That's the real value of maxUnavailable: 0: a bad version doesn't eat your capacity.

After 120 seconds the Deployment's Progressing condition turned False with reason ProgressDeadlineExceeded. kubectl rollout status exited with code 1; that's the signal that turns your pipeline's deploy step red. But note: Kubernetes doesn't roll back by itself. The deadline only reports a status; the broken ReplicaSet sits there in ErrImagePull. Going back is your job, or your pipeline's. Automatic rollback and rollouts that decide based on metrics are the territory of tools like Argo Rollouts and of the upcoming Deployment Strategies series.

Rollout History, Undo and Pause

$ kubectl rollout history deploy/api
deployment.apps/api
REVISION  CHANGE-CAUSE
1         release 1.0.0
2         release 1.1.0
3         release 1.2.0
$ kubectl rollout history deploy/api --revision=2
deployment.apps/api with revision #2
Pod Template:
  Labels:	app.kubernetes.io/name=api
	app.kubernetes.io/part-of=bookstore
	pod-template-hash=b995dcccb
  Annotations:	kubernetes.io/change-cause: release 1.1.0
  Containers:
   api:
    Image:	bookstore-api:1.1.0

The CHANGE-CAUSE column comes from the kubernetes.io/change-cause annotation on the Deployment. kubectl apply --record used to write it automatically; that flag is deprecated, so you write the annotation yourself, in the manifest or the pipeline. Now let's roll back:

$ kubectl rollout undo deploy/api
Warning: resource deployments/api was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.
deployment.apps/api rolled back
$ kubectl rollout history deploy/api
REVISION  CHANGE-CAUSE
1         release 1.0.0
3         release 1.2.0
4         release 1.1.0
$ kubectl get rs -l app.kubernetes.io/name=api
NAME             DESIRED   CURRENT   READY   AGE
api-6ddb5d7947   0         0         0       2m55s
api-8596656d5    0         0         0       2m7s
api-b995dcccb    4         4         4       2m39s

Three things stand out. First, a rollback creates a new revision: revision 2 disappeared and the same template came back as revision 4. Second, the rollback is fast: the b995dcccb ReplicaSet already existed and was simply scaled to 4. Third, and most important, kubectl 1.37's warning: rollout undo changes the cluster but not the file in Git. If the file still says 1.2.0, the next kubectl apply brings the broken version back. In production the real rollback is fixing the manifest in Git and applying it again; rollout undo is a tool that buys you time in an emergency.

To bundle several changes into a single rollout, use pause:

$ kubectl rollout pause deploy/api
deployment.apps/api paused
$ kubectl set env deploy/api Bookstore__StoreName="Bookstore Istanbul"
deployment.apps/api env updated
$ kubectl set resources deploy/api -c api --requests=cpu=100m,memory=128Mi
deployment.apps/api resource requirements updated
$ kubectl rollout resume deploy/api
deployment.apps/api resumed
$ kubectl rollout history deploy/api
REVISION  CHANGE-CAUSE
1         release 1.0.0
3         release 1.2.0
4         release 1.1.0
5         release 1.1.0

The two changes produced a single new ReplicaSet and a single revision (5). But revision 5's change-cause is wrong: since I didn't update the annotation, the old value on the Deployment was copied. If you want to trust the history, you need a pipeline that writes the change-cause on every change.

Image Tags and imagePullPolicy

Rolling updates and rollbacks rely on knowing exactly which image each revision runs. A moving tag like bookstore-api:latest breaks that assumption: if revision 3 and revision 4 point at the same tag, "undo" undoes nothing, and different nodes may have pulled the tag at different moments. As we saw in post 2, :latest also brings imagePullPolicy: Always by default.

Two rules are enough for production. Give every build a tag that never changes: a semantic version (1.1.0), the commit SHA (Secure Hash Algorithm) or both (1.1.0-3f2c9ab). Disable overwriting tags in the registry (most registries have an "immutable tags" setting). For a stronger guarantee, reference the image by digest: registry.example.com/bookstore-api@sha256:.... A digest is the hash of the content and can't change; we'll set tags or digests from CI with Kustomize in post 14.

Production Notes

A rolling update is safe when three things hold: a correct readiness probe (a pod doesn't get traffic before it's really ready), proper graceful shutdown (requests aren't cut off while an old pod stops; post 10) and backward-compatible changes (during the rollout the old and new versions serve the same database and the same clients at the same time). The third is especially critical for schema changes; the expand-contract approach is in the upcoming Deployment Strategies series.

Set progressDeadlineSeconds according to the application's realistic startup time and wait in the pipeline with kubectl rollout status --timeout. The default is 600 seconds; for a fast-starting API that means not noticing a broken deploy for ten minutes.

Plan capacity with the rollout in mind. The nodes need CPU (Central Processing Unit) and memory room for maxSurge extra pods; otherwise the new pod stays Pending and the rollout hits the deadline.

Common Pitfalls

Putting labels that may change into the selector. If a version label is in the selector, you'll have to delete the Deployment on the next version. Keep the selector minimal.

Two Deployments sharing a selector. If two Deployments select pods with the same labels, they try to adopt each other's pods. Each Deployment's label set must be unique.

Not fixing the file after rollout undo. Take the warning above seriously; otherwise the next apply brings the broken version back.

Rolling updates without a readiness probe. Without a probe a container counts as ready the moment it starts; the old pod is shut down before the application has even opened its port, and you see brief 502s.

Trying to maintain change-cause by hand. A forgotten annotation produces a wrong history, like revision 5. Either have the pipeline write it or treat Git as the source of history.

A Small Experiment

★☆☆ Apply the Deployment with the default maxSurge: 25%, maxUnavailable: 25% and record the same 1.0.0 → 1.1.0 rollout. Find the moments when the available pod count drops to 3.

★★☆ Do the same rollout with strategy.type: Recreate and, while running while true; do curl -s -o /dev/null -w "%{http_code}\n" localhost:18710/; sleep 0.2; done in a terminal (the port-forward should target the Service; see post 5), count how many requests fail.

★★★ Write a small bash script that deploys a non-existent tag, runs kubectl rollout undo if kubectl rollout status --timeout=60s fails, and exits with code 1. Then show the gap this approach creates between Git and the cluster with kubectl diff.

Our pods now update in a controlled way, but every new pod comes with a new IP (Internet Protocol) address. In the next post we look at the Service, which puts a stable name in front of these changing addresses: ClusterIP, NodePort, LoadBalancer, cluster DNS (Domain Name System) and EndpointSlices.