Environments with Kustomize: Bases, Overlays, Image Tags and configMapGenerator

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, kubectl 1.37 (with Kustomize 5.8.1 built in), standalone Kustomize 5.8.3 and .NET 10.

Guess First

You add one line to the prod overlay: commonLabels, which puts the app.kubernetes.io/version: "1.2.0" label on every resource. kubectl apply -k returns an error: the Deployment's selector is immutable. The question: since you got an error, nothing changed, right? Is the service still taking traffic? The answer is the most expensive lesson in this post, and it's below with real output.

The Problem: Three Environments, Three Copies

In earlier posts we wrote the Bookstore manifests for a single environment. In reality there are at least three: dev with one replica and discounted test prices, staging with settings close to prod, and prod with an HPA (Horizontal Pod Autoscaler), a PDB (PodDisruptionBudget) and bigger resources. The first reflex is to copy the folder three times. Six months later the readiness probe is fixed in dev but still old in prod; an env variable added to staging is forgotten in prod. Copies drift over time, and the drift is only noticed in production.

Kustomize solves this without templates: shared manifests live in a base, each environment writes only its differences in an overlay, and the result is produced as plain YAML (YAML Ain't Markup Language). No template language, no variables; every file is a valid Kubernetes manifest on its own. Kustomize ships inside kubectl (kubectl apply -k, kubectl kustomize); the standalone binary brings a newer version and the edit commands. Our kubectl 1.37.2 had Kustomize 5.8.1 inside, the standalone binary was 5.8.3; having CI (Continuous Integration) and your local machine use the same version keeps small differences from becoming surprises.

Base: The Environment-Independent Core

The folder structure:

deploy/
  base/
    kustomization.yaml
    deployment.yaml
    service.yaml
    appsettings.json
  components/
    autoscaling/
      kustomization.yaml
      hpa.yaml
      pdb.yaml
  overlays/
    dev/
      kustomization.yaml
      appsettings.json
    prod/
      kustomization.yaml
      appsettings.json
      resources.yaml

deployment.yaml and service.yaml in the base are the same as in earlier posts, written without a namespace. The real work is in kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
labels:
  - pairs:
      app.kubernetes.io/part-of: bookstore
    includeSelectors: false
    includeTemplates: true
configMapGenerator:
  - name: api-config
    files:
      - appsettings.json
images:
  - name: bookstore-api
    newTag: 1.0.0

The labels transformer adds a label to every resource and (with includeTemplates) to pod templates, but with includeSelectors: false it doesn't touch selectors. That detail is half of the answer to Guess First. configMapGenerator produces a ConfigMap from the appsettings.json file; as in the seventh post of the series, the API (Application Programming Interface) reads that file from /app/config with reloadOnChange. images sets the tag of the bookstore-api image in the manifests from a single place.

Overlays: Only the Difference

The dev overlay changes the namespace, the config and the environment name:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: bookstore-dev
resources:
  - ../../base
configMapGenerator:
  - name: api-config
    behavior: merge
    files:
      - appsettings.json
patches:
  - target:
      kind: Deployment
      name: api
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/0/value
        value: Development

The patch here is a JSON (JavaScript Object Notation) Patch (RFC 6902, RFC: Request for Comments); good for small, pinpoint changes, but fragile because it depends on array positions (containers/0/env/0). The prod overlay uses a strategic merge patch instead: a partial copy of the Kubernetes object, with containers matched by name:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: bookstore-prod
resources:
  - ../../base
components:
  - ../../components/autoscaling
configMapGenerator:
  - name: api-config
    behavior: merge
    files:
      - appsettings.json
patches:
  - path: resources.yaml
images:
  - name: bookstore-api
    newTag: 1.1.0

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  template:
    spec:
      containers:
        - name: api
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi

components are for optional pieces shared by several overlays. We put the HPA and PDB in a component; staging and prod use it, dev doesn't. The component also removes the replicas field from the Deployment when there is an HPA; it's the Kustomize counterpart of the "every apply overrides the HPA's decision" pitfall we saw in the eleventh post:

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
  - hpa.yaml
  - pdb.yaml
patches:
  - target:
      kind: Deployment
      name: api
    patch: |-
      - op: remove
        path: /spec/replicas

To see the result without sending it to the cluster, run kubectl kustomize overlays/prod. The relevant part of the output:

kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/part-of: bookstore
  name: api-config-tmh5dh7hbh
  namespace: bookstore-prod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/part-of: bookstore
  name: api
  namespace: bookstore-prod
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: api
  template:
    ...
        image: bookstore-api:1.1.0
        ...
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 256Mi
      ...
      volumes:
      - configMap:
          name: api-config-tmh5dh7hbh
        name: config
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
  namespace: bookstore-prod

Three things stand out. The Deployment has no replicas. A hash has been appended to the ConfigMap's name, and the reference in the Deployment was updated automatically. The PDB and HPA don't have the part-of label: the base's labels only apply to the base's resources, not to those coming from a component in the overlay. If you want the label everywhere, move the transformer to the overlay. We applied both environments and asked each for its config:

{"pod":"api-68775884bd-xgknm","version":"1.0.0","environment":"Development","storeName":"Bookstore (dev)","discountPercent":50,"database":"in-memory"}
{"pod":"api-7c766654d9-qgm5d","version":"1.1.0","environment":"Production","storeName":"Bookstore","discountPercent":10,"database":"in-memory"}

configMapGenerator and the Hash Suffix: Automatic Rollout on Config Change

We saw it in the seventh post: when a mounted ConfigMap changes, pods don't restart; the config either propagates slowly through the file or not at all (subPath, env). The hash suffix fixes this at the root: if the content changes, the ConfigMap's name changes, the changed reference changes the pod template, and a Deployment whose pod template changes does a rolling update. We raised the prod discount from 10 to 15 and looked with kubectl diff -k first:

-          name: api-config-tmh5dh7hbh
+          name: api-config-86ddt625g5
+apiVersion: v1
+data:
+  appsettings.json: |
+    {
+      "Bookstore": {
+        "StoreName": "Bookstore",
+        "DiscountPercent": 15
+      }
+    }
+kind: ConfigMap

configmap/api-config-86ddt625g5 created
service/api unchanged
deployment.apps/api configured
deployment "api" successfully rolled out
NAME                    DATA   AGE
api-config-86ddt625g5   1      12s
api-config-tmh5dh7hbh   1      28s
{"pod":"api-57577764d5-hnlf7","version":"1.1.0","environment":"Production","storeName":"Bookstore","discountPercent":15,"database":"in-memory"}

The old ConfigMap wasn't deleted. That's both good and bad: good, because kubectl rollout undo goes back to the old ReplicaSet's template and that template points at the old ConfigMap; config is rolled back together with code. Bad, because every config change leaves a ConfigMap behind. Cleaning up needs the prune feature of a GitOps tool (in the upcoming GitOps and Argo CD (CD: Continuous Delivery) series) or an occasional manual cleanup; plain kubectl apply doesn't do it.

Setting the Image Tag from CI

After the pipeline builds the new image and pushes it to the registry, it updates the tag in the overlay. Change the file with kustomize edit, not sed:

cd deploy/overlays/prod
kustomize edit set image bookstore-api=bookstore-api:1.2.0

 images:
-  - name: bookstore-api
-    newTag: 1.1.0
+- name: bookstore-api
+  newName: bookstore-api
+  newTag: 1.2.0

That's only part of the diff. kustomize edit rewrote the whole file in its own format: every list indentation changed and the keys in configMapGenerator became alphabetical. To avoid this noise in every commit the pipeline makes, keep your kustomization.yaml files in Kustomize's format from the start, or run kustomize edit fix once and commit that. Using a digest instead of a tag in production (kustomize edit set image bookstore-api=registry.example.com/bookstore-api@sha256:...) also closes the risk of the same tag being pushed again. The GitHub Actions side of this step is in the deploy posts of the GitHub Actions series, and moving the image to the cluster with GitOps is in the upcoming GitOps and Argo CD series.

Pitfall: commonLabels and a Half-Finished Apply

The line from Guess First:

commonLabels:
  app.kubernetes.io/version: "1.2.0"

# Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
configmap/api-config-86ddt625g5 configured
service/api configured
poddisruptionbudget.policy/api configured
horizontalpodautoscaler.autoscaling/api configured
The Deployment "api" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/name":"api","app.kubernetes.io/version":"1.2.0"}}: field is immutable

commonLabels adds the label to selectors as well. Since the Deployment's selector is immutable, the Deployment was rejected; but apply sends resources one by one, and the Service went first. A Service's selector is not immutable; it was updated with the new label, and now no pod matches it:

{"app.kubernetes.io/name":"api","app.kubernetes.io/version":"1.2.0"}
NAME        ADDRESSTYPE   PORTS     ENDPOINTS   AGE
api-v6xgn   IPv4          <unset>   <unset>     4m36s
wget: can't connect to remote host (10.96.228.122): Connection refused

The assumption "I got an error, so nothing changed" is wrong: the command failed, and the service was down. Putting a version number in a selector means changing the selector with every version; selectors must be built from labels that stay constant for the app's lifetime. Since Kustomize 5 the right way is labels with includeSelectors: false. After fixing the overlay and applying again, the endpoints came back.

A Short Comparison with Helm

Helm solves the same problem with templates: values from values.yaml are filled into Go templates. Helm shines at packaging and distribution: for an app others will install, with dozens of settings (an ingress controller, a database operator, an autoscaler), a chart is the natural format, plus release history and hooks. Kustomize shines at your own app's environment differences: files read as plain YAML, the output of kubectl kustomize is exactly what goes to the cluster, and there's no template language to learn. A common combination: third-party components with Helm charts, your own app with Kustomize; you can also render a chart through Kustomize's helmCharts field and patch on top of it. Details in the upcoming Helm series.

Production Notes

Overlay differences should stay small; if an overlay is longer than the base, the base is set up wrong. Rendering every overlay with kubectl kustomize in each pull request and passing it through a schema check (e.g. kubectl apply --dry-run=server) catches a broken patch before merge. Reviewing kubectl diff -k before going to prod is the cheapest way to spot unexpected selector changes like the one above. Putting Secrets into the repo as plain text with secretGenerator is tempting but wrong; external secret management is in the upcoming Configuration and Secret Management series. Setting the namespace in the overlay makes it harder to accidentally apply one environment's manifests to another namespace, but doesn't make checking your context unnecessary (the second post of the series). Finally, the version difference between kubectl's built-in Kustomize and a standalone Kustomize in CI can, rarely, produce different output; use locally whichever one your pipeline uses.

Common Pitfalls

Copying environment folders and letting them drift over time. Putting a changing label into selectors with commonLabels and taking the service down with a half-finished apply. Relying on array indexes in a JSON Patch and overwriting the wrong variable when the env order changes in the base. Forgetting to remove replicas in an overlay that uses an HPA. Writing the ConfigMap as a plain manifest instead of a generator and being surprised that config changes don't trigger a rollout. Changing the tag with sed in the pipeline and matching the wrong line. Thinking the base's labels apply to resources coming from a component or the overlay. And assuming "nothing changed" when kubectl apply returns an error.

A Small Experiment

★☆☆ Add a staging overlay: namespace bookstore-staging, the autoscaling component, prod's resource patch and DiscountPercent: 0. Compare the output of the three environments from kubectl kustomize with diff.

★★☆ Break the JSON Patch in the dev overlay by adding a new env variable to the base before ASPNETCORE_ENVIRONMENT. Then write the same change as a strategic merge patch and explain why it doesn't break.

★★★ Set up a way to clean old ConfigMaps from the overlays: try kubectl apply --prune with ApplySets and preview which resources would be deleted with --dry-run=server. Decide how many old versions to keep, taking into account that rollout undo needs the old ConfigMap.

The environments now live in separate namespaces and are generated from a single source. But the pods in these namespaces can still reach each other freely, run with the default ServiceAccount and may run as root. In the next post we turn to security: ServiceAccounts, RBAC (Role-Based Access Control), Pod Security Standards, securityContext and NetworkPolicy.