.NET Configuration with ConfigMaps and Secrets: Environment Variables, Volumes and Picking Up Changes

Version note: The examples in this post were run with kind 0.33 on Kubernetes 1.37, kubectl 1.37, .NET 10 (ASP.NET Core 10.0.14) and Angular 22.

Guess First

The Bookstore API's (Application Programming Interface) store name lives in a ConfigMap; the ConfigMap is mounted into the pod as a volume and the application reads the file with reloadOnChange: true. You change the name in the ConfigMap from "Bookstore Ankara" to "Bookstore Izmir". When does /api/info return the new name: right away, within a minute, or never? What happens in a pod that mounts the same file with subPath? Last question: if the same setting is in both an environment variable and this file, which one wins?

Two Ways from a ConfigMap to .NET Configuration

A ConfigMap is Kubernetes' standard way to hold non-secret configuration as key/value pairs. A pod can consume it in two ways: as environment variables or as files in a volume. ASP.NET Core's configuration system reads both naturally; the mechanism is covered in the Configuration and Options Pattern post of the ASP.NET Core Web API series. On the Kubernetes side the only rule you need is the double underscore: since environment variable names can't contain :, .NET treats __ as the hierarchy separator. ConnectionStrings__Default maps to the ConnectionStrings:Default key.

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-config
  namespace: bookstore
data:
  appsettings.json: |
    {
      "Bookstore": {
        "StoreName": "Bookstore Ankara",
        "DiscountPercent": 10
      }
    }

In the Deployment we use both: the connection string comes from a Secret as an environment variable, the store settings come from the ConfigMap as a file. For comparison I also pass the store name through an environment variable:

      containers:
        - name: api
          image: bookstore-api:1.0.0
          env:
            - name: ConnectionStrings__Default
              valueFrom:
                secretKeyRef:
                  name: api-db
                  key: connection-string
            - name: Bookstore__StoreName
              value: "Bookstore (from env)"
          volumeMounts:
            - name: config
              mountPath: /app/config
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 384Mi
      volumes:
        - name: config
          configMap:
            name: api-config

We'll unpack the resource settings in post 9; why they're here is explained in the Production Notes at the end of this post. On the application side there's a single line: adding the mounted file to configuration.

var builder = WebApplication.CreateBuilder(args);

// optional config directory mounted from a ConfigMap
builder.Configuration.AddJsonFile("config/appsettings.json", optional: true, reloadOnChange: true);

builder.Services.Configure<StoreOptions>(builder.Configuration.GetSection("Bookstore"));

The /api/info endpoint reads the value through IOptionsMonitor<StoreOptions>.CurrentValue, so it sees the new value if configuration is reloaded. With IOptions<T> the value would stay fixed for the application's lifetime.

Provider Order: Who Wins?

$ kubectl exec client -- wget -qO- http://api/api/info
{"pod":"api-55dfb49c7d-7d767","version":"1.0.0","environment":"Production","storeName":"Bookstore Ankara","discountPercent":10,"database":"postgres"}

Even though the environment variable says "Bookstore (from env)", the file's "Bookstore Ankara" won. The answer to the last question lies in provider order: WebApplication.CreateBuilder adds the default providers (appsettings.json, appsettings.{Environment}.json, environment variables, command line); our AddJsonFile call comes after them, so it sits at the end of the list, and a provider added later overrides earlier values. Most of the time that's not what you want: on Kubernetes you usually treat environment variables as "the last word". There are two fixes. Either rearrange the provider list so the mounted file is added before environment variables, or supply each setting from a single source only. For Bookstore we choose the second: store settings in the file, connection details and small environment-specific values in environment variables.

Inside the Volume Mount: the ..data Symlink and subPath

The API image is chiseled, so there's no ls in it; I mounted the same ConfigMap into a busybox pod in two ways: as a whole directory at /config, and as a single file via subPath at /single.

$ kubectl exec cm-viewer -- ls -la /config /single
/config:
drwxrwxrwx    3 root     root          4096 Nov  8 04:21 .
drwxr-xr-x    1 root     root          4096 Nov  8 04:22 ..
drwxr-xr-x    2 root     root          4096 Nov  8 04:21 ..2026_11_08_04_21_56.280075211
lrwxrwxrwx    1 root     root            31 Nov  8 04:21 ..data -> ..2026_11_08_04_21_56.280075211
lrwxrwxrwx    1 root     root            23 Nov  8 04:21 appsettings.json -> ..data/appsettings.json

/single:
-rw-r--r--    1 root     root            88 Nov  8 04:21 appsettings.json

In the directory mount, appsettings.json isn't a real file: it's a symlink to ..data/appsettings.json, and ..data is a second symlink pointing at a timestamped directory. When the ConfigMap changes, the kubelet writes a new timestamped directory and switches the ..data symlink to it in a single atomic operation. The application never sees a half-written file. The subPath mount, on the other hand, binds the file's current copy directly; it stays outside this atomic swap mechanism.

When Does a Change Reach the Pod?

I changed the ConfigMap to "Bookstore Izmir" and every 3 seconds recorded both files in the busybox pod and the /api/info responses of both API pods:

apply at 04:25:18
04:25:19 volume/subPath=Bookstore Izmir/Bookstore Ankara/ api= Bookstore Ankara Bookstore Ankara
# the same line for 3 minutes
04:28:29 volume/subPath=Bookstore Izmir/Bookstore Ankara/ api= Bookstore Ankara Bookstore Ankara
$ kubectl debug api-687f7ddc54-mfmgj --image=busybox:1.37 --target=api --profile=general -q -- cat /proc/1/root/app/config/appsettings.json
{
  "Bookstore": {
    "StoreName": "Bookstore Izmir",
    "DiscountPercent": 10
  }
}

Three separate results. The directory mount got updated. The subPath mount never did; the Kubernetes documentation states this explicitly. The real surprise is in the application: the file inside the API pod carries the new value too (I read it with an ephemeral debug container attached to the shell-less pod; details in post 16), yet the application with reloadOnChange: true still returns the old name three minutes later. .NET's default file watcher uses inotify on Linux, and in this setup it didn't notice the update the kubelet makes by swapping symlinks.

I repeated the experiment with the DOTNET_USE_POLLING_FILE_WATCHER=true environment variable. This setting switches the file provider from waiting for events to checking the file periodically (every four seconds):

apply at 04:28:54
04:29:04 volume/subPath=Bookstore Izmir/Bookstore Ankara/ api= Bookstore Izmir Bookstore Bursa
04:29:59 volume/subPath=Bookstore Bursa/Bookstore Ankara/ api= Bookstore Izmir Bookstore Bursa
04:30:15 volume/subPath=Bookstore Bursa/Bookstore Ankara/ api= Bookstore Bursa Bookstore Bursa

This time both API pods saw the new name ("Bookstore Bursa"): one after 10 seconds, the other after 81. The difference comes from the kubelet, not the application: the file of the API pod on the same node as the busybox pod was only updated around 04:29:59. According to the documentation, the delay can be as long as "kubelet sync period + cache propagation delay"; with default settings that's on the order of a minute. We saw the same on the Angular side, below.

Environment variables are never updated: they're read when the process starts, and even if the ConfigMap changes, the old value stays until the pod restarts. In table form:

Triggering a Rollout on Change

Live reload looks attractive, but in production most teams prefer the opposite: renewing pods through a controlled rollout when configuration changes. The reason is simple: if the new configuration is broken, the rolling update and readiness probes catch the error on the first pod while the old pods keep carrying traffic. With live reload, the broken value reaches all pods at roughly the same time and there's no way back.

The simplest way is kubectl rollout restart:

$ kubectl rollout restart deploy/api
deployment.apps/api restarted
$ kubectl get deploy api -o jsonpath='{.spec.template.metadata.annotations}'
{"kubectl.kubernetes.io/restartedAt":"2026-11-08T07:33:32+03:00"}

All the command does is add an annotation to the pod template; since the template changes, a new ReplicaSet and a normal rolling update start. There are two common ways to automate this. The first is writing a hash of the ConfigMap's content into an annotation on the pod template (known as checksum/config in Helm charts): when the content changes, the annotation, and therefore the template, changes. The second is appending the content hash to the ConfigMap's name (for example api-config-7g2k9m4b5c) and having the Deployment reference that name: every change means a new ConfigMap and a new reference. Kustomize's configMapGenerator does this for you; we'll see it in post 14.

The second approach pairs very well with immutable ConfigMaps. immutable: true, stable since 1.21, says the ConfigMap's content can never change again; the kubelet stops watching it, which reduces API server load in clusters with many ConfigMaps:

$ kubectl apply -f immutable.yaml
configmap/api-config-v1 created
$ sed 's/Ankara/Izmir/' immutable.yaml | kubectl apply -f -
The ConfigMap "api-config-v1" is invalid: data: Forbidden: field is immutable when `immutable` is set

To change it you create a new ConfigMap under a new name; since old ReplicaSets keep referencing the old ConfigMap, rollout undo rolls back the configuration too.

Secrets: base64 Isn't Encryption

A Secret is the ConfigMap's sibling for sensitive data; the ways to consume it (env, volume) are the same. But the name "secret" can mislead:

$ kubectl create secret generic demo-secret --from-literal=password='not-a-real-password' --dry-run=client -o yaml
apiVersion: v1
data:
  password: bm90LWEtcmVhbC1wYXNzd29yZA==
kind: Secret
metadata:
  name: demo-secret
$ echo bm90LWEtcmVhbC1wYXNzd29yZA== | base64 -d
not-a-real-password

base64 is an encoding, not encryption; it only lets binary data fit into YAML (YAML Ain't Markup Language). The Kubernetes documentation warns explicitly: Secrets are stored unencrypted in etcd by default; anyone with API access, anyone with access to etcd and anyone who can create a pod in a namespace (by mounting the Secret into the pod) can read the Secrets in that namespace. Really protecting a Secret takes several layers: narrowing get/list permission on Secrets with RBAC (Role-Based Access Control), enabling encryption at rest on the API server (usually with a KMS (Key Management Service) on managed clusters), keeping Secret YAMLs out of Git and, where possible, syncing the values from a secret store outside the cluster (Azure Key Vault, AWS (Amazon Web Services) Secrets Manager, HashiCorp Vault). That last topic belongs to the upcoming Configuration and Secret Management series; External Secrets Operator and the CSI (Container Storage Interface) Secret Store driver live there.

A small but important habit on the .NET side: don't log connection strings. Debug endpoints that dump all of IConfiguration and "print every setting at startup" code carry Secrets into your log system, and logs are usually open to a much wider audience than Secrets.

Serving Angular's Runtime config.json from a ConfigMap

An Angular application is compiled at build time; an API address written into environment.ts gets baked into the image and every environment needs its own image. For Bookstore we use runtime configuration instead: on startup the application reads /config/config.json with fetch and hands the values out through an InjectionToken. The file itself comes from a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  namespace: bookstore
data:
  config.json: |
    {
      "apiBaseUrl": "",
      "storeName": "Bookstore"
    }

The web Deployment mounts it into the /usr/share/nginx/html/config directory, and nginx returns a Cache-Control: no-store header for this file so the browser doesn't cache old configuration. Since it's a directory mount, the update arrived without restarting the pod:

$ kubectl create configmap web-config --from-literal=config.json='{"apiBaseUrl": "", "storeName": "Bookstore Izmir"}' --dry-run=client -o yaml | kubectl apply -f -
configmap/web-config configured
applied 04:32:21
04:32:21 {  "apiBaseUrl": "",  "storeName": "Bookstore"}
04:33:32 {"apiBaseUrl": "", "storeName": "Bookstore Izmir"}

71 seconds. The same image can now run with a different config.json in every environment. Never put a secret in this file: everything that goes to the browser is public.

Production Notes

Treat configuration changes like deploys: keep them in Git, put them through review, trigger a rollout with a hashed name or a checksum annotation. Consider live reload only for settings that genuinely need to change instantly and won't do damage if broken (log level, feature flags); if you use it, set DOTNET_USE_POLLING_FILE_WATCHER and measure on your own cluster.

During this post's experiments the API pods restarted with OOMKilled on the heavily loaded local cluster; at that moment the pods had no resource requests at all. When a node runs short of memory, the kernel kills pods without requests (BestEffort) first. That's why I added requests to the pods; we'll measure this in detail in post 9.

Restrict access to Secrets at the namespace and RBAC level, verify that encryption at rest is enabled and make sure the application doesn't log Secret values.

Common Pitfalls

Mounting with subPath and expecting updates. subPath is handy for putting a single file into an existing directory (for example /app/appsettings.Production.json), but that file is never updated when the ConfigMap changes. Mount into a separate directory and add the file from there.

Overwriting the application's directory with a directory mount. If you mount the ConfigMap at /app, every file from the image (DLLs included) becomes invisible and the application won't start. Always use a separate, empty directory.

Using a single underscore. ConnectionStrings_Default isn't a hierarchical key for .NET; it's silently ignored and the application fails with "connection string not found". Two underscores.

Putting secrets in a ConfigMap. ConfigMaps are usually opened with broader read permissions, and their values show up in kubectl describe output. Every sensitive value goes into a Secret.

Forgetting provider order. As you saw above, a JSON (JavaScript Object Notation) file added later can override an environment variable; the classic cause of "I changed the setting in env but nothing changed".

A Small Experiment

★☆☆ Write a Deployment that mounts api-config with subPath at /app/config/appsettings.json, change the ConfigMap and see /api/info return the old value. Then get the new value with kubectl rollout restart.

★★☆ In Program.cs, rearrange the provider list so the mounted file is added before the environment variable provider (hint: builder.Configuration.Sources is a list). Verify with the same Deployment that the environment variable now wins.

★★★ Prepare a small bash script that computes the SHA (Secure Hash Algorithm) 256 hash of the ConfigMap's content and writes it into the pod template as a checksum/config annotation. Show that a new ReplicaSet is born when you change the ConfigMap and run the script, and that none is born when you run it without changes.

The application now runs with the right configuration. But how does Kubernetes know it's actually healthy? In the next post we wire startup, liveness and readiness probes to ASP.NET Core health checks, and watch one of the most expensive mistakes, putting a database check in liveness, cause a real cascade of restarts.