Secrets in Containers and Kubernetes: External Secrets Operator, Secrets Store CSI Driver and Workload Identity

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, Helm 4.3, External Secrets Operator 2.15, Secrets Store CSI Driver 1.6.2, the Vault CSI provider (Vault Helm chart 0.34.1), HashiCorp Vault 2.2.2, Docker 29.2 (BuildKit and Compose) and .NET 10. The AKS Workload Identity steps were written from Microsoft Learn and the External Secrets documentation and were not run on an AKS cluster.

Guess First

The API of Bookstore, the online bookstore used as the shared example across this site's series, runs in Kubernetes in two ways: one pod gets the JWT key as an environment variable, the other gets the same Secret as a file in a volume. The Secret comes from Vault; the External Secrets Operator syncs it every 15 seconds. You change the key in Vault. Which pod sees the new key, and how long does it take? And another one: if a Dockerfile receives a NuGet token through ARG and deletes the file in the same RUN line after using it, does the token remain in the image?

The Limits of a Kubernetes Secret

We covered the basics of the Kubernetes Secret object in the ConfigMap and Secret post of the Kubernetes series: values are base64-encoded, not encrypted; encryption at rest has to be enabled separately to keep them encrypted in etcd, and anyone with permission to read Secrets in the namespace sees the value. This post adds three questions on top of that. When the real source of a secret is Vault or Key Vault, how does it get into Kubernetes? How does a change reach the pod? Does a pod need a secret to reach the cloud?

Putting Kubernetes Secrets into Git as manifests solves none of these. Ways to keep encrypted secrets in Git (Sealed Secrets, SOPS) are covered in the GitOps and Argo CD series. The approach in this post is different: Git holds not the value of the secret but only a description of where to get it.

External Secrets Operator: From Vault to a Kubernetes Secret

The External Secrets Operator (ESO) is a controller that reads values from an external secret store and produces Kubernetes Secrets. It has two core resources: a SecretStore (or a cluster-wide ClusterSecretStore) says how to connect to the store, and an ExternalSecret says which value goes into which Secret. For the experiment I ran the Vault from post five in dev mode inside the kind cluster and installed ESO with Helm. ESO signs in to Vault with the Kubernetes auth method, so ESO is not given a stored token either:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault
  namespace: bookstore
spec:
  provider:
    vault:
      server: http://vault.vault.svc:8200
      path: secret
      version: v2
      auth:
        # Kubernetes auth: with the service account token, no stored secret
        kubernetes:
          mountPath: kubernetes
          role: bookstore-eso
          serviceAccountRef:
            name: bookstore-eso

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: bookstore-api
  namespace: bookstore
spec:
  refreshInterval: 15s
  secretStoreRef:
    name: vault
    kind: SecretStore
  target:
    name: bookstore-api-secrets
    creationPolicy: Owner
  data:
    # Key names in the form .NET configuration expects: "__" = ":"
    - secretKey: Jwt__SigningKey
      remoteRef:
        key: bookstore/api
        property: jwt_signing_key
    - secretKey: Email__ApiKey
      remoteRef:
        key: bookstore/api
        property: email_api_key

$ helm install external-secrets external-secrets/external-secrets --version 2.15.0 -n external-secrets --create-namespace --wait
$ kubectl apply -f namespace.yaml -f secretstore.yaml -f externalsecret.yaml
$ kubectl -n bookstore get secretstore,externalsecret
NAME                                    AGE   STATUS   CAPABILITIES   READY
secretstore.external-secrets.io/vault   8s    Valid    ReadWrite      True

NAME                                               STORETYPE     STORE   REFRESH INTERVAL   STATUS         READY   LAST SYNC
externalsecret.external-secrets.io/bookstore-api   SecretStore   vault   15s                SecretSynced   True    7s

In ESO 2.x the API version is external-secrets.io/v1; the v1beta1 you see in older examples is no longer served, and only v1 is enabled in the CRD on the cluster. Having ESO write the Secret's keys directly with the names .NET expects (Jwt__SigningKey) means no mapping code at all in the app. creationPolicy: Owner makes the Secret go away when the ExternalSecret is deleted. One detail: permission to read Kubernetes Secrets is held not by the app's service account but by ESO's controller. When you ask who can access secrets, count that controller too:

$ kubectl auth can-i get secrets -n bookstore --as=system:serviceaccount:bookstore:bookstore-api
no
$ kubectl auth can-i get secrets -n bookstore --as=system:serviceaccount:external-secrets:external-secrets
yes

How Does Rotation Reach the Pod? Environment Variables and Volumes

I gave the Secret produced by ESO to two Deployments in different ways. api-env receives every key as an environment variable through envFrom. api-volume mounts the Secret at /app/secrets and the app reads the folder with the key-per-file provider: the file name is the key, the content is the value.

          # Every key of the Secret as an environment variable
          envFrom:
            - secretRef:
                name: bookstore-api-secrets

          env:
            - name: Secrets__Directory
              value: /app/secrets
            # To reliably see files that Kubernetes updates through a symlink
            - name: DOTNET_USE_POLLING_FILE_WATCHER
              value: "true"
          volumeMounts:
            - name: secrets
              mountPath: /app/secrets
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: bookstore-api-secrets

// Secrets mounted as files in Docker/Kubernetes: file name is the key, content is the value
if (builder.Configuration["Secrets:Directory"] is { Length: > 0 } secretsDirectory)
{
    builder.Configuration.AddKeyPerFile(secretsDirectory, optional: true, reloadOnChange: true);
}

I turned on the polling watcher because of behavior we measured in the Kubernetes series: the kubelet updates files in the volume by switching the ..data symlink, and .NET's default file watcher does not always see that change. Then I changed the JWT key in Vault and printed, every five seconds, the fingerprint of the value in the Secret and of the key each pod sees:

09:51:39 vault kv patch secret/bookstore/api jwt_signing_key=...
+0s secret=c425ba68 env-pod=c425ba68 volume-pod=c425ba68
+5s secret=ed300634 env-pod=c425ba68 volume-pod=c425ba68
+10s secret=ed300634 env-pod=c425ba68 volume-pod=c425ba68
...
+51s secret=ed300634 env-pod=c425ba68 volume-pod=c425ba68
+56s secret=ed300634 env-pod=c425ba68 volume-pod=ed300634

ESO updated the Secret within five seconds (the refresh interval is 15 seconds). The file in the volume changed after about 56 seconds; that delay comes from the kubelet's sync loop and Secret cache, not from ESO. The pod with environment variables never saw the new value: the environment is read once when the process starts. The only way to update it is a new pod:

$ kubectl -n bookstore rollout restart deploy/api-env
deployment.apps/api-env restarted
$ curl -s localhost:18087/api/info | grep -o '"jwtKeyFingerprint":"[a-f0-9]*"'
"jwtKeyFingerprint":"ed300634"

That is the answer to the opening question: the pod reading the volume in about a minute, the pod with environment variables only after a restart. When you plan a rotation you need to account for that one-minute delay and the need for a restart. If you must use environment variables, a controller that triggers a rollout automatically when a Secret changes (such as Stakater Reloader) or a deploy step that writes the Secret's hash as an annotation on the pod template can be used; I ran neither in this post.

Secrets Store CSI Driver: Secrets Directly as a Volume

The second approach removes the Kubernetes Secret from the picture. The Secrets Store CSI Driver reads values through a provider (Vault, Azure Key Vault, AWS, GCP) when the pod starts and writes them directly into the pod's volume. The pod signs in to Vault with its own service account; no copy is created in etcd:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: bookstore-api-vault
  namespace: bookstore
spec:
  provider: vault
  parameters:
    vaultAddress: http://vault.vault.svc:8200
    # Sign in to Vault with the pod's own service account
    roleName: bookstore-api
    objects: |
      - objectName: "Jwt__SigningKey"
        secretPath: "secret/data/bookstore/api"
        secretKey: "jwt_signing_key"
      - objectName: "Email__ApiKey"
        secretPath: "secret/data/bookstore/api"
        secretKey: "email_api_key"

      volumes:
        - name: secrets
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: bookstore-api-vault

$ helm install csi secrets-store-csi-driver/secrets-store-csi-driver --version 1.6.2 -n kube-system --set syncSecret.enabled=true
$ helm install vault-csi hashicorp/vault --version 0.34.1 -n vault \
    --set server.enabled=false --set injector.enabled=false --set csi.enabled=true

$ kubectl -n bookstore exec deploy/api-csi -- ls -l /app/secrets
lrwxrwxrwx 1 root root 20 Nov 14 09:54 Email__ApiKey -> ..data/Email__ApiKey
lrwxrwxrwx 1 root root 22 Nov 14 09:54 Jwt__SigningKey -> ..data/Jwt__SigningKey
$ curl -s localhost:18089/api/info
{"environment":"Staging","freeShippingThreshold":750,"jwtKeyFingerprint":"ed300634","secrets":{"ConnectionStrings:Bookstore":false,"Jwt:SigningKey":true,"Email:ApiKey":true,"Payments:Webhook:Secret":false}}
$ kubectl get secret -n bookstore
NAME                    TYPE     DATA   AGE
bookstore-api-secrets   Opaque   2      3m19s

Nothing changes on the app side; the same key-per-file provider reads the same folder. The only Secret in the namespace is the one ESO produced; values that arrive through CSI never appear in the Kubernetes API. There are two important limits. First, rotation: the chart's defaults have enableSecretRotation: false and the feature is marked alpha; unless you enable it, the file in the pod keeps the value read first for as long as the pod lives. Second, for an app that wants the value as an environment variable, the driver has to sync a separate Kubernetes Secret through secretObjects, which removes CSI's "no copy in etcd" advantage.

ESO or the CSI Driver?

Both solve the same problem from different places. Here is the decision table for Bookstore:

Bookstore's default is ESO: apps read the Secret as a volume, a short outage of the store does not keep pods from starting, and a single identity connects to Vault. We consider CSI for a few high-value secrets whose values must never sit in etcd.

AKS Workload Identity: A Cloud Identity for Pods Without a Secret

In post three we saw managed identity on App Service. The pod equivalent in Kubernetes is AKS Workload Identity: the cluster publishes an OIDC issuer, a federated credential is added to a user-assigned managed identity in Entra ID, and that credential is bound to a specific service account. The pod is given no secret; the short-lived service account token Kubernetes writes into the pod is exchanged for an Entra ID token. A SecretStore that connects ESO to Key Vault this way (not run on AKS; I validated the schema on the kind cluster with kubectl apply --dry-run=server):

# AKS: ESO connects to Key Vault with workload identity (not run in this post)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookstore-eso
  namespace: bookstore
  annotations:
    azure.workload.identity/client-id: "<user-assigned-identity-client-id>"
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: azure-kv
  namespace: bookstore
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: https://bookstore-kv-prod.vault.azure.net
      serviceAccountRef:
        name: bookstore-eso

If the app itself connects to Key Vault, the pod template gets the azure.workload.identity/use: "true" label; the webhook writes the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_FEDERATED_TOKEN_FILE variables into the pod. In that case you need WorkloadIdentityCredential instead of ManagedIdentityCredential; I added this branch to Bookstore's credential selection and pinned the selection down with unit tests:

// The AKS Workload Identity webhook adds this variable to the pod
if (!string.IsNullOrEmpty(configuration["AZURE_FEDERATED_TOKEN_FILE"]))
{
    return new WorkloadIdentityCredential();
}

$ dotnet test
Passed!  - Failed:     0, Passed:    14, Skipped:     0, Total:    14

Secrets in Containers: Build Time and Docker Compose

Some secrets are needed at build time rather than at runtime, such as the token for a private NuGet feed or npm registry. I tried the second opening question. A Dockerfile that receives the token through ARG and deletes the file in the same line:

FROM alpine:3
ARG NUGET_TOKEN
# The token is used only in this step and the file is deleted right away... but
RUN printf 'feed=https://pkgs.example.com/bookstore\ntoken=%s\n' "$NUGET_TOKEN" > /tmp/feed.conf \
 && echo "restoring with $(wc -c < /tmp/feed.conf) bytes of config" && rm /tmp/feed.conf

$ docker buildx build --load -t secretseries-build:arg --build-arg NUGET_TOKEN=$NUGET_TOKEN -f Dockerfile.arg .
$ docker history --no-trunc --format "{{.CreatedBy}}" secretseries-build:arg | head -1
RUN |1 NUGET_TOKEN=7df0abb391cb75d278e16e3801b3c3ec8c8dccba /bin/sh -c printf 'feed=https://pkgs.example.com/bookstore\ntoken=%s\n' "$NUGET_TOKEN" > /tmp/feed.conf  && echo "restoring with $(wc -c < /tmp/feed.conf) bytes of config" && rm /tmp/feed.conf # build

The file was deleted, but the token sits in plain text in the image's history; anyone who can pull the image from the registry sees it. In a multi-stage build, an ARG used only in the build stage does not reach the final image's history, but any value given through ENV or used in the final stage does. The right way is BuildKit's secret mount: the value is visible under /run/secrets only during that RUN step and is written to no layer:

# syntax=docker/dockerfile:1
FROM alpine:3
# The secret is visible under /run/secrets only during this RUN step and is never written to a layer
RUN --mount=type=secret,id=nuget_token \
    printf 'feed=https://pkgs.example.com/bookstore\ntoken=%s\n' "$(cat /run/secrets/nuget_token)" > /tmp/feed.conf \
 && echo "restoring with $(wc -c < /tmp/feed.conf) bytes of config" && rm /tmp/feed.conf

$ docker buildx build --load -t secretseries-build:secret --secret id=nuget_token,src=nuget_token.txt -f Dockerfile.secret .
#7 0.089 restoring with 87 bytes of config
$ docker history --no-trunc --format "{{.CreatedBy}}" secretseries-build:secret | head -1
RUN /bin/sh -c printf 'feed=https://pkgs.example.com/bookstore\ntoken=%s\n' "$(cat /run/secrets/nuget_token)" > /tmp/feed.conf  && echo "restoring with $(wc -c < /tmp/feed.conf) bytes of config" && rm /tmp/feed.conf # buildkit

87 bytes show that the token was read during the build; the history contains only the command itself. In CI the same token is written from a pipeline secret to a file and passed with --secret; the pipeline side of that is covered in the secrets post of the GitHub Actions series.

In environments without Kubernetes, the secrets section of Docker Compose gives the same key-per-file layout. The secret is attached to the container as a file under /run/secrets and does not appear in the environment:

name: secretseries

services:
  api:
    image: secretseries-api:06
    init: true
    ports:
      - "18086:8080"
    environment:
      ASPNETCORE_ENVIRONMENT: Staging
      # Not a secret, the folder that holds the secrets
      Secrets__Directory: /run/secrets
    secrets:
      - source: jwt_signing_key
        target: Jwt__SigningKey
      - source: email_api_key
        target: Email__ApiKey

secrets:
  jwt_signing_key:
    file: ./secrets/jwt_signing_key.txt
  email_api_key:
    file: ./secrets/email_api_key.txt

$ mkdir secrets
$ openssl rand -base64 32 | tr -d '\n' > secrets/jwt_signing_key.txt
$ openssl rand -hex 16 | tr -d '\n' > secrets/email_api_key.txt
$ docker compose up -d
$ curl -s localhost:18086/api/info
{"environment":"Staging","freeShippingThreshold":750,"jwtKeyFingerprint":"aa0f5a7a","secrets":{"ConnectionStrings:Bookstore":false,"Jwt:SigningKey":true,"Email:ApiKey":true,"Payments:Webhook:Secret":false}}
$ docker exec secretseries-api-1 ls -l /run/secrets
-rw-r--r-- 1 app app 32 Nov 14 09:47 Email__ApiKey
-rw-r--r-- 1 app app 44 Nov 14 09:47 Jwt__SigningKey
$ docker inspect secretseries-api-1 --format '{{range .Mounts}}{{.Type}} {{.Destination}}{{"\n"}}{{end}}'
bind /run/secrets/Email__ApiKey
bind /run/secrets/Jwt__SigningKey

In Compose (outside Swarm) a secret is really a bind mount: the file sits in plain text on the host and the secrets/ folder is in .gitignore. The gain is that the docker inspect and /proc/<pid>/environ channels we saw in post one are closed.

Production Notes

Remember that ESO sends one request to the store per ExternalSecret per refresh interval. 200 ExternalSecrets and a 15-second interval mean 800 reads per minute against Vault; an hour is enough for most secrets, so choose a shorter interval only for the few that must spread quickly. Because ESO's controller can write Secrets in every namespace, it is one of the most privileged components in the cluster; if you use a ClusterSecretStore, limit which namespaces may use it with conditions. Right after setup I saw a single could not get secret data from provider warning in the ExternalSecret's events; the first attempt, made before Vault's Kubernetes auth setup was complete, failed and ESO created the Secret on the next try. Transient errors like this are normal, but set an alert for ExternalSecrets that are not SecretSynced; otherwise rotation stops silently.

Don't skip enabling etcd encryption at rest and reviewing permissions to read Secrets (get, list, watch): someone with list sees the values of every Secret in the namespace. Permission to kubectl exec is also permission to read secrets; it can read both the file in the volume and the environment.

Common Pitfalls

Passing a secret as an environment variable and not restarting the pod on rotation. Mounting the volume with subPath; subPath mounts are never updated. Trusting reloadOnChange without knowing that .NET's default file watcher can miss symlink changes. Not knowing that the CSI driver's rotation is off by default. Passing a token through ARG or ENV in a Dockerfile and leaving it in docker history. Copying old examples that use ESO's v1beta1 API. Storing a service principal secret as a Kubernetes Secret instead of using workload identity; that just restarts the cycle.

A Small Experiment

★☆☆ Run docker history --no-trunc on your own images and look for a token left in an ARG, an ENV or a command line. If you find one, switch to a BuildKit secret mount.

★★☆ Install ESO on kind, sync a secret from Vault or a fake provider, and give it to one pod as an environment variable and to another as a volume. Change the value and measure when each pod sees it; repeat the measurement with DOTNET_USE_POLLING_FILE_WATCHER on and off.

★★★ Mount the same secret with the Secrets Store CSI Driver, enable enableSecretRotation, shorten rotationPollInterval and compare when the file changes with ESO. Then stop the store, start a new pod and note how each approach behaves in that situation.

Secrets now reach pods from the store, and we know when changes show up. The last post of the series puts that knowledge to use: seamless rotation with two active secrets, what to do in the first 60 minutes after a secret leaks, and a secretless architecture that brings Bookstore's secret count close to zero.