Storage and Stateful Apps in Kubernetes: Volumes, PersistentVolumeClaims, StatefulSets and the Database Decision

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, kubectl 1.37, PostgreSQL 18, Redis 7 and .NET 10.

Guess First

The Bookstore API (Application Programming Interface) writes cover images uploaded by users into /app/uploads. So the folder isn't lost, you attach it to a PVC (PersistentVolumeClaim) with access mode ReadWriteOnce, and the Deployment has 3 replicas. How many pods run: one or three? What changes if you switch the access mode to ReadWriteOncePod? And one more: in a two-replica API, can one pod decrypt a value that the other pod encrypted? The answers to the first two are not what the access mode names suggest, and the third is the .NET-specific face of the volume topic.

A Container's File System Is Ephemeral: emptyDir

Every file a container writes disappears with the container. Even when a container restarts, its file system is reset from the image. For temporary data that two containers must share or that must survive a restart, the simplest volume is emptyDir: it is created as an empty folder when the pod lands on a node, it survives container restarts, and it is deleted when the pod is deleted. A pod that imitates an export job writing a report to disk before uploading it:

apiVersion: v1
kind: Pod
metadata:
  name: report-export
  namespace: bookstore
spec:
  restartPolicy: Never
  containers:
    - name: export
      image: busybox:1.37
      command: ["sh", "-c", "dd if=/dev/zero of=/scratch/report.bin bs=1M count=80 && sleep 3600"]
      volumeMounts:
        - name: scratch
          mountPath: /scratch
  volumes:
    - name: scratch
      emptyDir:
        sizeLimit: 50Mi

The pod writing 80 MB stayed Running for a while, then got caught in the kubelet's periodic check:

NAME            READY   STATUS   RESTARTS   AGE
report-export   0/1     Error    0          111s
Status:           Failed
Reason:           Evicted
Message:          Usage of EmptyDir volume "scratch" exceeds the limit "50Mi".

An emptyDir without sizeLimit can fill the node's disk without bound and get every pod on the node evicted under disk pressure. By default emptyDir lives on the node's disk; with medium: Memory it moves to RAM (tmpfs), and then every byte written counts against the container's memory limit. In the fifteenth post of the series we'll use emptyDir to cover .NET's /tmp needs together with readOnlyRootFilesystem.

PV, PVC and StorageClass: Who Asks for the Disk, Who Provides It

For persistent data Kubernetes separates two objects. A PV (PersistentVolume) is an actual disk in the cluster: a cloud disk, an NFS (Network File System) share, a folder on a node in kind. A PVC is the app's request: "I need a 1Gi disk with this access mode". The bridge between them is the StorageClass; the provisioner that sees the PVC creates a new PV and binds the two (dynamic provisioning). As an application developer you usually write only the PVC; the StorageClasses are defined by the team that runs the cluster. The default that ships with kind:

NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  19m

Three columns carry production decisions. RECLAIMPOLICY: Delete means that when the PVC is deleted, the PV underneath and its data are deleted too; we'll actually see that below. WaitForFirstConsumer means the PV is created not when the PVC is created but when the pod using it is scheduled; that way the disk is opened on the node the pod goes to (in the cloud, in the pod's zone). In Immediate mode the disk is opened in an arbitrary zone first, and the pod is then stuck with the nodes in that zone. ALLOWVOLUMEEXPANSION tells you whether the PVC can be grown later; it's off for local-path and usually on for cloud providers' CSI (Container Storage Interface) drivers.

One property of kind's local-path provisioner matters: the PV it creates is tied to a node.

{"key":"kubernetes.io/hostname","operator":"In","values":["k8swork2-1-worker"]}
/var/local-path-provisioner/pvc-929be5b9-77e2-4aaf-a3c9-acf9dafe5746_bookstore_data-postgres-0

A pod using this PV can now only run on k8swork2-1-worker. Cloud disks are similarly tied to a zone: an Azure Disk or an EBS (Amazon Elastic Block Store) disk cannot be attached to a node outside the zone it was created in.

Access Modes: ReadWriteOnce Doesn't Mean a Single Pod

There are four access modes: ReadWriteOnce (RWO), ReadOnlyMany (ROX), ReadWriteMany (RWX) and ReadWriteOncePod (RWOP). "Once" and "Many" in the names describe the number of nodes, not pods; except for RWOP. We started the Deployment from Guess First with an RWO PVC and 3 replicas:

NAME READY STATUS NODE
api-uploads-79f7c58494-8bmsc 1/1 Running k8swork2-1-worker
api-uploads-79f7c58494-tnkcg 1/1 Running k8swork2-1-worker
api-uploads-79f7c58494-ztmwn 1/1 Running k8swork2-1-worker

All three ran, all three on the same node. RWO means "can be mounted read-write by a single node"; multiple pods on the same node can share the same volume. The scheduler had to gather the pods on the node the PV is tied to. This "it works" picture is dangerous: as replicas grow the pods can't spread, if the node goes down they all go down together, and three processes write to the same files without locking. On a cloud disk it's even worse: a pod that lands on another node gets stuck in ContainerCreating with a Multi-Attach error.

When we started the same Deployment with ReadWriteOncePod, Kubernetes enforced the single-pod rule itself; RWOP has been GA (General Availability) since 1.29:

NAME READY STATUS NODE
api-uploads-79f7c58494-kd499 0/1 Pending <none>
api-uploads-79f7c58494-klvts 0/1 Pending <none>
api-uploads-79f7c58494-lmjnv 1/1 Running k8swork2-1-worker
Warning  FailedScheduling  25s  default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) unavailable due to PersistentVolumeClaim with ReadWriteOncePod access mode already in-use by another pod. ...

RWOP is honest, but it has a price: since a Deployment's rolling update starts the new pod before stopping the old one, the new pod stays Pending too and the rollout hangs; such a Deployment needs strategy: Recreate. The real lesson: if multiple replicas need to write the same files, the answer is not a volume but object storage (below).

PostgreSQL with a StatefulSet

Pods in a Deployment are interchangeable: their names are random, they all share the same PVC or none at all. For workloads like databases, where each instance has its own disk and a stable identity, there is the StatefulSet: pod names are ordinal (postgres-0, postgres-1), each pod gets its own PVC from volumeClaimTemplates, and a recreated pod returns to the same PVC. PostgreSQL in the dev cluster:

apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: bookstore
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: postgres
  ports:
    - name: postgres
      port: 5432
      targetPort: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: bookstore
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  template:
    metadata:
      labels:
        app.kubernetes.io/name: postgres
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: postgres
          image: postgres:18-alpine
          ports:
            - name: postgres
              containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: bookstore
            - name: POSTGRES_USER
              value: bookstore
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres
                  key: password
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "bookstore", "-d", "bookstore"]
            periodSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 512Mi
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

The mount path is deliberately not /var/lib/postgresql/data. The PostgreSQL 18 images moved PGDATA into a version-specific subfolder; we checked inside the container: PGDATA=/var/lib/postgresql/18/docker. If you copy the path from old tutorials, the container never starts; we tried the same image in Docker with an empty volume mounted at /var/lib/postgresql/data and the entrypoint exited with this error:

Error: in 18+, these Docker images are configured to store database data in a
       format which is compatible with "pg_ctlcluster" (specifically, using
       major-version-specific directory names).
...
       The suggested container configuration for 18+ is to place a single mount
       at /var/lib/postgresql which will then place PostgreSQL data in a
       subdirectory, allowing usage of "pg_upgrade --link" without mount point
       boundary issues.

In Kubernetes this shows up as a CrashLoopBackOff; at least the data isn't silently lost. The PVC waited in Pending until the pod was scheduled (WaitForFirstConsumer), then got bound:

NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
data-postgres-0   Bound    pvc-929be5b9-77e2-4aaf-a3c9-acf9dafe5746   1Gi        RWO            standard

The PVC name follows the <template name>-<pod name> rule. We created a table, inserted two orders and deleted the pod:

NAME STATUS IP NODE
postgres-0 Running 10.244.2.17 k8swork2-1-worker
pod "postgres-0" deleted from bookstore namespace
NAME STATUS RESTARTS IP NODE
postgres-0 Running 0 10.244.2.18 k8swork2-1-worker
  id  | customer | total
------+----------+-------
 1001 | Ada      | 97.40
 1002 | Grace    | 55.00
(2 rows)

The IP (Internet Protocol) address changed, the name and data stayed. That's why apps connect through a DNS (Domain Name System) name, not the pod IP. A headless Service with clusterIP: None gives each pod a stable DNS record:

Name:	postgres-0.postgres.bookstore.svc.cluster.local
Address: 10.244.2.31

Writing Host=postgres in the connection string is enough with a single replica; in a replicated setup writes must go to the primary, and Kubernetes doesn't know which pod is the primary. This is the first place where a StatefulSet falls short for a database.

persistentVolumeClaimRetentionPolicy decides what happens to PVCs when the StatefulSet is deleted or scaled down. With Retain we deleted the StatefulSet, the PVC stayed, and when we applied it again the data came back. Then we deleted the PVC by hand:

persistentvolumeclaim "data-postgres-0" deleted from bookstore namespace
$ kubectl get pv
No resources found

Because the StorageClass's reclaim policy is Delete, the PV and the folder on the node went too. Deleting a PVC without knowing the reclaim policy of the production StorageClass is the same as deleting the database.

The Database in Production: Managed or Operator?

The StatefulSet above is fine for a dev environment; what's missing for production is not Kubernetes' job but database operations: backups, PITR (Point-In-Time Recovery), replication, failover, minor/major upgrades, connection pooling, monitoring. There are three paths:

An operator is a Kubernetes extension that encodes what a database expert does as a controller: in CloudNativePG you write a Cluster resource, and the operator manages the primary and replica pods, the PVCs, backups and failover, and gives the app an -rw Service that always points at the primary. Still, the question of who meets the RPO (Recovery Point Objective, acceptable data loss) and RTO (Recovery Time Objective, acceptable downtime) targets doesn't change; with managed services it's a responsibility included in the bill. My recommendation for Bookstore is clear: managed PostgreSQL in production, only the app in the cluster.

Uploads and Data Protection Keys: A Shared Store, Not a Volume

The right place for cover images is not a PVC but object storage (Azure Blob Storage, Amazon S3 (Simple Storage Service), Google Cloud Storage): every replica reads and writes at the same time, it isn't tied to a node or zone, and backups and CDN (Content Delivery Network) integration come ready. The API writes the image directly to the blob, or hands the client a short-lived upload URL (Uniform Resource Locator).

The sneakier one is ASP.NET Core Data Protection. Cookie authentication, antiforgery tokens and everything encrypted with IDataProtector rely on a key ring. If nothing is configured, keys are written to the container's file system and the runtime warns about it explicitly:

03:49:38.333 warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60] Storing keys in a directory '/home/app/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning

We tried this on the two-replica API. We encrypted order-1001 on one pod with /api/diag/protect, then had the same payload decrypted first by the same pod and then by the other one:

{"pod":"api-5c4bc79dd4-6x4cr","text":"order-1001"}
wget: server returned error: HTTP/1.1 500 Internal Server Error
System.Security.Cryptography.CryptographicException: The key {fc83aeff-ea88-4161-9361-3b8cc0e9ffca} was not found in the key ring.

For the user this means "my session dropped" or antiforgery errors on every request the load balancer sends to another pod; and since every deploy replaces all pods, everyone goes back to the login screen. Putting the keys on a PVC locks them to a single node again because of RWO. The fix is to move the key ring to a store that all replicas can reach:

// keys live in the container file system unless a shared store is configured
IDataProtectionBuilder dataProtection = builder.Services.AddDataProtection().SetApplicationName("bookstore");
string? redis = builder.Configuration["DataProtection:Redis"];
if (redis is not null)
{
    dataProtection.PersistKeysToStackExchangeRedis(ConnectionMultiplexer.Connect(redis), "bookstore-dp-keys");
}

We put a Redis in the cluster, gave the Deployment DataProtection__Redis=redis:6379, and ran the same test:

{"pod":"api-5fbb756d86-wxm75","text":"order-1001"}
$ kubectl -n bookstore exec deploy/redis -- redis-cli LLEN bookstore-dp-keys
1

The encrypting and decrypting pods differ, the result is correct, and there's a single key in Redis. Our Redis is not persistent, so a Redis restart loses the keys again; in production the usual choice is blob storage (PersistKeysToAzureBlobStorage from Azure.Extensions.AspNetCore.DataProtection.Blobs), with the keys themselves encrypted by Key Vault (ProtectKeysWithAzureKeyVault). If the app already uses EF (Entity Framework) Core, PersistKeysToDbContext is also a solid option. And SetApplicationName must be the same for apps that share a key ring so they can decrypt each other's payloads.

Production Notes

Disk performance comes with the StorageClass: at cloud providers the IOPS (Input/Output Operations Per Second) limits of standard and premium classes differ a lot, and a small disk means a small IOPS quota. When the database is slow, check whether the disk is hitting its limit before blaming the query. Watch PVC capacity; a full disk leaves PostgreSQL unable to write, and if allowVolumeExpansion is off, growing it means a new disk and a data move. In a zonal cluster, a pod tied to a disk in a lost zone cannot move to another zone; that's why a single-replica StatefulSet doesn't come back during a zone outage. Backups use VolumeSnapshots (if the CSI driver supports them) or the database's own backup tool, but a backup is only a backup once its restore has been tested. Finally, terminationGracePeriodSeconds: give PostgreSQL enough time to write its checkpoint on shutdown; remember that 30 seconds is the default and may not be enough for a busy database.

Common Pitfalls

Attaching an RWO PVC to a Deployment and scaling it up: either everything piles onto one node or, on a cloud disk, it gets stuck with Multi-Attach error. Using RWOP and being surprised when the rolling update hangs. Mounting the old /var/lib/postgresql/data path with the PostgreSQL 18 image. Deleting a PVC on a StorageClass with a Delete reclaim policy thinking "I'll recreate it later". Leaving sizeLimit off an emptyDir. Leaving Data Protection keys in the container while running multiple replicas, then getting random "my session dropped" complaints. Trying to connect to the database by IP while the app pod restarts. And the most expensive one: running a single-replica PostgreSQL in the cluster in production without backups.

A Small Experiment

★☆☆ Set up the PostgreSQL StatefulSet with mountPath: /var/lib/postgresql/data. What state does the pod fall into? Read the entrypoint's message with kubectl logs --previous, then go back to the right path and check where PGDATA actually points with kubectl exec.

★★☆ Set persistentVolumeClaimRetentionPolicy.whenScaled to Delete, scale the StatefulSet to 3 replicas, then back to 1. Which PVCs were deleted, and what happened to data-postgres-0?

★★★ Move the Data Protection key ring from Redis to PostgreSQL: write a DbContext with EF Core, implement IDataProtectionKeyContext and use PersistKeysToDbContext. Repeat the encrypt/decrypt test on two replicas, then delete all API pods and verify the old payload can still be decrypted.

In this post we created the table by hand with psql; in a real app the schema comes with EF Core migrations. In the next post we look at Jobs and CronJobs and run the EF Core migration bundle as a Job that runs once, at the right moment of a deploy.