Kubernetes Security: ServiceAccounts, RBAC, Pod Security Standards, securityContext and NetworkPolicy

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37 (with the kindnet network plugin), kubectl 1.37, PostgreSQL 18 and .NET 10.

Guess First

The Bookstore API (Application Programming Interface) runs in one namespace, PostgreSQL in another. You haven't configured any security. Can any pod in the same namespace, say a busybox someone left behind for debugging, connect directly to PostgreSQL's port 5432? Is there a token inside the API pod that grants access to the Kubernetes API? And which user does an image based on mcr.microsoft.com/dotnet/aspnet:10.0 run as: the app user that has shipped since .NET 8, or root? We tried all three on a real cluster; the answer to the first two is "yes", and the third isn't what most people guess.

ServiceAccounts and Tokens: A Pod's Identity in Kubernetes

Every pod runs with a ServiceAccount; if you don't specify one, with the namespace's default ServiceAccount. By default Kubernetes mounts that identity's token into every pod. We looked with an image that has a shell:

$ kubectl -n bookstore-sec exec tokencheck -- sh -c 'id; ls /var/run/secrets/kubernetes.io/serviceaccount/'
uid=0(root) gid=0(root) groups=0(root)
ca.crt
namespace
token

The token is a JWT (JSON Web Token); we decoded its payload:

{"aud": ["https://kubernetes.default.svc.cluster.local"], "exp": 1825648071, "iat": 1794112071, "sub": "system:serviceaccount:bookstore-sec:default"}
lifetime_s= 31536000

This is a projected token: bound to the pod, refreshed regularly by the kubelet, invalidated when the pod is deleted. With kind's default API server settings its validity came out as one year, an extended lifetime kept for compatibility with older clients. The default ServiceAccount has almost no permissions today (kubectl auth can-i --list showed only discovery and self-review permissions). The danger is tomorrow: if someone binds a broad role to default to try out a controller, every pod in the namespace inherits it, and an SSRF (Server-Side Request Forgery) or RCE (Remote Code Execution) flaw in the app finds an identity that reaches straight into the cluster.

The rule is simple: give every app its own ServiceAccount, and don't mount the token at all for apps that don't talk to the Kubernetes API. The Bookstore API never calls the Kubernetes API. The worker, however, uses a Lease object for the leader election we covered in the Background Jobs series:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api
  namespace: bookstore-sec
automountServiceAccountToken: false
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: worker
  namespace: bookstore-sec
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: worker-leader-election
  namespace: bookstore-sec
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    resourceNames: ["order-outbox-leader"]
    verbs: ["delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: worker-leader-election
  namespace: bookstore-sec
subjects:
  - kind: ServiceAccount
    name: worker
    namespace: bookstore-sec
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: worker-leader-election

RBAC: Granting Least Privilege and Testing It

RBAC (Role-Based Access Control) consists of four objects: Role (permissions within a namespace), ClusterRole (permissions cluster-wide or reusable in every namespace), RoleBinding and ClusterRoleBinding (which bind permissions to an identity). The Role above can only read and update Leases, and allows delete only for a single named Lease. Don't guess what you wrote, test it: kubectl auth can-i impersonates an identity and asks the API server.

can-i update leases --as=system:serviceaccount:bookstore-sec:worker -> yes
can-i delete leases/order-outbox-leader --as=system:serviceaccount:bookstore-sec:worker -> yes
can-i delete leases/other --as=system:serviceaccount:bookstore-sec:worker -> no
can-i list secrets --as=system:serviceaccount:bookstore-sec:worker -> no
$ kubectl auth can-i get leases -n bookstore-sec --as=system:serviceaccount:bookstore-sec:api
no
$ kubectl auth can-i get leases -n bookstore --as=system:serviceaccount:bookstore-sec:worker
no

The last line shows the Role's namespace boundary: the same worker can't reach a Lease in another namespace. These checks can run in the pipeline too; an RBAC change flipping the answer to "can it read secrets?" is something to catch in review. Three things to avoid: verbs: ["*"] or resources: ["*"], giving an app's ServiceAccount a ClusterRoleBinding, and list/watch on secrets (list returns the content of every secret in the namespace).

Pod Security Standards: Mandatory Rules via a Namespace Label

PSS (Pod Security Standards) defines three levels: privileged (unrestricted), baseline (closes known privilege escalation paths) and restricted (current hardening best practices). They are enforced by the built-in PSA (Pod Security Admission) controller; a namespace label is enough. This replaced PodSecurityPolicy, which was removed in Kubernetes 1.25 (reading old manifests is the eighteenth post of the series):

kubectl label ns bookstore-sec \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=v1.37 \
  pod-security.kubernetes.io/warn=restricted

Warning: existing pods in namespace "bookstore-sec" violate the new PodSecurity enforce level "restricted:v1.37"
Warning: tokencheck: allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
namespace/bookstore-sec labeled

The label doesn't delete existing pods, it only warns. When we applied the plain API Deployment from earlier posts to this namespace, the Deployment was created, but its pods couldn't be:

Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "api" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "api" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "api" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "api" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/api created
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
api    0/2     0            0           5s
Warning   FailedCreate   replicaset/api-5c4bc79dd4   ... Error creating: pods "api-5c4bc79dd4-69vnx" is forbidden: violates PodSecurity "restricted:v1.37": ...

Know this behavior: kubectl apply looks successful, the error is only visible in the ReplicaSet's events, and the Deployment sits at 0/2. Pinning enforce-version prevents existing workloads from being rejected overnight when the definition of restricted tightens in a cluster upgrade. Observing for weeks with warn and audit modes before moving to a new level is the safest path.

securityContext: Hardening the .NET App

A Deployment that meets the restricted level and also makes the root file system read-only:

    spec:
      serviceAccountName: api
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: bookstore-api:1.2.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          env:
            - name: ConnectionStrings__Default
              valueFrom:
                secretKeyRef:
                  name: api-db
                  key: connection-string
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 64Mi

Every line has a reason. runAsNonRoot refuses to start the container if the image wants to run as root. allowPrivilegeEscalation: false blocks gaining privileges through setuid binaries. capabilities.drop: ["ALL"] zeroes the Linux capabilities; Kestrel listening on a port above 1024 like 8080 needs none of them. seccompProfile: RuntimeDefault turns on the container runtime's default syscall filter. readOnlyRootFilesystem makes it harder for an attacker to drop files inside the container.

The third question from Guess First matters here. The chiseled image's user is 1654 (app). In the default Ubuntu 24.04-based aspnet:10.0 image, however, the APP_UID=1654 environment variable is defined but there's no USER line; if you don't write USER $APP_UID in your Dockerfile, the container runs as root (the uid=0(root) output above comes from exactly this image). When we tried the same Deployment with that image:

api-full-7fc798f798-z6srz   0/1     CreateContainerConfigError   0          12s
Warning   Failed   pod/api-full-7fc798f798-z6srz   Error: container has runAsNonRoot and image will run as root (pod: "api-full-7fc798f798-z6srz_bookstore-sec(b90f6f93-ac19-4d09-8a68-dba149846624)", container: api-full)

The fix is to add USER $APP_UID to the Dockerfile or write runAsUser: 1654 in the securityContext; the former makes the image safe everywhere.

The .NET app started with a read-only file system, but there's a change in the log to watch:

04:28:44.078 warn: Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository[50] Using an in-memory repository. Keys will not be persisted to storage.
04:28:44.078 warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[59] Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.

Since Data Protection can't write to /home/app/.aspnet, it keeps keys in memory; the key ring problem we saw in the twelfth post now happens even on a container restart. A read-only file system makes a shared key store mandatory. The emptyDir for /tmp protects something else: the .NET runtime opens its diagnostics IPC (Inter-Process Communication) socket in /tmp; if that's not writable, dotnet-counters and dotnet-dump can't find the process. We'll use this with a debug container in the next post.

One more step: user namespaces. hostUsers: false, GA (General Availability) in Kubernetes 1.36, maps the container's UIDs (User IDs) to a separate range on the node; a process that is root inside the container is an unprivileged user on the node. We tried it on kind:

$ kubectl -n bookstore-sec exec userns -- sh -c 'id; cat /proc/self/uid_map'
uid=1000 gid=0(root) groups=0(root)
         0 1017053184      65536
$ kubectl -n bookstore-sec exec client -- sh -c 'id; cat /proc/self/uid_map'
uid=1000 gid=0(root) groups=0(root)
         0          0 4294967295

In the pod with hostUsers: false, the container's 0–65535 range starts at 1017053184 on the node; in the normal pod the mapping is one to one. The node's kernel and container runtime must support it; check your provider's documentation before turning it on in a managed cluster.

NetworkPolicy: By Default Everyone Reaches Everyone

We tried the first question from Guess First without any policy. An ordinary client pod in the same namespace could connect directly to PostgreSQL:

$ kubectl -n bookstore-sec exec client -- nc -z -w 3 postgres.bookstore-data 5432; echo "nc exit=$?"
nc exit=0

A NetworkPolicy describes traffic between pods by labels. The first policy that selects a pod switches that pod into "deny by default" mode, and policies only add permissions. We put a default deny in both namespaces:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: bookstore-sec
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]

$ kubectl -n bookstore-sec exec client -- wget -q -T 5 -O- http://api/api/books
wget: bad address 'api'

The error isn't "connection refused" but "bad address": since we closed egress, not even the DNS (Domain Name System) query could go out. The first rule to add after a default deny is always DNS. Then only the paths that are really needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: bookstore-sec
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api
  namespace: bookstore-sec
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: client
      ports:
        - port: 8080
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: bookstore-data
          podSelector:
            matchLabels:
              app.kubernetes.io/name: postgres
      ports:
        - port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-from-api
  namespace: bookstore-data
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: postgres
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: bookstore-sec
          podSelector:
            matchLabels:
              app.kubernetes.io/name: api
      ports:
        - port: 5432

We also added a small egress policy (client-to-api) for the client's traffic to the API. The result:

$ kubectl -n bookstore-sec exec client -- wget -q -T 5 -O- http://api/api/books
[{"id":1,"title":"Clean Architecture","author":"Robert C. Martin","price":42.50}, ...]
$ kubectl -n bookstore-sec exec client -- nc -z -w 3 postgres.bookstore-data 5432; echo "nc exit=$?"
nc exit=1
$ kubectl -n bookstore-sec exec client -- wget -q -T 5 -O- http://api.bookstore.svc.cluster.local/
wget: download timed out

The API reaches the database (the books came from PostgreSQL), the client reaches the API, but the client can no longer reach the database or a service in another namespace. Having namespaceSelector and podSelector in the same to item means "AND"; as separate list items it would mean "OR" and allow every pod in bookstore-data. That one-indent difference is the most common NetworkPolicy mistake. One more warning: NetworkPolicy is enforced by the CNI (Container Network Interface) plugin. kind's kindnet enforces it; on a CNI that doesn't, these objects are created without error and nothing is blocked. On a managed cluster, check that policy support is on (the network policy option on AKS, that is Azure Kubernetes Service; the VPC (Virtual Private Cloud) CNI's policy mode or Calico/Cilium on EKS, that is Amazon Elastic Kubernetes Service), and always verify with a negative test like the one above.

Image Pull Secrets and Where Images Come From

To pull images from a private registry, imagePullSecrets is added to the pod or to the ServiceAccount; adding it to the ServiceAccount leaves less room for error than writing it into every Deployment. On managed clusters, secretless access is better: on AKS, granting the kubelet identity pull rights on the registry; on EKS, giving the node role read access to ECR (Elastic Container Registry). Being sure of what actually runs is a separate topic: deploying by digest instead of tag, signing the image and verifying the signature at admission, producing an SBOM (Software Bill of Materials). Those are topics of the upcoming DevSecOps series; what matters here is knowing that bookstore-api:1.2.0 in a manifest isn't always the same bytes.

Production Notes

Security settings are turned on gradually: first warn/audit, then enforce; first one namespace, then all. When applying a default deny to production for the first time, forgotten dependencies show up: an external payment service, an SMTP (Simple Mail Transfer Protocol) server, a metrics collector, traffic coming from the Gateway. Build policies in staging first, looking at traffic logs. After turning on readOnlyRootFilesystem, find from the logs where the app tries to write; typical candidates in .NET are /tmp, Data Protection keys and the cache folders of some libraries. Audit RBAC regularly: saving the output of kubectl auth can-i --list --as=... for each ServiceAccount and watching for changes is a cheap start. And remember: every setting here is a defense layer and doesn't replace the app's own security; for the flaws on the OWASP (Open Worldwide Application Security Project) list, the Web Security (OWASP Top 10) series still applies.

Common Pitfalls

Running every app with the default ServiceAccount and a mounted token. Assuming the aspnet:10.0 image runs as non-root on its own. Turning on the restricted level and trusting that kubectl apply looks successful; the fact that pods can't be created only shows up in ReplicaSet events. Turning on readOnlyRootFilesystem and forgetting /tmp and Data Protection keys. Not allowing DNS after a default deny. Writing namespaceSelector and podSelector as separate list items in a NetworkPolicy and unintentionally opening the door to the whole namespace. Writing policies on a CNI that doesn't support them and believing you're protected. And seeing list permission on secrets as harmless because "it only lists".

A Small Experiment

★☆☆ In the bookstore-sec namespace, keep the warn=restricted label, remove enforce and apply the RabbitMQ Deployment from the eleventh post. Which warnings do you get, and which can you fix without changing the RabbitMQ image?

★★☆ Rewrite the egress rule in the API's NetworkPolicy so that namespaceSelector and podSelector are separate list items. Put a second pod in bookstore-data and show that the API (or a test pod with the same label) can reach it.

★★★ Actually run leader election for the worker: write a small BackgroundService in .NET that acquires and renews a Lease with the KubernetesClient package, run two replicas and watch holderIdentity change hands when a pod is deleted with kubectl get lease order-outbox-leader -o yaml. What do you see in the log when you remove the update verb from the Role?

As you tighten security settings, things will break: pods won't start, connections will time out, processes will be OOMKilled. In the next post we look at diagnosing those breakages systematically: CrashLoopBackOff, ImagePullBackOff, Pending and kubectl debug on shell-less chiseled images.