Secrets in GitOps: Sealed Secrets, SOPS and Keeping Secrets Out of Git Entirely

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, Argo CD 3.6.1, Sealed Secrets 0.41.2 (kubeseal 0.41.2), sops 3.13.3, age 1.3.2, ksops 4.5.1 and Kustomize 5.8.

Guess First

Bookstore's database Secret now lives in Git as an encrypted SealedSecret. One day you lose the cluster and build a new one: Argo CD, the Sealed Secrets controller and the config repo are the same, the only difference is that the controller is new. When Argo CD syncs the applications, do the SealedSecrets turn into Secrets? If not, what do you see in Argo CD and how do you fix it? We created this situation on purpose in the lab; the answer with real output is in the middle of the post.

The Problem: "Everything in Git" and Secrets

The first OpenGitOps principle asks for the whole desired state to be kept declaratively. Until now the PostgreSQL password of Bookstore, the online bookstore used as the shared example across this site's series, was the exception: we created the api-db Secret by hand in each namespace with a lab script and kept it out of Git. A new environment meant "run this script first", and so did rebuilding the cluster. Putting the Secret into Git as it is solves nothing, because the value in a Kubernetes Secret is only base64-encoded:

kubectl create secret generic api-db --from-literal=password=example-password-change-me --dry-run=client -o yaml

apiVersion: v1
data:
  password: ZXhhbXBsZS1wYXNzd29yZC1jaGFuZ2UtbWU=
kind: Secret
metadata:
  name: api-db

echo ZXhhbXBsZS1wYXNzd29yZC1jaGFuZ2UtbWU= | base64 -d
example-password-change-me

As the ConfigMaps and Secrets post of the Kubernetes series said, base64 is not encryption, just an encoding. Anyone who can read the repo reads the password, and Git history keeps it forever. In this post we try three approaches on Bookstore's connection string: Sealed Secrets (dev and prod), SOPS with age (staging), and keeping the secret out of Git entirely. Choosing where a Secret comes from and rotating it is the subject of the Configuration and Secret Management series; the question here is only "the secret in Git".

Sealed Secrets: Encrypting with the Cluster's Public Key

Sealed Secrets has two parts: a controller in the cluster and the kubeseal CLI on the developer's machine. On startup the controller generates a key pair and stores the private key as a Secret in kube-system. kubeseal fetches only the public key (the certificate) from the controller and encrypts the Secret; only the controller in that cluster can open the resulting SealedSecret. Installation uses the official manifest:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.41.2/controller.yaml

level=INFO msg="Starting sealed-secrets controller" version=0.41.2
level=INFO msg="Searching for existing private keys"
level=INFO msg="New key written" namespace=kube-system name=sealed-secrets-keyw8mxl

We encrypted the Secret in dev as it was, without generating a new password; PostgreSQL was initialised with that password on first start, so changing the value would mean an API that cannot reach its database. The values go through a pipe without landing in any file:

kubectl -n bookstore-dev get secret api-db -o yaml | kubeseal --format yaml > apps/api/overlays/dev/sealed-api-db.yaml
cd apps/api/overlays/dev && kustomize edit add resource sealed-api-db.yaml

# Sealed with the cluster's public key: only the controller in this cluster can decrypt it,
# and only as the Secret api-db in bookstore-dev (strict scope).
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: api-db
  namespace: bookstore-dev
  annotations:
    # Before postgres (wave -1): the controller turns this into the api-db Secret.
    argocd.argoproj.io/sync-wave: "-2"
spec:
  encryptedData:
    connection-string: AgBjPoUe80QDT1+A4Gf8jh...
    password: AgA//J8pi6byvYTyeV5O94...
  template:
    metadata:
      name: api-db
      namespace: bookstore-dev
    type: Opaque

The values are shortened; in the real file each is a blob of several hundred characters. This file could even go into a public repo: it does not open without the private key. We set sync wave -2 because in the order from the fourth post PostgreSQL sits at -1 and needs the Secret.

Taking Over an Existing Secret

We committed, Argo CD applied the SealedSecret, and the application went Degraded:

bitnami.com  SealedSecret  bookstore-dev  api-db  Synced  Degraded  failed update: Resource "api-db" already exists and is not managed by SealedSecret

level=ERROR msg="Error updating, giving up" key=bookstore-dev/api-db error="failed update: Resource \"api-db\" already exists and is not managed by SealedSecret"

The controller does not overwrite a Secret it did not create; that is a good default. Argo CD has a built-in health check for SealedSecret, so the problem shows up directly in the application's health. To take over, the existing Secret gets the sealedsecrets.bitnami.com/managed: "true" annotation. One more surprise: the "giving up" in the log is meant seriously. After we added the annotation, the controller did not retry on its own; the state stayed the same until the SealedSecret changed or the controller restarted:

kubectl -n bookstore-dev annotate secret api-db sealedsecrets.bitnami.com/managed=true
kubectl -n kube-system rollout restart deploy/sealed-secrets-controller

Synced=True
SealedSecret/api-db
Health Status:      Healthy

The second line is the Secret's ownerReferences: the Secret now belongs to the SealedSecret, and deleting the SealedSecret deletes the Secret. In prod we reversed the order: annotation first, then encryption and commit. This time it was Healthy on the first sync. Because kubeseal copies the annotations of the existing Secret, managed: "true" appears in the template of the prod file; that is harmless.

Scopes: Encrypted Data Is Bound to Name and Namespace

The default scope is strict: the Secret's name and namespace are part of the encrypted data. We took the dev file, changed only its namespace and applied it to another namespace:

Synced=False no key could decrypt secret (connection-string, password)
REASON            MESSAGE
ErrUnsealFailed   Failed to unseal: no key could decrypt secret (connection-string, password)

This stops dev's password from opening in prod just because someone copied the file. namespace-wide allows renaming within the same namespace and cluster-wide allows opening anywhere; both are chosen with kubeseal --scope and recorded as an annotation on the SealedSecret. As long as you use separate values per environment, stay with strict.

The Key Itself: Renewal, Backup and a Disaster Drill

Every 30 days the controller generates a new sealing key and adds it to the list of active keys (change it with --key-renew-period). Old keys are not deleted, so old SealedSecrets keep opening; new encryptions use the newest key. kubeseal --re-encrypt re-encrypts existing files with the newest key. The project's documentation stresses an important point: renewing the key is not rotating the secret. If a password leaked you have to change it; an old commit encrypted with an old key stays in Git history and opens if that key leaks.

The answer to Guess First. We backed up the key, then deleted it and restarted the controller; that is what a new cluster amounts to: a new controller with a new key.

kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key-backup.yaml
kubectl -n kube-system delete secret -l sealedsecrets.bitnami.com/sealed-secrets-key
kubectl -n kube-system rollout restart deploy/sealed-secrets-controller

dev: Synced=False no key could decrypt secret (connection-string, password)
prod: Synced=False no key could decrypt secret (connection-string, password)
secret/api-db
Health Status:      Degraded

The files in Git now mean nothing. Existing Secrets were not deleted (secret/api-db is still there), so running pods stay up; but in an empty cluster no Secret would be created and PostgreSQL would not start. After restoring the backup:

kubectl apply -f sealed-secrets-key-backup.yaml
kubectl -n kube-system rollout restart deploy/sealed-secrets-controller

dev: Synced=True
prod: Synced=True
Health Status:      Healthy
NAME                      TYPE                DATA   AGE
sealed-secrets-keyw8mxl   kubernetes.io/tls   2      31s
sealed-secrets-keywmqw5   kubernetes.io/tls   2      66s

The controller uses both keys: it opens the old files with the key from the backup and encrypts new ones with the newest key. The backup file holds private keys; it goes to a secret manager or an encrypted vault, not to Git, and it has to be refreshed after every key renewal. A Sealed Secrets installation without a backup is ready to lose every secret together with the cluster.

SOPS and age: Encrypting the File, Decrypting with KSOPS in Argo CD

SOPS is a different model: it encrypts the values inside a YAML or JSON file, and the encryption key belongs to the team, not to the cluster. It works with age (a modern, simple file encryption tool), Amazon's KMS, Azure Key Vault or PGP. We tried staging with age. The .sops.yaml at the repo root says which file is encrypted for which key:

# SOPS rules for this repo: staging Secrets are encrypted for the staging age key.
# Only values under data/stringData are encrypted; kind and metadata stay readable.
creation_rules:
  - path_regex: apps/.*/overlays/staging/.*\.enc\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1ukq6ax5v...

age-keygen -o age.agekey
sops encrypt --filename-override apps/api/overlays/staging/api-db.enc.yaml api-db.plain.yaml > apps/api/overlays/staging/api-db.enc.yaml

apiVersion: v1
data:
    connection-string: ENC[AES256_GCM,data:3lvZS8...]
    password: ENC[AES256_GCM,data:Z8aF3C...]
kind: Secret
metadata:
    name: api-db
    namespace: bookstore-staging
type: Opaque
sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            ...
            -----END AGE ENCRYPTED FILE-----
          recipient: age1ukq6ax5v...
    encrypted_regex: ^(data|stringData)$
    mac: ENC[AES256_GCM,data:ZgbVk/...]
    version: 3.13.3

The difference from Sealed Secrets is that the file is still a Kubernetes Secret: only the values are encrypted, the structure stays readable and a pull request shows which key changed. The mac field protects the integrity of the file. Argo CD cannot open this file; the repo-server that produces the manifests has to run SOPS. KSOPS does that by plugging into Kustomize as a generator:

# KSOPS: kustomize runs the ksops binary, which decrypts api-db.enc.yaml with SOPS.
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: api-db
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - ./api-db.enc.yaml

We added generators: [secret-generator.yaml] to the staging overlay and committed. Argo CD's first reaction:

ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = `kustomize build <path to cached source>/apps/api/overlays/staging` failed exit status 1: Error: loading generator plugins: failed to load generator: external plugins disabled; unable to load external plugin 'ksops'

Three things are needed: plugin and exec permission for Kustomize, the ksops binary in the repo-server, and the age private key. The permission goes into argocd-cm, the other two into a patch on the repo-server Deployment:

kubectl -n argocd create secret generic sops-age --from-file=keys.txt=age.agekey
kubectl -n argocd patch configmap argocd-cm --type merge -p '{"data":{"kustomize.buildOptions":"--enable-alpha-plugins --enable-exec"}}'
kubectl -n argocd patch deployment argocd-repo-server --patch-file repo-server-ksops.yaml

      initContainers:
        - name: install-ksops
          image: viaductoss/ksops:v4.5.1
          # The image has no shell: ksops copies itself (without --with-kustomize, Argo CD keeps its own kustomize).
          command: ["/usr/local/bin/ksops", "install", "/custom-tools"]
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
      containers:
        - name: argocd-repo-server
          env:
            - name: SOPS_AGE_KEY_FILE
              value: /sops-age/keys.txt
          volumeMounts:
            - mountPath: /usr/local/bin/ksops
              name: custom-tools
              subPath: ksops
            - mountPath: /sops-age
              name: sops-age
              readOnly: true

In our first attempt the init container copied the binary with cp; the ksops image has no shell, so the pod did not start: exec: "/bin/sh": stat /bin/sh: no such file or directory. KSOPS's own install command solved it. After the repo-server restarted, staging became Synced and Healthy; Argo CD applied the Secret over the existing one and, since the values were the same, no pod restarted.

Does the decrypted Secret show up in Argo CD? Not through the API: argocd app manifests masks the values of both the Secret generated from Git and the live one:

argocd app manifests bookstore-api-staging --source git

connection-string ++++++++
password ++++++++

But masking only affects display. Argo CD's documentation spells out two problems of decrypting secrets during manifest generation: generated manifests sit in plain text in the Redis cache and can be fetched from the repo-server's API, and a secret change is tied to an unrelated application sync. On top of that, --enable-exec is a global setting: a kustomization in any repo Argo CD reads can run a script from that repo in the repo-server as an exec function. We confirmed locally that Kustomize does this. That script can also read the file where the age key is mounted. That is why Argo CD's documentation recommends approaches that decrypt secrets in the destination cluster; Sealed Secrets belongs to that group, KSOPS does not.

Keeping Secrets Out of Git Entirely: External Secrets Operator

The third approach puts no encrypted value in Git either. With the External Secrets Operator (ESO), Git holds only an ExternalSecret: "take the value at this path in Vault and make a Secret named api-db". The value is fetched in the cluster by the operator from the secret manager; Argo CD never sees it. We covered the mechanics, together with the Secrets Store CSI Driver and Workload Identity, in the Secrets in Containers and Kubernetes post of the Configuration and Secret Management series; we did not install it here. The three approaches compared for Bookstore's connection string:

None of them is "the right one". If the company already has a secret manager, ESO is usually the path with the fewest moving parts; if not, Sealed Secrets needs only a controller and a key backup.

Production Notes

Automate the Sealed Secrets key backup: the key is renewed every 30 days, so a one-off backup is incomplete a month later. Actually run a disaster drill as we did; "there is a backup" and "the backup opens the secrets" are different things. Putting the certificate from kubeseal --fetch-cert into the repo lets developers without cluster access encrypt too (the certificate is public). Any commit that changes a Secret may need a pod restart: Kubernetes does not restart a pod when a Secret read as an environment variable changes (the ConfigMaps and Secrets post of the Kubernetes series); you need a checksum annotation or a tool such as Reloader. With SOPS, keep the age private key in at least two places (the repo-server and a vault) and add more than one recipient to .sops.yaml; losing the only key means losing the files.

Common Pitfalls

Committing the plain Secret once before encrypting it. Deleting it in the next commit is not enough, it stays in Git history; the password has to change.

Converting an existing Secret to a SealedSecret without the annotation and waiting for the controller to recover by itself after "giving up".

Not backing up the Sealed Secrets private key. In a new cluster every SealedSecret in Git becomes meaningless.

Leaving the plain file (api-db.plain.yaml) in the repo folder. Produce the unencrypted file outside the repo and delete it when you are done; adding *.plain.yaml to .gitignore is a second safeguard.

Enabling --enable-exec for KSOPS without restricting who can commit to the config repo.

Forgetting the tools that render the overlay outside Argo CD. The ci/promote.sh script from the sixth post rendered the staging overlay with kustomize build to promote from staging to prod; once the KSOPS generator was added, the script stopped with this error: Error: loading generator plugins: failed to load generator: external plugins disabled; unable to load external plugin 'ksops'. Instead of giving CI the age key, we changed the script to read the images list of the overlay directly; promotion does not need to decrypt the secret.

A Small Experiment

★☆☆ Produce the SealedSecret of a new environment with lab/scripts/seal-db-secret.sh bookstore-qa and encrypt it once more with --scope namespace-wide. Compare the annotations of the two files and apply the second one to the same namespace under the name api-db-copy.

★★☆ Add a second age recipient to .sops.yaml (a new key from age-keygen), update the staging file with sops updatekeys and confirm that sops decrypt works with the second key. Look at what changed in the diff.

★★★ Use Sealed Secrets instead of KSOPS for staging, then remove kustomize.buildOptions from argocd-cm and the repo-server patch. Check in Argo CD which applications are affected and write down what moving secret decryption out of the repo-server gains in terms of attack surface.

The secrets are now in Git, and Argo CD knows what happens after a commit. But shipping a new version to every user at once is still a big step. The next post moves to progressive delivery with Argo Rollouts: a canary that advances or rolls back by itself based on Prometheus metrics.