Running Argo CD in Production and What Changed from 2.x to 3.6: SSO, RBAC, Notifications, Metrics and Upgrades

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37 and Argo CD 3.6.1 (CLI 3.6.1). The SSO configuration, the HA installation and the upgrade command are based on the official documentation; the post says so where they appear.

Guess First

You give a role p, role:app-editor, applications, update, bookstore/*, allow. That role can change the bookstore-api-prod Application itself. Can the same role edit the api Rollout or a Pod inside that application from the Argo CD UI? In Argo CD 2.x the answer was yes. The answer in 3.6, with the output of argocd admin settings rbac can, is in the RBAC section.

Argo CD's Settings Belong in Git Too

Since the start of the series the whole state of Bookstore, the online bookstore used as the shared example across this site's series, has been in Git, but Argo CD's own settings were changes made with kubectl patch: argocd-cm for the secret decryption plugin from the seventh post, argocd-cmd-params-cm for progressive sync. In this post we add users, RBAC and notifications; doing those by hand too would leave Argo CD's most critical settings outside Git. We put one more Application under the App of Apps root:

# Argo CD's own settings (accounts, RBAC, notifications) from Git, applied by Argo CD.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd-settings
  namespace: argocd
spec:
  project: default
  source:
    repoURL: http://git-server.git-server.svc.cluster.local/bookstore-config
    targetRevision: main
    path: argocd/settings
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
      # Manage only the keys in Git; the install's defaults in the same ConfigMaps stay untouched.
      - ServerSideApply=true

Two details matter. argocd-cm contains defaults placed by the installation (such as resource.exclusions since 3.0); the file in Git carries only our keys. With server-side apply Argo CD owns only the fields it writes and the other keys stay where they are; since 3.6 the diff of applications with ServerSideApply=true is also computed with server-side diff. Second, every ConfigMap carries the argocd.argoproj.io/sync-options: Prune=false annotation and automated has no prune: a file deleted by mistake must not delete Argo CD's settings. On the first sync the three ConfigMaps were serverside-applied and resource.exclusions stayed as it was.

One step further is managing Argo CD itself, the installation manifest, with an Application: a kustomization.yaml takes the version-pinned install.yaml as a resource, the patches (the repo-server patch from the seventh post, HA settings) sit next to it, and an upgrade becomes a pull request that changes the version in the URL. Since 3.3 the ApplicationSet CRD exceeds the client-side apply limit, so such an Application needs ServerSideApply=true. We did not take that step in the lab; Argo CD stayed as installed by hand.

Users, SSO and RBAC

Authentication in Argo CD comes from two places: local accounts defined in argocd-cm, and SSO. In production people should come through SSO; local accounts remain for machines such as CI and for emergencies. With no SSO provider in the lab we created two local accounts: a developer who can log in (ada) and a CI account that only has an API key (ci-bot):

data:
  # KSOPS (post 07).
  kustomize.buildOptions: --enable-alpha-plugins --enable-exec
  # Local accounts: a person who logs in, and CI with an API key only.
  accounts.ada: login
  accounts.ci-bot: apiKey

Permissions live in argocd-rbac-cm, in Casbin format. A p line grants a role a permission (role, resource, action, object, effect), a g line binds a user or an SSO group to a role:

data:
  # Nobody gets anything by default; every right comes from a role below.
  policy.default: ""
  policy.csv: |
    # Developers: see every Bookstore app and its logs, sync only dev.
    p, role:bookstore-developer, applications, get, bookstore/*, allow
    p, role:bookstore-developer, logs, get, bookstore/*, allow
    p, role:bookstore-developer, applications, sync, bookstore/bookstore-*-dev, allow
    # Release: may sync every environment and restart workloads, may not delete anything.
    p, role:bookstore-release, applications, get, bookstore/*, allow
    p, role:bookstore-release, applications, sync, bookstore/*, allow
    p, role:bookstore-release, applications, action/argoproj.io/Rollout/*, bookstore/*, allow
    # Platform team: everything (an SSO group in a real setup).
    g, platform-team, role:admin
    g, ada, role:bookstore-developer
    g, ci-bot, role:bookstore-release

We logged in as ada and tried it. She sees the applications (only those in the bookstore project; the root and argocd-settings are not in the list), can sync dev, cannot sync prod and cannot delete anything:

$ argocd app sync bookstore-api-prod
{"level":"fatal","msg":"rpc error: code = PermissionDenied desc = permission denied: applications, sync, bookstore/bookstore-api-prod, sub: ada, iat: 2026-11-15T05:09:27Z"}
$ argocd app sync bookstore-web-dev
Phase:              Succeeded
Message:            successfully synced (all tasks run)
$ argocd app delete bookstore-web-dev
{"level":"fatal","msg":"rpc error: code = PermissionDenied desc = permission denied: applications, delete, bookstore/bookstore-web-dev, sub: ada, iat: 2026-11-15T05:09:27Z"}

Instead of trying each policy with a user, argocd admin settings rbac can checks it against the file; this is how CI can validate an RBAC change in a pull request:

argocd admin settings rbac can ci-bot sync applications bookstore/bookstore-api-prod --policy-file argocd-rbac-cm.yaml

ada get logs bookstore/bookstore-api-prod -> Yes
ada sync applications bookstore/bookstore-api-prod -> No
ci-bot get logs bookstore/bookstore-api-prod -> No
ci-bot sync applications bookstore/bookstore-api-prod -> Yes
ci-bot delete applications bookstore/bookstore-api-prod -> No
ci-bot action/argoproj.io/Rollout/restart applications bookstore/bookstore-api-prod -> Yes

For SSO Argo CD offers two routes: through the bundled Dex (with connectors such as GitHub, LDAP and SAML) or directly against an OIDC provider. Direct OIDC for Microsoft Entra ID looks like this:

data:
  url: https://argocd.example.com
  oidc.config: |
    name: Entra ID
    issuer: https://login.microsoftonline.com/<tenant-id>/v2.0
    clientID: <application-id>
    clientSecret: $oidc.entra.clientSecret
    requestedScopes: ["openid", "profile", "email"]
    requestedIDTokenClaims: {"groups": {"essential": true}}

The value $oidc.entra.clientSecret is read from argocd-secret; the secret is never written into the ConfigMap. Groups come from the groups claim of the token and are bound to RBAC with scopes: '[groups]' in argocd-rbac-cm: g, <group-id>, role:bookstore-developer. 3.0 brought a change for Dex users: the user identity in RBAC is no longer Dex's sub claim but federated_claims.user_id. If your policies name users by their sub value, update them before upgrading. This section is based on the official documentation.

RBAC Changes in 3.0: update, delete and logs

The answer to Guess First. Before 3.0, update or delete on an Application also applied to the resources that application manages. Not any more:

role:app-editor update applications bookstore/bookstore-api-prod -> Yes
role:app-editor update/argoproj.io/Rollout/bookstore-prod/api applications bookstore/bookstore-api-prod -> No
role:app-editor update/*/Pod/*/* applications bookstore/bookstore-api-prod -> No

The role can change the Application but not the Rollout or a Pod inside it; that needs a separate permission of the form update/<group>/<kind>/<namespace>/<name>. The old behaviour can be restored with server.rbac.disableApplicationFineGrainedRBACInheritance: "false" in argocd-cm, but the new one is safer. The second change is about logs: since 3.0 viewing logs is a separate permission (logs, get) that does not come with applications, get. In the output above ci-bot can sync prod but cannot see its logs. A 2.x installation without a policy.default and without log enforcement switched on finds its users without a logs tab after the upgrade; the fix is adding logs, get explicitly to every role.

A CI Token and Disabling the Admin Account

When CI needs to reach Argo CD (to trigger a sync, to wait for its status), it gets a time-limited token of its own account, not a person's password. We wrote the token to a file and never displayed it:

argocd account generate-token --account ci-bot --expires-in 1h > ci-bot-token.txt
argocd app sync bookstore-api-staging --auth-token "$(cat ci-bot-token.txt)"
argocd app delete bookstore-api-staging --yes --auth-token "$(cat ci-bot-token.txt)"

Phase:              Succeeded
Message:            successfully synced (no more tasks)
{"level":"fatal","msg":"rpc error: code = PermissionDenied desc = permission denied: applications, delete, bookstore/bookstore-api-staging, sub: ci-bot, iat: 2026-11-15T05:10:13Z"}

The AppProject role from the fifth post (proj:bookstore:deployer) does the same job at project level; the difference is where the permission is defined: a project role inside the project, a global role in argocd-rbac-cm. The last step is the admin account that comes with the installation. We changed its initial password in the second post; in production this account exists only for setup and is disabled once SSO works. We did that from Git as well (admin.enabled: "false"); after the sync:

$ argocd login 127.0.0.1:52611 --username admin
{"level":"fatal","msg":"rpc error: code = Unauthenticated desc = Account admin is disabled"}

Before disabling it there must be an SSO group bound to role:admin and a login that has been confirmed to work; otherwise you end up managing Argo CD only through kubectl.

Notifications and Metrics

In the eighth post an aborted Rollout was only noticed when someone looked. The Argo CD Notifications controller comes with the installation; its settings live in argocd-notifications-cm: a service (Slack, Microsoft Teams, e-mail or a plain webhook), message templates and triggers that say when to send. In the lab we installed a small webhook receiver that writes incoming requests to its log:

  service.webhook.lab: |
    url: http://webhook-receiver.lab-tools.svc.cluster.local:8080/argocd
    headers:
      - name: Content-Type
        value: application/json
  trigger.on-deployed: |
    - description: A sync succeeded and the application is Healthy
      oncePer: app.status.sync.revision
      send: [app-event]
      when: app.status.operationState != nil and app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
  trigger.on-health-degraded: |
    - description: The application is Degraded
      send: [app-event]
      when: app.status.health.status == 'Degraded'
  # Every Bookstore app, without annotating each generated Application.
  subscriptions: |
    - recipients: [lab]
      triggers: [on-deployed, on-health-degraded]
      selector: component in (api, web)

We defined the subscription globally in the ConfigMap because the Applications are generated by the ApplicationSet and their annotations come from the template. oncePer stops a second notification for the same revision. As soon as the settings were applied, every application sent one on-deployed notification (there had been none for that revision yet), followed by ada's sync:

POST /argocd HTTP/1.1
Host: webhook-receiver.lab-tools.svc.cluster.local:8080
User-Agent: Go-http-client/1.1
Content-Length: 176
Content-Type: application/json
Accept-Encoding: gzip

{"app": "bookstore-web-dev", "revision": "f10adb6a69763aaf9279107d78f79985630ed118",
 "sync": "Synced", "health": "Healthy",
 "message": "successfully synced (all tasks run)"}

Argo CD also ships a catalog of ready triggers and templates (on-sync-failed, on-health-degraded, Slack and Teams message formats); installing it and wiring the service is enough for most teams instead of writing everything from scratch. Notifications are for events; trends and alerts need metrics. The application controller's argocd-metrics Service (8082) publishes one argocd_app_info series per application:

argocd_app_info{autosync_enabled="true",dest_namespace="bookstore-prod",dest_server="https://kubernetes.default.svc",health_status="Healthy",hydrator_status="",name="bookstore-api-prod",namespace="argocd",operation="",phase="Succeeded",project="bookstore",repo="http://git-server.git-server.svc.cluster.local/bookstore-config",sync_status="Synced"} 1

Sync and health status are not separate metrics but labels of this series. 3.0 removed the old argocd_app_sync_status, argocd_app_health_status and argocd_app_created_time metrics entirely; the endpoint has not a single line with those names. Dashboards and alerts left over from 2.x have to move to these labels. The hydrator_status label arrived with 3.6. A Degraded alert can be written like this in Prometheus:

- alert: ArgoCDAppDegraded
  expr: argocd_app_info{health_status="Degraded"} == 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.name }} has been Degraded for 5 minutes"

Alongside these, argocd_app_sync_total, argocd_app_reconcile (a histogram of reconcile duration), argocd_cluster_connection_status and the repo-server's git request metrics show the health of Argo CD itself. 3.6 also added metrics for ApplicationSet progressive sync.

Scaling: HA, Sharding and Many Clusters

The lab installation is the non-HA install.yaml: one pod per component, a single Redis. For production there is manifests/ha/install.yaml: several replicas of the API server and the repo-server, and Redis with three replicas behind HAProxy (with Sentinel). Because of pod anti-affinity rules the HA installation needs at least three nodes. The repo-server is the most demanding part: every manifest generation is a kustomize build or a helm template, and plugins that decrypt secrets make it heavier; tune its replica count and reposerver.parallelism.limit first. This section is based on the official documentation.

Sharding comes in when the application controller manages several clusters: raise the replica count of the StatefulSet and set ARGOCD_CONTROLLER_REPLICAS to the same value, and clusters are split into shards; --sharding-method chooses legacy, round-robin or consistent-hashing. Sharding works per cluster: a single dev cluster with hundreds of applications still lands in one shard. For many clusters there are two models: a hub-spoke model where a central Argo CD manages every cluster (one UI, one RBAC, but the hub holds the credentials of every cluster), and one Argo CD per cluster (good isolation, more operational load). The cluster generator from the fifth post is how the same ApplicationSet spreads to every cluster in the hub-spoke model.

What Changed from 2.x to 3.6, and Upgrading

Argo CD releases four minor versions a year, and only the last three minors receive patches. 3.6.0 came out on 2026-11-03, so the supported versions are now 3.4, 3.5 and 3.6; 3.3 reached end of life the same day. Keeping an installation supported means upgrading roughly every six months. The important changes for someone coming from 2.x:

The upgrade path is reading the upgrade notes of every minor: going from 3.0 to 3.6 means all six notes in between apply. First a backup; argocd admin export writes every Application, ApplicationSet and project and Argo CD's ConfigMaps and Secrets into a single file (it contains secrets, so it goes to a vault, not to Git):

argocd admin export -n argocd > argocd-backup.yaml
grep '^kind:' argocd-backup.yaml | sort | uniq -c

   2 kind: AppProject
   8 kind: Application
   1 kind: ApplicationSet
   4 kind: ConfigMap
   2 kind: Secret

Then the whole manifest is applied with a pinned version. It is the same command as the installation in the second post:

kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.6.1/manifests/install.yaml

Changing only the image is not enough; parameter changes in the manifest would be missed. If Argo CD manages Argo CD, the same job is a pull request that changes the version in the URL; first on a test cluster, then in production. This section is based on the official documentation.

Production Notes

At the end of the series, here is the difference between "we set up GitOps" and "we run GitOps" as a checklist. Each item is something we tried in these nine posts:

• The whole desired state of the cluster is in Git; the only thing applied by hand is the App of Apps root (and after this post Argo CD's settings are in Git too).
• The config repo is separate from the application repo with one folder per environment; promotion is a commit or a pull request, rolling back is a revert.
• No kubeconfig in CI; the identities that write to the config repo are few, and the prod overlay sits behind branch protection and CODEOWNERS.
• Images are pinned by tag and digest; one writer per overlay (CI or Image Updater).
• Secrets are decrypted either in the cluster (Sealed Secrets, External Secrets Operator) or, with the risks understood, in the repo-server; the key is backed up and a disaster drill has been run.
• Migrations are ordered with hooks and waves and follow expand-contract; prod releases go through canary analysis.
• Every Application is in a project; people come through SSO, CI with a time-limited token; admin is disabled.
• Degraded and sync failures produce notifications; there is an alert on argocd_app_info; Argo CD is within the last three minors.

Argo CD is not the answer to every need. When should you look at Flux? If you want a model where everything is expressed as Kubernetes objects instead of a UI and a central control plane, if you prefer every one of many clusters to run its own reconciler, or if you want to manage Helm releases as HelmRelease objects and use OCI artifacts as sources, Flux's toolkit approach may fit better. Both projects are graduated in the CNCF; the comparison in the first post still holds. Argo CD's strengths are its application-centric UI, its project and RBAC model and its fit with tools from the same family such as Rollouts.

Common Pitfalls

Putting Argo CD's settings into Git and leaving prune on. When a file is deleted, argocd-cm is deleted and Argo CD's own settings are gone.

Disabling the admin account before SSO has been confirmed.

Leaving policy.default: role:readonly. Everyone sees every application of every project, the clusters and the repositories; the default should be empty.

Moving from 2.x to 3.x without reading the upgrade notes. Developers who lose the logs tab, empty dashboards and every resource being relabelled on the first sync are all described in those notes.

Giving CI the token of a personal account. When the person leaves, the pipeline stops; if a token without expiry leaks, it is hard to take back.

A Small Experiment

★☆☆ Prepare a change in a branch that lets role:bookstore-developer sync staging, and validate it against the file with argocd admin settings rbac can before merging.

★★☆ Install Argo CD's notifications catalog, connect the on-sync-failed trigger to the lab webhook and cause a failing sync in dev on purpose (an image tag that does not exist). Compare the message you receive with your own template.

★★★ Manage Argo CD itself with an Application: a kustomization.yaml that takes the version-pinned install.yaml as a resource, the repo-server patch from the seventh post and ServerSideApply=true. Then try moving from 3.6.1 to the next patch release as a pull request.

This post closes the GitOps and Argo CD series. We started from the drift a push-based pipeline cannot see; we installed Argo CD, separated sync from health, ordered migrations with hooks, generated six applications from a single ApplicationSet, turned CI into a part that promotes through Git, put secrets into Git safely, sent prod releases through metric-guarded canaries, and finally made Argo CD itself something a team can operate. What comes next is in the Helm series: there we write from scratch the Helm charts that we only touched as an Argo CD source type.