DevOps GitOps and Argo CD — #03

Sync, Health and Diff in Argo CD: Automated Sync, Prune, Self-Heal and Sync Options

Mert Özen Nov 15, 2026 10 min 8 views

In Argo CD, sync status says whether Git and the cluster match and health status says whether the objects in the cluster work; the two are independent. With real output we see a Synced but Degraded deploy, self-heal undoing a manual scale within a second while missing a manually added env variable, prune deleting a StatefulSet, and Prune=false, ignoreDifferences, RespectIgnoreDifferences and the other sync options.

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, Argo CD 3.6.1, Kustomize 5.8 and .NET 10.

Guess First

The dev Application of the Bookstore API has automated sync and self-heal enabled. The incident from the first post happens again: someone scales to 3 replicas with kubectl scale and adds Logging__LogLevel__Default=Debug with kubectl set env. How long does Argo CD take to undo these two changes? A hint: neither answer is "three minutes", and one of them is never undone.

Sync Status and Health Status: Two Different Questions

In the previous post we put Bookstore, the online bookstore used as the shared example across this site's series, into dev with Argo CD's first Application. Argo CD computes two independent states for every Application. Sync status compares Git with the cluster: Synced or OutOfSync. Health status looks at whether the objects in the cluster work: Healthy, Progressing, Degraded, Suspended, Missing or Unknown. To see why they are separate, we committed a tag that does not exist in the registry to dev. First we added progressDeadlineSeconds: 120 to the base; as we saw in the fourth post of the Kubernetes series, when that time runs out the Deployment marks its rollout as stuck.

Sync Status:        Synced to main (8852c9d)
Health Status:      Progressing

GROUP  KIND         NAMESPACE      NAME      STATUS  HEALTH       HOOK  MESSAGE
apps   Deployment   bookstore-dev  api       Synced  Progressing        deployment.apps/api unchanged

The sync succeeded: the Deployment from Git is in the cluster. The new pod is in ErrImagePull and the old 1.1.0 pod keeps carrying the traffic. Two minutes later:

Health Status:      Degraded

GROUP  KIND         NAMESPACE      NAME      STATUS  HEALTH    HOOK  MESSAGE
apps   Deployment   bookstore-dev  api       Synced  Degraded        deployment.apps/api unchanged

Updated health status: Progressing -> Degraded (Caused by apps/Deployment:bookstore-dev/api)
ReplicaSet "api-57ffbcfc94" has timed out progressing.

Synced and Degraded at the same time. The "(Caused by ...)" suffix in the event message is new in 3.6: it names up to three resources that caused the health change. The fix is in Git again: the git revert commit was applied by automated sync and the application went back to Healthy. Argo CD does not go back to the old version on its own; with automated sync enabled argocd app rollback cannot be used either, so the way back is Git.

How Is Health Computed?

Argo CD comes with built-in health checks for standard Kubernetes types. For Deployment, ReplicaSet, StatefulSet and DaemonSet it checks that the observed generation equals the desired generation and that the number of updated replicas has reached the desired count; when progressDeadlineSeconds is exceeded it becomes Degraded. A Service only waits for an address when it is of type LoadBalancer. A PersistentVolumeClaim must be Bound, a Job must complete. The application's overall health is the health of its worst resource; one Degraded Deployment makes the whole Application Degraded.

CRDs are different: every operator writes its status field in its own format. Argo CD ships ready-made Lua scripts for hundreds of CRDs; 3.6 added CloudNativePG's Database and Backup, cert-manager's CertificateRequest, the ACME Order and Challenge objects, Velero, Tekton, Kyverno, and Gateway API's TLSRoute and ListenerSet types. For a CRD without one you write your own Lua script under the resource.customizations.health.<group>_<kind> key in the argocd-cm ConfigMap; the script looks at the object in the obj variable and returns a status and a message. Another 3.6 change concerns resources being deleted: the default check now lists pending finalizers and custom checks can return a deletionMessage, so it is easier to see what an object stuck in "Pending deletion" is waiting for.

One more thing to know: the health check for the Application object itself was removed in 1.8. In an App of Apps setup, the parent Application does not wait for the health of its child Applications by default; we come back to this in the fifth post.

Automated Sync, Prune and Self-Heal

We added a sync policy to the Application definition in the config repo and applied it with kubectl apply:

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - PruneLast=true

Sync Policy:        Automated (Prune)
Sync Status:        Synced to main (274d30f)
Health Status:      Healthy

The three settings say three different things. automated: when the Application is OutOfSync, Argo CD starts the sync itself; when a new commit appears in Git we no longer need to type argocd app sync. Argo CD does not retry an automated sync that failed for the same commit, and it does not sync a second time if a sync for the same commit already succeeded, unless self-heal is on. selfHeal: it syncs even when the change comes from the cluster rather than Git. prune: it deletes from the cluster the resources removed from Git; it is off by default for safety. The first half of Guess First:

scale at 06:12:52.465
06:12:50.459 1
06:12:52.713 3
06:12:53.265 1

A loop reading spec.replicas every half second saw 3 only once; within a second the value was 1 again. The three-minute polling does not apply here: the controller watches the Kubernetes API and compares as soon as the live object changes. The Application's events:

Normal   OperationStarted     application/bookstore-api-dev   Initiated automated sync to '274d30fe5d72ce9c05587873ab0b800f9d2772cc'
Normal   ResourceUpdated      application/bookstore-api-dev   Updated sync status: Synced -> OutOfSync
Normal   ResourceUpdated      application/bookstore-api-dev   Updated health status: Healthy -> Progressing (Caused by apps/Deployment:bookstore-dev/api)
Normal   OperationCompleted   application/bookstore-api-dev   Sync operation to 274d30fe5d72ce9c05587873ab0b800f9d2772cc succeeded
Normal   ResourceUpdated      application/bookstore-api-dev   Updated sync status: OutOfSync -> Synced
Normal   ResourceUpdated      application/bookstore-api-dev   Updated health status: Progressing -> Healthy

With drift coming in back to back, Argo CD slows itself down: according to the defaults in argocd-cmd-params-cm, there is a backoff between self-heal attempts that starts at 2 seconds, grows by a factor of three and stops at 300 seconds. If another controller in the cluster keeps fighting over the same field (Argo CD writes, the other writes back), the backoff protects the API server but does not end the fight; ignoreDifferences below does.

Drift Self-Heal Does Not See

The second half of Guess First:

kubectl -n bookstore-dev set env deployment/api Logging__LogLevel__Default=Debug

Sync Status:        Synced to main (274d30f)
Health Status:      Healthy
ASPNETCORE_ENVIRONMENT Database__Provider ConnectionStrings__BookstoreDb Logging__LogLevel__Default

The env variable stayed and the Application is Synced. Health briefly went to Progressing (a new ReplicaSet was created) but sync status never became OutOfSync. The reason is the same as with kubectl apply in the first post: Argo CD's default diff is a three-way comparison using the last-applied-configuration annotation, and it treats a list element that was never in Git as "a field someone else added". This is not a bug but a deliberate design: otherwise no application could ever be Synced because of sidecars added by mutating webhooks or fields written by controllers. We tried again with the ServerSideApply=true sync option; in 3.6 this option also switches the diff to server-side diff, but the result did not change: the owner of the field is kubectl set env.

The way to remove this kind of drift is a replace sync:

argocd app sync bookstore-api-dev --replace --resource apps:Deployment:bookstore-dev/api

Deployment api deployment.apps/api replaced
ASPNETCORE_ENVIRONMENT Database__Provider ConnectionStrings__BookstoreDb

Replace rewrites the object exactly as it is in Git; every field that is not in Git goes away. That is also its danger: fields written by an HPA or an operator go too. And for an object whose immutable field changed, replace can go as far as deleting and recreating it. Do not make Replace=true a permanent sync option on an Application; when you see this kind of drift, use it by hand, for one resource and deliberately.

Prune: Removed from Git Means Deleted

With prune enabled, removing a file from Git means deleting that resource. Let us try an accident first. We commented out the resources line in the dev overlay and committed; the overlay now renders to nothing:

CONDITION  MESSAGE                                                                                                     LAST TRANSITION
SyncError  Skipping sync attempt to [5ca0c415e447b2d5e5ddc6008f0d60af3afad242]: auto-sync will wipe out all resources  2026-11-15 06:14:43 +0300 +03

Argo CD stopped. allowEmpty is off by default and an automated sync does not apply a "delete everything" result. This protection has a limit: it only kicks in when the result is completely empty. In the second attempt we removed only postgres.yaml from the base, as if we had moved the database to another folder and forgotten to commit the move:

GROUP  KIND         NAMESPACE      NAME      STATUS     HEALTH   HOOK  MESSAGE
apps   StatefulSet  bookstore-dev  postgres  Succeeded  Pruned         pruned
       Service      bookstore-dev  api       Synced     Healthy
apps   Deployment   bookstore-dev  api       Synced     Healthy
NAME                                    STATUS   VOLUME                                     CAPACITY
persistentvolumeclaim/data-postgres-0   Bound    pvc-ec9a9806-7bca-41b9-bd61-8359f3d7005c   1Gi

The StatefulSet and the postgres Service were deleted, 15 seconds later the API's readiness probe could not reach the database and the Application dropped to Progressing. The data survived: Argo CD did not create the PVC that comes from the StatefulSet's volumeClaimTemplates, so it does not track it and does not prune it. With a git revert commit the StatefulSet came back, bound to the same PVC, and the application was Healthy again.

For objects that are expensive to lose, such as a database, there is a per-resource brake:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  annotations:
    # Never delete the database automatically, even if it disappears from Git.
    argocd.argoproj.io/sync-options: Prune=false

We repeated the same accident with the annotation:

Sync Status:        OutOfSync from main (8cfb652)

GROUP  KIND         NAMESPACE      NAME      STATUS     HEALTH   HOOK  MESSAGE
apps   StatefulSet  bookstore-dev  postgres  OutOfSync  Healthy        ignored (no prune)

The StatefulSet stayed and the Application kept saying, as OutOfSync, "there is something here that is not in Git". We only annotated the StatefulSet, so the postgres Service was deleted again; the brake is applied object by object. PruneLast=true moves deletions to the end of the sync: new objects are created first, old ones deleted last. It avoids a gap when a resource is moved (deleted under an old name and created under a new one).

ignoreDifferences: Fields Git Does Not Own

What happens when an HPA changes the replica count while self-heal is on? Argo CD writes the value from Git back every time, the HPA changes it again. In the eleventh post of the Kubernetes series we saw this pitfall, and in the fourteenth its Kustomize-side fix: removing replicas from the manifest when an HPA exists. The Argo CD-side counterpart is to take the field out of the comparison:

  # An HPA (or a person during an incident) owns replicas, not Git.
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: api
      jsonPointers:
        - /spec/replicas

We scaled to 3 by hand and the Application stayed Synced. Then we made an unrelated change in Git (revisionHistoryLimit: 5) and the automated sync ran:

2d54b18 api: keep five old ReplicaSets
Sync Status:        Synced to main (2d54b18)
api   1/1   1     1     6m4s

Back to 1 replica. ignoreDifferences only affects the comparison; during a sync the whole manifest from Git is applied, and replicas: 1 goes with it. For the sync to leave those fields alone as well, a separate option is needed:

    syncOptions:
      - PruneLast=true
      - RespectIgnoreDifferences=true

5489d22 api: keep three old ReplicaSets
Sync Status:        Synced to main (5489d22)
api   3/3   3     3     6m21s
3

This time the sync set revisionHistoryLimit to 3 and the replica count stayed at 3. The same approach is used for fields added by mutating webhooks, caBundle values filled in by cert-manager and fields managed by operators; instead of jsonPointers you can also write jqPathExpressions, or managedFieldsManagers to leave out fields owned by a given field manager. If the same rule applies to all applications, it can be defined system-wide in argocd-cm.

Sync Options

Options that change sync behaviour are written on the Application (spec.syncPolicy.syncOptions) or on a single resource (the argocd.argoproj.io/sync-options annotation). The most common ones:

Option	What it does	Watch out
`CreateNamespace=true`	Creates the destination namespace if it is missing	Namespace labels need `managedNamespaceMetadata`
`ServerSideApply=true`	Uses kubectl's server-side apply; for large CRDs and field ownership	Since 3.6 the diff is server-side too; dry-run load on the API server
`PruneLast=true`	Moves deletions to the end of the sync	With waves in the fourth post
`ApplyOutOfSyncOnly=true`	Applies only OutOfSync resources	Shortens syncs for applications with thousands of resources
`Prune=false`	Protects a resource from prune	As an annotation, object by object
`Replace=true`	replace (or create) instead of apply	Removes fields others wrote; do not make it permanent
`RespectIgnoreDifferences=true`	Keeps the ignoreDifferences fields during sync too	Only effective on existing objects

In Production

Automated sync and self-heal can be on in dev from the start; in prod it makes sense to run manual syncs for a while first and watch what Argo CD sees. Before enabling prune, clear the OutOfSync list on the Application and put Prune=false on critical objects (database, PVCs, namespaces). Alert on health, not on sync status: a Synced but Degraded application is more urgent than an OutOfSync but Healthy one. The argocd_app_info metric and notifications exist for this (last post). A sync has no time limit either: controller.sync.timeout.seconds defaults to 0, so a stuck sync can wait forever; choose it and the retry settings deliberately for long jobs.

Common Pitfalls

Trying a hotfix with kubectl during an incident while self-heal is on. The change is undone within a second, and realising that in the middle of an incident costs time. Either write the fix to Git, or first turn off automated sync on the Application (spec.syncPolicy.automated.enabled: false) and then reflect that in Git too.

Enabling prune on day one and emptying a namespace with a wrong path or a missing resources line. allowEmpty only stops a completely empty result, not a commit that deletes half of it.

Reading Synced as "all is well". Synced and Degraded can happen together.

Writing ignoreDifferences and forgetting RespectIgnoreDifferences; the HPA's decision is overwritten on the next sync.

Believing self-heal closes every drift. Fields that were never in Git do not show up in the diff.

A Small Experiment

★☆☆ With self-heal on, add a new label to the Deployment with kubectl label, then change the value of the app.kubernetes.io/part-of label that is already in Git. Which one does Argo CD undo? Why?

★★☆ Add an HPA (if metrics-server is not installed, minReplicas: 2 is enough) and watch through the events how Argo CD and the HPA fight over the replica field without ignoreDifferences. Then try two fixes: removing replicas from the overlay, and ignoreDifferences with RespectIgnoreDifferences.

★★★ For your own CRD (for example a Book CRD) write a Lua health check that looks at status.phase: Healthy for Ready, Degraded for Failed, Progressing otherwise. Add it to argocd-cm and verify the result in the UI by changing an object's state.

We now know what sync, health and diff see. In the next post we look at the order inside a sync: sync waves and hooks for running the migration Job before the deploy.