Sync Waves and Hooks: Running the Migration Job Before the Deploy

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, Argo CD 3.6.1, .NET 10, EF Core 10.0.15, Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3 and PostgreSQL 18.

Guess First

The Bookstore API's database migration sits in the config repo as a Kubernetes Job and Argo CD applies it together with the other manifests. The first deploy went fine. For the next release you commit a change that moves the migrator image tag from 1.0.0 to 1.1.0. Automated sync is on. What happens: does the Job run again with the new image, is the old Job deleted and a new one created, or does the sync fail?

Putting the Migration in a Plain Job

The API of Bookstore, the online bookstore used as the shared example across this site's series, uses EF Core and builds its schema with migrations. In the thirteenth post of the Kubernetes series we separated the migration from the Deployment and ran it as a Job; we build the same idea with Argo CD. In this series the API uses PostgreSQL inside the cluster, so the PostgreSQL migrations live in a separate project (Bookstore.Migrations.Postgres), and a small console app in that project applies the pending migrations and exits:

using Bookstore.Data;
using Bookstore.Migrations.Postgres;
using Microsoft.EntityFrameworkCore;

// Applies pending PostgreSQL migrations and exits: 0 = success, 1 = failure.
// Connection string: ConnectionStrings__BookstoreDb environment variable.
await using BookstoreContext db = new DesignTimeContextFactory().CreateDbContext(args);

try
{
    List<string> pending = [.. await db.Database.GetPendingMigrationsAsync()];
    Console.WriteLine(pending.Count == 0
        ? "No pending migrations."
        : $"Applying {pending.Count} migration(s): {string.Join(", ", pending)}");
    await db.Database.MigrateAsync();
    Console.WriteLine($"Done. Last migration: {(await db.Database.GetAppliedMigrationsAsync()).LastOrDefault()}");
    return 0;
}
catch (Exception ex)
{
    Console.Error.WriteLine($"Migration failed: {ex.GetType().Name}: {ex.Message}");
    return 1;
}

Why not the migration bundle (dotnet ef migrations bundle) from the Kubernetes series? The migrations project references Bookstore.Api, which is an executable project, and the bundle build stopped with NETSDK1152 because of identically named apphost and runtimeconfig.json files coming from two different RID outputs. With the context in a separate class library the bundle would work; to keep Bookstore's structure intact, a migrator of a few lines was simpler. For Argo CD there is no difference: both are images that "run, finish, exit".

The Job manifest went into the base:

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
  labels:
    app.kubernetes.io/name: migrate
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 300
  template:
    metadata:
      labels:
        app.kubernetes.io/name: migrate
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: gitopsseries-bookstore-migrator
          env:
            - name: ConnectionStrings__BookstoreDb
              valueFrom:
                secretKeyRef:
                  name: api-db
                  key: connection-string

NAME      STATUS     COMPLETIONS   DURATION   AGE
migrate   Complete   1/1           3s         20s
Applying 1 migration(s): 20261115025642_InitialCreate
Done. Last migration: 20261115025642_InitialCreate

The Job ran and the catalog was filled; /api/books?pageSize=2 now returns "total":20. At the top of the log there are two more lines about the Kerberos library Npgsql cannot find in the chiseled image; as explained in the thirteenth post of the Kubernetes series, they are harmless if you do not use Kerberos. Now the commit from Guess First:

batch  Job          bookstore-dev  migrate   OutOfSync  Healthy        error when patching "/dev/shm/2661775648": Job.batch "migrate" is invalid: spec.template: Invalid value: {...}: field is immutable

The answer is the third option: the sync fails. A Job's pod template is immutable; Kubernetes refuses to write the new image into the existing Job. In the Kubernetes series we solved this by putting the version in the Job name (migrate-1-1-0). Argo CD has a cleaner way: turn the Job from a normal resource into a hook.

Sync Phases and Hooks

Argo CD splits a sync into phases. PreSync, Sync and PostSync run in order; if hooks in a phase fail, the sync stops and SyncFail hooks run. What makes a resource a hook is the argocd.argoproj.io/hook annotation. Hooks are created again on every sync and Argo CD waits for them to finish; for a Job "finished" means Complete, and failure means Failed. When they are deleted is decided by argocd.argoproj.io/hook-delete-policy:

• BeforeHookCreation: the old hook with the same name is deleted on the next sync before the new one is created. This is the default when no policy is written.
• HookSucceeded: the hook is deleted when it succeeds.
• HookFailed: the hook is deleted when it fails.

Our first attempt was to put the migration in PreSync, as in the plan. We added a PostSync smoke test and a SyncFail notice next to it:

apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 120
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke-test
          image: curlimages/curl:8.17.0
          args:
            - --fail
            - --silent
            - --show-error
            - --retry
            - "5"
            - --retry-all-errors
            - --max-time
            - "5"
            - http://api/api/books?pageSize=1

apiVersion: batch/v1
kind: Job
metadata:
  name: sync-failed
  annotations:
    argocd.argoproj.io/hook: SyncFail
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: notify
          image: curlimages/curl:8.17.0
          # In a real setup this posts to a chat/webhook; here it only writes a log line.
          command: ["sh", "-c", "echo \"sync of $(APP) failed, migrations or deploy did not finish\""]
          env:
            - name: APP
              value: bookstore-api

After the commit something unexpected happened: Argo CD deleted the old, non-hook migrate Job as "pruned" and showed the application as Synced, but no hook ran. The event gave the reason: "Partial sync operation ... succeeded". Self-heal had started a selective sync that only syncs the OutOfSync resource, and the docs are clear: hooks do not run during a selective sync. We started a full sync by hand:

argocd app sync bookstore-api-dev

Phase:              Succeeded
Duration:           18s
Message:            successfully synced (no more tasks)

GROUP  KIND         NAMESPACE      NAME        STATUS     HEALTH   HOOK      MESSAGE
batch  Job          bookstore-dev  migrate     Succeeded  Synced   PreSync   Reached expected number of succeeded pods
       Service      bookstore-dev  postgres    Synced     Healthy            service/postgres unchanged
       Service      bookstore-dev  api         Synced     Healthy            service/api unchanged
apps   Deployment   bookstore-dev  api         Synced     Healthy            deployment.apps/api unchanged
apps   StatefulSet  bookstore-dev  postgres    Synced     Healthy            partitioned roll out complete: 1 new pods have been updated...
batch  Job          bookstore-dev  smoke-test  Succeeded  Synced   PostSync  Reached expected number of succeeded pods

The migration ran in PreSync and the smoke test in PostSync; the migrator said "No pending migrations." because the schema was already up to date. The smoke test was deleted after succeeding because of HookSucceeded. The immutable error is gone as well: BeforeHookCreation deletes the old Job on every sync and creates a new one, so the Job's template is never updated.

The Problem with PreSync in an Empty Environment

In dev the database was already running. What about a new environment? We deleted the Application with argocd app delete bookstore-api-dev (its resources went too because of the finalizer) and recreated it from the same YAML; the PVC holding PostgreSQL's data and the Secret we keep outside Git stayed, everything else was deleted. A minute later:

GROUP  KIND         NAMESPACE      NAME         STATUS     HEALTH   HOOK     MESSAGE
batch  Job          bookstore-dev  migrate      Running    Synced   PreSync  job.batch/migrate created
       Service      bookstore-dev  api          OutOfSync  Missing
       Service      bookstore-dev  postgres     OutOfSync  Missing
apps   Deployment   bookstore-dev  api          OutOfSync  Missing
apps   StatefulSet  bookstore-dev  postgres     OutOfSync  Missing
batch  Job          bookstore-dev  sync-failed             Healthy           

Migration failed: NpgsqlException: Name or service not known
sync of bookstore-api failed, migrations or deploy did not finish
one or more synchronization tasks completed unsuccessfully. Retrying attempt #3 at 2026-11-15T03:25:32Z.

PreSync runs before everything in the Sync phase, and the database's StatefulSet and Service are in the Sync phase too. The migration could not resolve the not-yet-existing postgres name, the Job failed after three attempts, the sync counted as failed and the sync-failed hook wrote its message to the log. Automated sync's default retry kicked in and started producing the same error again and again. We stopped the loop with argocd app terminate-op bookstore-api-dev. The "migration in PreSync" recipe is right when the database is managed in another Argo CD Application or outside the cluster; when it is in the same Application, the order is wrong.

Sync Waves: Order Within a Phase

Phases give the coarse order; the argocd.argoproj.io/sync-wave annotation decides the order within a phase. The value is an integer, defaults to 0 and can be negative. Argo CD sorts resources by phase, then wave, then kind (namespaces first, for example) and then name; it does not move to the next wave until everything in the current wave is Healthy, and there is a default delay of 2 seconds between waves. We moved the migration into the Sync phase as a hook and put the database in the wave before it and the API in the wave after:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  annotations:
    argocd.argoproj.io/sync-options: Prune=false
    argocd.argoproj.io/sync-wave: "-1"

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate
  annotations:
    # A Sync hook in wave 0: runs after postgres (wave -1) is healthy, before the api (wave 1).
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/sync-wave: "0"
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  annotations:
    argocd.argoproj.io/sync-wave: "1"

We set "-1" on the postgres Service as well. After the commit, automated sync built the empty environment from scratch:

GROUP  KIND         NAMESPACE      NAME         STATUS     HEALTH   HOOK      MESSAGE
       Service      bookstore-dev  postgres     Synced     Healthy            service/postgres created
apps   StatefulSet  bookstore-dev  postgres     Synced     Healthy            partitioned roll out complete: 1 new pods have been updated...
       Service      bookstore-dev  api          Synced     Healthy            service/api created
batch  Job          bookstore-dev  migrate      Succeeded  Synced   Sync      Reached expected number of succeeded pods
apps   Deployment   bookstore-dev  api          Synced     Healthy            deployment.apps/api created
batch  Job          bookstore-dev  smoke-test   Succeeded  Synced   PostSync  Reached expected number of succeeded pods
batch  Job          bookstore-dev  sync-failed             Healthy            

The timeline we pulled from the namespace events shows the waves really flowing in order: the postgres pod was created at 03:25:57, the migration pod at 03:25:59 and it finished at 03:26:02; the API pod started at 03:26:07, the smoke test at 03:26:09 and it completed at 03:26:12. This time the migrator said "No pending migrations.": the PVC was not deleted, so the schema was still there. The sync-failed in the list is the hook left over from the previous failed sync; BeforeHookCreation only deletes it at the next SyncFail.

There is a deletion side too: the PreDelete hook added in 3.3 runs only when the Application is being deleted, before its resources are deleted (not during normal syncs or prune), and PostDelete runs after the deletion finishes. They are used to back up a database or clean up an external record before removing an environment. During prune the wave order is reversed: resources in higher waves are deleted first.

Fitting Expand-Contract

Waves solve ordering but do not change one fact: the moment the migration finishes, the old version's pods are still running and talking to the new schema. During the rolling update in wave 1, old and new versions take traffic together. The expand-contract rule from the "Where Migrations Fit in a Deploy" post of the Deployment Strategies series applies here too: a migration that runs inside the sync must only be an expand step (a new column, a new table, a nullable field). Contract steps such as dropping or renaming a column belong to a separate deploy after the old version is fully gone. The Argo CD hook does not enforce that rule for you; running a migration "before the deploy" does not make it safe. A concrete Bookstore example: you want to rename the Description column of the catalog.Books table to Summary. Do it in one migration and wave 0 renames the column, while old pods keep querying Description until wave 1's rolling update finishes, and the book details page fails. With expand-contract you first add the new column and the application writes to both; reads move to the new column; only when the old version runs nowhere does a separate commit drop the old column. Every step is a separate sync, a separate commit and, if needed, a separate git revert.

In Production

Hook Jobs must be idempotent. A hook runs on every sync: every manually started sync, every retry and every new environment runs the migration again, even without a change. EF Core's __EFMigrationsHistory table and the migration lock that came with EF Core 9 make that safe; for a hand-written SQL script you have to provide the same guarantee. Watch out for long migrations: activeDeadlineSeconds on the Job cuts the migration off, while Argo CD's sync has no time limit by default (controller.sync.timeout.seconds: "0"); a stuck hook can keep a sync "Running" for hours. Do not use ttlSecondsAfterFinished on hook Jobs: as the docs warn, if Kubernetes deletes the Job before Argo CD reads its result, the sync can end up waiting for a hook that no longer exists; leave cleanup to hook-delete-policy. Finally, the docs suggest SyncFail hooks for cleanup tasks; for notifications Argo CD Notifications is the sturdier route (last post).

Common Pitfalls

Leaving the migration as a plain Job. You get the immutable template error on the second release.

Putting the migration in PreSync while the database is in the same Application. It works in existing environments and fails on the first install or a rebuild after a disaster.

Thinking waves work across Applications. A wave orders things only inside a single Application's sync; a wave in bookstore-api does not hold back bookstore-web. Ordering across applications needs waves on the Applications themselves in an App of Apps, and the Application health check turned back on (fifth post).

Assuming self-heal runs hooks. Self-heal only syncs the drifted resource, and hooks do not run in a selective sync.

Thinking migrations can be rolled back release by release. git revert rolls back the image, not the schema.

A Small Experiment

★☆☆ Change the smoke-test URL to /api/does-not-exist and commit. What is the result of the sync, does the sync-failed hook run? What does the application's health show?

★★☆ Write a migration that adds a nullable Subtitle column to Bookstore, put the new migrator and API images into the overlay in the same commit, and verify that the old pod still answers fine during the rollout. Then explain why the migration that drops the column belongs to a separate deploy.

★★★ Add a PreDelete hook that backs up the database with pg_dump to a PVC before the Application is deleted. Show that a failing hook blocks the deletion and leaves the Application waiting in DeletionError.

The order inside a single Application is now ours. In the next post we scale up: generating the api and web applications for dev, staging and prod with a single ApplicationSet, and drawing boundaries with App of Apps and AppProject.