Jobs, CronJobs and Workers: Running an EF Core Migration Bundle as a Job

Version note: The examples in this post were run with kind 0.33 and Kubernetes 1.37, kubectl 1.37, .NET 10, EF (Entity Framework) Core 10.0.14, Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3 and PostgreSQL 18.

Guess First

The Bookstore API (Application Programming Interface) has three replicas, and each pod runs EF Core migrations in an init container on startup. Two migrations are to be applied to an empty database: InitialCreate and AddOrderStatus. All three pods start in the same second. Since EF Core 9 there is a database lock for migrations. What do you think happens: do the three pods wait their turn and produce one clean result, or does one of them fail and restart? The answer is in the real logs below; no spoilers, but it's too early to say "there's a lock, no problem".

Job: Work That Runs Once and Finishes

A Deployment's job is to keep pods up forever; if the process exits it restarts it. For "run, finish, exit" work like migrations, report generation or data-fix scripts there is the Job: it runs a pod until it succeeds (retrying if needed), then stops. Bookstore's migration Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-1-0-0
  namespace: bookstore
  labels:
    app.kubernetes.io/name: migrate
    app.kubernetes.io/part-of: bookstore
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 300
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app.kubernetes.io/name: migrate
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: bookstore-migrations:1.0.0
          args: ["--connection", "$(DB_CONNECTION)"]
          env:
            - name: DB_CONNECTION
              valueFrom:
                secretKeyRef:
                  name: api-db
                  key: connection-string
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi

Three fields define the Job's behavior. backoffLimit: 2 means at most two retries after the first attempt; between attempts there is a delay that starts at 10 seconds and doubles. activeDeadlineSeconds: 300 caps the Job's total lifetime regardless of the attempt count; a hung migration doesn't hold the deploy forever. ttlSecondsAfterFinished: 3600 deletes the finished Job and its pods after an hour; otherwise one Job piles up per deploy. restartPolicy: Never runs each attempt in a new pod, so failed pods stay around with their logs for inspection; OnFailure restarts the container in the same pod, and the previous attempt's log is only visible with --previous.

The $(DB_CONNECTION) in args is not a shell; it is Kubernetes' own variable expansion: it puts the value from the container's env into the argument. Since the chiseled image has no shell, this is the cleanest way to pass the connection string from a Secret to the bundle.

Packaging the EF Core Migration Bundle as an Image

The migration bundle is the single-file migration runner we covered in the EF Core 7 post: it applies pending migrations to the target database without needing the SDK (Software Development Kit) or source code. In Kubernetes we put it in an image. We produce the bundle self-contained for linux-arm64 in the SDK image and copy it into the runtime-deps chiseled image, which doesn't even contain the .NET runtime:

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY dotnet-tools.json .
COPY Bookstore.Migrations.csproj .
RUN dotnet tool restore && dotnet restore
COPY . .
RUN dotnet ef migrations bundle --self-contained -r linux-arm64 -o /out/efbundle

FROM mcr.microsoft.com/dotnet/runtime-deps:10.0-noble-chiseled
COPY --from=build /out/efbundle /efbundle
ENTRYPOINT ["/efbundle"]

dotnet-ef is not global; it comes from the project's tool manifest (dotnet-tools.json). With the .NET SDK 10.0.402 we used, dotnet new tool-manifest wrote this file to the project root rather than a .config folder, and the Dockerfile's COPY line follows that. In production the target architecture must match your cluster nodes (linux-x64 for most cloud nodes). The image is 47.5 MB. We applied the Job:

job.batch/migrate-1-0-0 created
job.batch/migrate-1-0-0 condition met
NAME            STATUS     COMPLETIONS   DURATION   AGE
migrate-1-0-0   Complete   1/1           3s         3s

Cannot load library libgssapi_krb5.so.2
Error: libgssapi_krb5.so.2: cannot open shared object file: No such file or directory
Failed executing DbCommand (9ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT "MigrationId", "ProductVersion"
FROM "__EFMigrationsHistory"
ORDER BY "MigrationId";
Acquiring an exclusive lock for migration application. See https://aka.ms/efcore-docs-migrations-lock for more information if this takes too long.
Applying migration '20261108040417_InitialCreate'.
Done.

The two "errors" in the log are harmless and good to know: Npgsql looks for the Kerberos (GSSAPI) library, which isn't in the chiseled image; if you don't use Kerberos authentication it doesn't matter. The failed __EFMigrationsHistory query is also expected on an empty database; EF Core creates the table right after. When you write alert rules, don't count these two lines as errors.

A Failing Job and Backoff

The same Job with a wrong password in the connection string:

NAME             STATUS   COMPLETIONS   DURATION   AGE
migrate-broken   Failed   0/1           34s        34s
NAME                   READY   STATUS   RESTARTS   AGE
migrate-broken-jkp7w   0/1     Error    0          34s
migrate-broken-wsxrp   0/1     Error    0          3s
migrate-broken-x4wmd   0/1     Error    0          23s
Normal   SuccessfulCreate      34s   job-controller  Created pod: migrate-broken-jkp7w
Normal   SuccessfulCreate      23s   job-controller  Created pod: migrate-broken-x4wmd
Normal   SuccessfulCreate      3s    job-controller  Created pod: migrate-broken-wsxrp
Warning  BackoffLimitExceeded  0s    job-controller  Job has reached the specified backoff limit

Npgsql.PostgresException (0x80004005): 28P01: password authentication failed for user "bookstore"

Three pods, 11 and 20 seconds apart: you can clearly see the backoff doubling. A wrong password doesn't get fixed by retrying; for such permanent failures podFailurePolicy, GA (General Availability) since Kubernetes 1.31, can fail the Job on the first attempt for specific exit codes. The bundle exits with 1 on failure and reports transient connection errors with the same code, so we stayed with a plain backoffLimit.

Also know that a Job is immutable: its pod template can't be changed. When we tried to apply a new image to the Job with the same name:

The Job "migrate-1-0-0" is invalid: spec.template: Invalid value: {...}: field is immutable

That's why we put the version into the Job name (migrate-1-1-0). The new Job applied only the pending migration:

Acquiring an exclusive lock for migration application. See https://aka.ms/efcore-docs-migrations-lock for more information if this takes too long.
Applying migration '20261108041246_AddOrderStatus'.
Done.
          MigrationId          | ProductVersion
-------------------------------+----------------
 20261108040417_InitialCreate  | 10.0.14
 20261108041246_AddOrderStatus | 10.0.14

Deploy Order: Migration First, Then the App

The migration Job should be a step of the deploy: the new API image isn't rolled out until the Job finishes, and if the Job fails the deploy stops. kubectl wait --for=condition=complete alone is not enough; if the Job fails it waits until the timeout. The script we use in the pipeline catches both cases:

#!/usr/bin/env bash
# usage: ./run-migration.sh <job-manifest> <job-name>
set -euo pipefail
manifest="$1"
job="$2"

kubectl apply -f "$manifest"

for _ in $(seq 1 60); do
    conditions=$(kubectl -n bookstore get job "$job" \
        -o jsonpath='{.status.conditions[?(@.status=="True")].type}')
    case "$conditions" in
        *Complete*)
            kubectl -n bookstore logs "job/$job" --tail=2
            echo "migration job $job completed"
            exit 0
            ;;
        *Failed*)
            kubectl -n bookstore logs "job/$job" --tail=1
            echo "migration job $job failed ($conditions), stopping the deploy" >&2
            exit 1
            ;;
    esac
    sleep 5
done

echo "migration job $job did not finish in time" >&2
exit 1

$ ./run-migration.sh m-ok.yaml migrate-1-1-0-rerun
job.batch/migrate-1-1-0-rerun created
No migrations were applied. The database is already up to date.
Done.
migration job migrate-1-1-0-rerun completed

$ ./run-migration.sh m-bad.yaml migrate-1-2-0
job.batch/migrate-1-2-0 created
Found 3 pods, using pod/migrate-1-2-0-hdf2g
28P01: password authentication failed for user "bookstore"
migration job migrate-1-2-0 failed (FailureTarget Failed), stopping the deploy

The second call returned exit code 1, so the pipeline's next step (the API rollout) doesn't run. Tools do the same thing in their own ways: a pre-upgrade hook in Helm (in the upcoming Helm series), sync waves and a PreSync hook in Argo CD (CD: Continuous Delivery; in the upcoming GitOps and Argo CD series). Whatever the order, one rule doesn't change: while the migration runs, pods of the old version are still taking traffic. The schema change must also work with the old code; the way to do that is the expand-contract approach from the migrations post of the Deployment Strategies series.

Pitfall: Running Migrations in Every Replica's Init Container

The easy-looking alternative is to add the bundle to the API Deployment as an init container: no separate pipeline step, every pod updates the database as it starts. We set up the scenario from Guess First: three replicas, an empty bookstore_init database, two pending migrations.

      initContainers:
        - name: migrate
          image: bookstore-migrations:1.1.0
          args: ["--connection", "$(DB_CONNECTION)"]
          env:
            - name: DB_CONNECTION
              valueFrom:
                secretKeyRef:
                  name: api-db-init
                  key: connection-string

The init container logs of the three pods, within the same second:

== api-init-5f48695b4b-vx6wn
Acquiring an exclusive lock for migration application. ...
Applying migration '20261108040417_InitialCreate'.
Applying migration '20261108041246_AddOrderStatus'.
Acquiring an exclusive lock for migration application. ...
Failed executing DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
ALTER TABLE "Orders" ADD "Status" text NOT NULL DEFAULT '';
Npgsql.PostgresException (0x80004005): 42701: column "Status" of relation "Orders" already exists
# second attempt after the init container restart
No migrations were applied. The database is already up to date.

== api-init-5f48695b4b-mvbrm
Acquiring an exclusive lock for migration application. ...
Applying migration '20261108041246_AddOrderStatus'.
Done.

== api-init-5f48695b4b-btk44
No migrations were applied. The database is already up to date.

There was a lock, but it didn't prevent the race. While vx6wn applied the first migration, mvbrm waited, and when the lock was released it applied the second migration itself; vx6wn, following the list it computed at the start, tried to add the same column and got error 42701. The log shows vx6wn acquiring the lock twice. Because the migration runs in a transaction the database stayed consistent, and after the init container's restart everything looked fine; so no harm this time. But luck decided the outcome, not design. More importantly, this approach has other problems: every replica connects with a user that has DDL (Data Definition Language) rights, a long migration holds up the startup of every pod, a broken migration sends every pod into Init:CrashLoopBackOff, and the migration is retried on scale-out and even on every pod restart. A migration should run once, in a separate Job, at a defined moment of the deploy.

CronJob: Scheduled Jobs

A CronJob creates Jobs on a schedule. In Bookstore, the nightly job that cleans up unpaid orders:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cleanup-pending-orders
  namespace: bookstore
spec:
  schedule: "30 3 * * *"
  timeZone: Europe/Istanbul
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 600
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: cleanup
              image: postgres:18-alpine
              command: ["psql", "$(DB_URL)", "-c"]
              args:
                - DELETE FROM "Orders" WHERE "Status" = 'Pending' AND "CreatedAt" < now() - interval '1 day';
              env:
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: postgres
                      key: password
                - name: DB_URL
                  value: postgresql://bookstore@postgres:5432/bookstore

NAME                     SCHEDULE     TIMEZONE          SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cleanup-pending-orders   30 3 * * *   Europe/Istanbul   False     0        <none>          0s

Without the timeZone field (GA since 1.27) the schedule is interpreted in kube-controller-manager's time zone; on managed clusters that is usually UTC (Coordinated Universal Time), and a "03:30 at night" job runs at 06:30 Turkey time. The old CRON_TZ= prefix is now rejected, and so is an invalid time zone:

The CronJob "cleanup-pending-orders" is invalid: spec.schedule: Invalid value: "CRON_TZ=UTC 30 3 * * *": cannot use TZ or CRON_TZ in schedule, use timeZone field instead
The CronJob "cleanup-pending-orders" is invalid: spec.timeZone: Invalid value: "Turkey/Ankara": not a directory

concurrencyPolicy: Forbid doesn't start a new run before the previous one finishes; for jobs that touch the same data, like cleanups or reports, the default Allow is dangerous. startingDeadlineSeconds allows the controller to start a run at most 10 minutes late if it missed the time for some reason (e.g. a control plane upgrade). To test without waiting for the schedule, create a Job from the CronJob by hand:

$ kubectl -n bookstore create job cleanup-manual-1 --from=cronjob/cleanup-pending-orders
job.batch/cleanup-manual-1 created
DELETE 1
 Id | Customer | Status
----+----------+---------
  2 | Grace    | Pending
  3 | Alan     | Paid

The details of cron expressions and time zone/DST (Daylight Saving Time) pitfalls are in the cron post of the Background Jobs series. A Kubernetes CronJob gives an "approximately once" guarantee, not at-least-once: rarely it may run twice or not at all. The job must be idempotent; the DELETE above is.

Workers: A Deployment, Not a Job

A .NET Worker Service that listens to a queue is not a "finish and exit" job; it runs continuously. Its place is a Deployment: like Bookstore.Worker in the eleventh post, with its replica count decided by KEDA (Kubernetes Event-driven Autoscaling) if needed. Since a worker has no HTTP (Hypertext Transfer Protocol) endpoint, its liveness probe needs an exec command or a small health endpoint; graceful shutdown depends on the BackgroundService's stoppingToken (the IHostedService post of the Background Jobs series and the tenth post of this series). You don't have to turn an app with its own scheduler, such as Hangfire or Quartz.NET, into CronJobs; but to keep the same job from running twice across replicas you need those libraries' distributed lock.

Production Notes

The migration Job's log is part of the deploy record: if ttlSecondsAfterFinished is too short, there's no pod left to inspect after a failure; collecting logs centrally is the topic of the upcoming Observability series. Separate the user that runs migrations from the app's user: the Job runs with a Secret that has DDL rights, the API with a Secret that has only DML (Data Manipulation Language) rights. On large tables, migrations that hold locks (creating an index, changing a column type) can take minutes; choose activeDeadlineSeconds accordingly and do such changes in lock-free ways, like CREATE INDEX CONCURRENTLY in PostgreSQL. CronJobs failing silently is the most common production problem: an alert watching the time of the last successful run matters as much as the job itself. Finally, give Job pods resource requests and limits too; an unlimited report Job can squeeze the API pods on its node.

Common Pitfalls

Running migrations in an init container for every replica. Applying a new image to a Job with the same name and hitting "field is immutable"; change the name with the version or delete it first. Waiting 5 minutes on a failed Job with kubectl wait --for=condition=complete. Choosing restartPolicy: OnFailure and losing the logs of failed attempts. Not setting ttlSecondsAfterFinished and accumulating hundreds of finished Jobs. A "nightly" job that runs on UTC because the CronJob has no timeZone. Leaving concurrencyPolicy at the default Allow and running two long reports at the same time. And forgetting that a migration lives alongside pods of the old version, then dropping or renaming a column.

A Small Experiment

★☆☆ Set the CronJob's schedule to */1 * * * *, give it successfulJobsHistoryLimit: 1 and watch kubectl get jobs -w for five minutes. When are old Jobs deleted?

★★☆ Add a podFailurePolicy to the migration Job that applies action: FailJob when the container exit code is 1. Run it again with the wrong-password Secret and compare the number of pods created with the earlier output. What does this policy cost you on a transient network error?

★★★ Repeat the init container scenario three times with 5 replicas and note which pod applied which migration each time. Then move the migrations to a single Job and write a small deploy script that starts the API Deployment's rollout only after run-migration.sh succeeds.

In this post we edited the same manifests by hand for each environment: the version in the Job name, the image tag, the Secret name. In the next post we set up bases and overlays with Kustomize and manage the differences between dev, staging and prod without copy-paste.