Scheduling with Quartz.NET 4: Jobs, Triggers, Persistent JobStore and Clustering

Version note: The code in this post was built and run with .NET 10, Quartz.NET 4.7.2, Npgsql 10.0.3 and PostgreSQL 17.

Guess First

Bookstore pulls the supplier's price list page by page: the next page every 10 seconds, each page taking 6 seconds. The job runs in a two-node Quartz cluster sharing the same PostgreSQL database. It's marked so it never runs twice at the same time, and it keeps the page it reached in the database. One of the nodes dies by kill -9 while processing page 3. When does the other node notice? Is page 3 left unprocessed, processed once, or twice?

Jobs and Triggers: Two Separate Objects

In Hangfire, the "work" and the "when" are in the same call: Schedule(() => ..., TimeSpan). Quartz.NET separates them. The job (IJobDetail) carries what runs, its identity (JobKey) and the job's data (JobDataMap); the trigger (ITrigger) carries when it runs. One job can have several triggers: the same price sync every 10 minutes on weekdays and hourly at weekends.

In Quartz 4, the job class's signature changed:

// IJob in Quartz 4.7.2, reflection output
TYPE Quartz.IJob
  System.Threading.Tasks.ValueTask Execute(Quartz.IJobExecutionContext, System.Threading.CancellationToken)

The return type is ValueTask, and the CancellationToken is now a parameter. In 3.x the token was in context.CancellationToken; as a parameter, the CA2016 code analysis rule can catch it when you forget to pass the token to lower calls. We'll see this and the other breaking changes with real compiler output in the last post of the series, when moving from 3.x to 4.x.

The most visible change in 4.0 is in the packages. According to the September 2026 announcement, DI (Dependency Injection), hosting, the health check and System.Text.Json serialization moved into the core Quartz package; the quartz.config file is no longer read, and the package targets only net10.0. For projects that need .NET Framework or netstandard, 3.x is in maintenance (currently 3.22.9).

Setup: AddQuartz and the PostgreSQL JobStore

By default Quartz keeps jobs in memory (RAMJobStore); they're all gone on restart. For durability and clustering you need the ActiveX Data Objects .NET (ADO.NET) JobStore:

var builder = Host.CreateApplicationBuilder(args);
string node = builder.Configuration["Node"] ?? "node-a";

builder.Services.AddQuartz(q =>
{
    q.ConfigureScheduler(options =>
    {
        options.InstanceName = "bookstore";   // the same on every node in the cluster
        options.InstanceId = node;            // different on every node
    });

    q.UsePersistentStore(store =>
    {
        store.UsePostgres(builder.Configuration.GetConnectionString("Quartz")!);
        store.UseClustering();
        if (builder.Environment.IsDevelopment())
            store.ProvisionSchema();
    });

    q.AddJob<SupplierPriceSyncJob>(job => job
        .WithIdentity("supplier-price-sync")
        .UsingJobData("lastSyncedPage", 0)
        .RequestRecovery());
    q.AddTrigger(t => t
        .ForJob("supplier-price-sync")
        .WithIdentity("every-10-seconds")
        .WithCronSchedule("0/10 * * ? * *"));
});

// don't let the code definition overwrite the stored job (and its JobDataMap) at every node startup
builder.Services.Configure<QuartzOptions>(o => o.Scheduling.IgnoreDuplicates = true);

builder.Services.AddQuartzHostedService(options => options.WaitForJobsToComplete = true);

var host = builder.Build();
host.Run();

A couple of details. Quartz loads the database driver by name; for UsePostgres you must add Npgsql to the project yourself, otherwise the error only shows up when the scheduler starts. The schema is set up by the database/tables/tables_postgres.sql script in the Quartz repository; the 4.7.2 script creates 15 tables (such as qrtz_job_details, qrtz_triggers, qrtz_cron_triggers, qrtz_fired_triggers, qrtz_scheduler_state, qrtz_locks and the new qrtz_execution_history). In Development, ProvisionSchema() creates missing tables itself; according to the documentation the default setting is Validate: if tables are missing, the scheduler refuses to start. In production, set up the schema with your own migration process.

WaitForJobsToComplete = true in AddQuartzHostedService makes the host wait for running jobs to finish when it shuts down. In the output below you'll see a page write "done" 4 seconds after SIGTERM (terminate signal) arrived; that wait is also inside the ShutdownTimeout budget from the second post of the series.

JobDataMap and [PersistJobDataAfterExecution]

[DisallowConcurrentExecution]      // the same JobKey runs once at a time across the cluster
[PersistJobDataAfterExecution]     // changes to the JobDataMap are written back to the database
public sealed class SupplierPriceSyncJob(ILogger<SupplierPriceSyncJob> logger) : IJob
{
    public async ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken)
    {
        int page = context.JobDetail.JobDataMap.GetInt("lastSyncedPage") + 1;
        logger.LogInformation("[{Node}] syncing page {Page} (recovering={Recovering})",
            context.Scheduler.SchedulerInstanceId, page, context.Recovering);

        await Task.Delay(TimeSpan.FromSeconds(6), cancellationToken); // the supplier API is slow

        context.JobDetail.JobDataMap["lastSyncedPage"] = page;
        logger.LogInformation("[{Node}] page {Page} done", context.Scheduler.SchedulerInstanceId, page);
    }
}

[DisallowConcurrentExecution] stops the same JobKey from running twice at the same time; according to the documentation it holds in a cluster too: the job runs on one node at a time and its other triggers wait. [PersistJobDataAfterExecution] writes the changes the job made to its JobDataMap back to the database when the run finishes; the next run reads the page it reached from there.

I fell into a real trap here. I ran the cluster for the first time and got to page 5, then stopped the nodes and started them again:

# first run, no IgnoreDuplicates
05:07:00.009 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 5 (recovering=False)
05:07:02.013 info: Microsoft.Hosting.Lifetime[0] Application is shutting down...
05:07:06.010 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] page 5 done
# nodes restarted
05:07:18.223 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
05:07:20.023 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-b] syncing page 1 (recovering=False)

Page 1, not page 6. The reason is that AddJob runs at every startup and UsingJobData("lastSyncedPage", 0) overwrites the record in the database. Quartz's XML (Extensible Markup Language) documentation says SchedulingOptions.OverwriteExistingData is "on by default": a job or trigger declared in code replaces the one stored under the same key. That makes sense for updating a job's code-defined properties on every deploy, but it's destructive for state that accumulates at runtime. The fix is the IgnoreDuplicates setting in Program.cs: if the record already exists, it's left alone. Sturdier still is writing work state such as progress to your own table instead of the JobDataMap; the checkpoint table in the ninth post is exactly that.

Clustering: Two Nodes Sharing One Database

There's no separate server to set up a cluster: nodes using the same database, the same InstanceName and the same table prefix, but different InstanceIds, are a cluster. Which node fires which trigger is decided by row locks in the database. I started two nodes (--Node node-a and --Node node-b):

05:06:19.558 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
05:06:20.024 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 1 (recovering=False)
05:06:22.330 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
05:06:26.027 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] page 1 done
05:06:30.007 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 2 (recovering=False)
05:06:36.009 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] page 2 done
05:06:40.006 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 3 (recovering=False)
05:06:46.008 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] page 3 done
05:06:50.005 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 4 (recovering=False)
05:06:56.007 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] page 4 done

node-a processed every page. That's not a bug: according to Quartz's clustering documentation, whichever node acquires a trigger runs it; there's no round-robin distribution. Load spreads statistically when there are many jobs and triggers. For a single job, the real benefit of clustering isn't sharing load but carrying on when a node dies.

Don't skip the documentation's two requirements: the nodes' clocks must be within a second of each other (with a time service such as NTP (Network Time Protocol)), and the InstanceId must differ on every node. In Kubernetes the pod name is a natural choice; GenerateInstanceId = true also generates one.

When a Node Dies: Recovery and Misfires

The opening question. With IgnoreDuplicates, on a clean database, I started two nodes and killed node-a with kill -9 while it was processing page 3:

05:09:00.056 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 1 (recovering=False)
05:09:06.057 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] page 1 done
05:09:10.009 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 2 (recovering=False)
05:09:16.011 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] page 2 done
05:09:20.010 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-a] syncing page 3 (recovering=False)
05:09:22.307 kill -9 node-a
05:09:37.897 warn: Quartz.Impl.AdoJobStore.LocalTransactionJobStore[3503] ClusterManager: detected 1 failed or restarted instances.
05:09:37.903 warn: Quartz.Impl.AdoJobStore.LocalTransactionJobStore[3511] ClusterManager: ......Deferred recovery of 1 executing [DisallowConcurrentExecution] job(s).
05:09:45.399 warn: Quartz.Impl.AdoJobStore.LocalTransactionJobStore[3503] ClusterManager: detected 1 failed or restarted instances.
05:09:45.430 warn: Quartz.Impl.AdoJobStore.LocalTransactionJobStore[3509] ClusterManager: ......Scheduled 1 recoverable job(s) for recovery.
05:09:45.468 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-b] syncing page 3 (recovering=True)
05:09:51.470 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-b] page 3 done
05:09:51.498 info: Bookstore.Scheduler.SupplierPriceSyncJob[0] [node-b] syncing page 4 (recovering=False)

Here's the timeline. node-a started page 3 at 09:20 and died at 09:22.3. node-b's ClusterManager noticed that node-a's check-ins had stopped at 09:37.9, about 15 seconds later: Quartz's defaults are a CheckinInterval of 7.5 seconds and a CheckinMisfireThreshold of 7.5 seconds. On its first attempt it deferred recovery ("Deferred recovery of 1 executing [DisallowConcurrentExecution] job(s)"), scheduled the job for recovery in the next check-in round, and at 09:45.5 page 3 ran again on node-b with recovering=True.

So page 3 was processed twice: once, half-way, on node-a, and once from the start on node-b. This isn't a Quartz shortcoming but the nature of distributed systems: you can only tell that a node died mid-work from its silence, without knowing how much of the work it did. RequestRecovery() means "at least once"; the job must tolerate redoing an interrupted run, i.e. be idempotent. According to the documentation, for a job without RequestRecovery that run is lost entirely; that's "at most once". We'll cover the choice between them, and idempotency, in detail in the next post.

Then there are the lost slots. The 09:30 and 09:40 triggers didn't fire, because the [DisallowConcurrentExecution] job still looked "running". When recovery finished (09:51.5), Quartz immediately ran one more round for the missed firing: page 4 started at 09:51.498, without waiting for a whole-second boundary. That's the cron trigger's default misfire behavior. According to Quartz's XML documentation, SmartPolicy is the default for cron triggers and means FireAndProceed: fire once right away for the missed firings, then resume the schedule. The other options are DoNothing (skip the missed firings and wait for the next slot) and IgnoreMisfires (run every missed firing right away, one after another):

// skip missed firings, wait for the next slot
ITrigger nightly = TriggerBuilder.Create()
    .WithIdentity("nightly-sales-report")
    .WithCronSchedule("0 0 3 * * ?", c => c
        .InTimeZone(istanbul)
        .WithMisfireInstruction(CronTriggerMisfireInstruction.DoNothing))
    .Build();
Console.WriteLine($"{nightly.Key.Name}: misfire instruction code {nightly.MisfireInstructionCode} ({(CronTriggerMisfireInstruction)nightly.MisfireInstructionCode})");

// nightly-sales-report: misfire instruction code 2 (DoNothing)

MisfireThreshold decides when a trigger counts as "missed". I printed the defaults:

var ado = new AdoJobStoreOptions();
var mem = new InMemoryJobStoreOptions();
var cl = new ClusteringOptions();
Console.WriteLine($"Ado.MisfireThreshold={ado.MisfireThreshold} InMemory.MisfireThreshold={mem.MisfireThreshold} Checkin={cl.CheckinInterval} CheckinMisfire={cl.CheckinMisfireThreshold}");

// Ado.MisfireThreshold=00:01:00 InMemory.MisfireThreshold=00:00:05 Checkin=00:00:07.5000000 CheckinMisfire=00:00:07.5000000

60 seconds in the ADO.NET store, 5 seconds in the in-memory store. According to the migration guide, in 4.0 only the threshold of an unconfigured in-memory store dropped from 60 seconds to 5. For a nightly report DoNothing is usually the right choice (if producing the nightly report at 09:00 in the morning helps nobody); for a stock sync, FireAndProceed (you want to get back to current data as soon as possible).

RFC 5545 Recurrence Triggers

Cron is powerful, but it can't express every rule. "Every other week on Tuesday and Thursday" can't be written in cron, because cron has no notion of "every other week". Quartz 4.0 brought RecurrenceTrigger, which accepts the recurrence rules of RFC (Request for Comments) 5545, the iCalendar standard that calendar apps use:

TimeZoneInfo istanbul = TimeZoneInfo.FindSystemTimeZoneById("Europe/Istanbul");
var from = new DateTimeOffset(2026, 11, 8, 0, 0, 0, TimeSpan.FromHours(3));

// the last business day of the month at 18:00: "LW" in cron, BYSETPOS=-1 in RFC 5545
ITrigger cron = TriggerBuilder.Create()
    .WithIdentity("month-end-cron")
    .StartAt(from)
    .WithCronSchedule("0 0 18 LW * ?", c => c.InTimeZone(istanbul))
    .Build();

ITrigger rrule = TriggerBuilder.Create()
    .WithIdentity("month-end-rrule")
    .StartAt(from)
    .WithRecurrenceSchedule("FREQ=MONTHLY;BYDAY=MO,TU,WE,TH,FR;BYSETPOS=-1;BYHOUR=18;BYMINUTE=0;BYSECOND=0",
        r => r.InTimeZone(istanbul))
    .Build();

// every other week on Tuesday and Thursday at 10:00: a rule cron can't express
ITrigger biweekly = TriggerBuilder.Create()
    .WithIdentity("biweekly-supplier-call")
    .StartAt(from)
    .WithRecurrenceSchedule("FREQ=WEEKLY;INTERVAL=2;BYDAY=TU,TH;BYHOUR=10;BYMINUTE=0;BYSECOND=0",
        r => r.InTimeZone(istanbul))
    .Build();

foreach (ITrigger t in new[] { cron, rrule, biweekly })
{
    var times = new List<string>();
    DateTimeOffset? next = t.GetFireTimeAfter(from.AddSeconds(-1));
    while (next is not null && times.Count < 6)
    {
        times.Add(TimeZoneInfo.ConvertTime(next.Value, istanbul).ToString("ddd yyyy-MM-dd HH:mm"));
        next = t.GetFireTimeAfter(next);
    }
    Console.WriteLine($"{t.Key.Name}:");
    Console.WriteLine("  " + string.Join(Environment.NewLine + "  ", times));
}

month-end-cron:
  Mon 2026-11-30 18:00
  Thu 2026-12-31 18:00
  Fri 2027-01-29 18:00
  Fri 2027-02-26 18:00
  Wed 2027-03-31 18:00
  Fri 2027-04-30 18:00
month-end-rrule:
  Mon 2026-11-30 18:00
  Thu 2026-12-31 18:00
  Fri 2027-01-29 18:00
  Fri 2027-02-26 18:00
  Wed 2027-03-31 18:00
  Fri 2027-04-30 18:00
biweekly-supplier-call:
  Tue 2026-11-17 10:00
  Thu 2026-11-19 10:00
  Tue 2026-12-01 10:00
  Thu 2026-12-03 10:00
  Tue 2026-12-15 10:00
  Thu 2026-12-17 10:00

"The last business day of the month" can be written both ways and the results are identical. The two-week rule, however, is only possible with RFC 5545. Since the week start (WKST) defaults to Monday, the first week of a rule starting on Sunday 8 November is 2–8 November; its Tuesday and Thursday fall before the start, so the first run is the Tuesday two weeks later, 17 November.

Compared with Hangfire

The rough rule is the same as in the previous post: if most of your work is "do this in the background and retry", Hangfire; if most of it is complex calendars and running once in a cluster, Quartz. Some teams use both in the same app; just remember that two job infrastructures mean two operational burdens.

In Production

Know your recovery time. With the default settings, a node's death is noticed in about 15 seconds, and during that time that node's [DisallowConcurrentExecution] jobs run nowhere. Irrelevant for a job that runs every minute; not for one that runs every second.

Watch the database. Every node in the cluster uses row locks in the database to acquire triggers; as the number of nodes and the trigger density grow, the qrtz_locks table can become a bottleneck. Quartz 4's OpenTelemetry metrics include job store operation durations and lock wait times; we'll look at them in the tenth post.

Run schema migrations before the deploy. 4.x's minor releases add columns too (4.6 brought three mandatory migrations, 4.7 one), and according to the migration guide a new-version node refuses to start with a missing column.

Common Pitfalls

Keeping progress in the JobDataMap and forgetting OverwriteExistingData. On every startup the code definition overwrites the accumulated state.

Putting RequestRecovery on a job that isn't idempotent. Recovery redoes the interrupted run from the start; if the job can't tolerate running twice, it produces duplicate records or duplicate emails.

Giving nodes the same InstanceId. One node takes the other's check-in record for its own; the dead node goes unnoticed.

Forgetting to add Npgsql or Microsoft.Data.SqlClient. Quartz loads the driver by name; a missing driver fails not at compile time but when the scheduler starts.

Leaving the misfire behavior at the default. For a nightly report, FireAndProceed produces the report during working hours when the app comes up in the morning.

A Small Experiment

★☆☆ Define the trigger with the DoNothing misfire instruction and repeat the failover experiment. When does page 4 start after recovery?

★★☆ Add a third node and define 10 different jobs (each with its own JobKey). Count how many jobs each node runs over one minute. How balanced is the distribution?

★★★ Write SupplierPriceSyncJob's progress to your own table instead of the JobDataMap, and put the page update in the same transaction as the work itself. In the failover experiment, is page 3 still processed twice? If so, is the result applied twice?

In the next post we go after the fact we saw twice in this one: background jobs run at least once, sometimes twice. Hangfire's AutomaticRetry, Quartz's retry policies, telling transient from permanent errors, and preventing the invoice email from going out twice with an idempotency key.