Jobs Across Multiple Instances: Distributed Locks, Leader Election and Double Execution
A BackgroundService in three replicas produces the nightly report three times; a lock taken through DistributedLock on SQL Server sp_getapplock, a PostgreSQL advisory lock or Redis brings that down to one. When the lock's owner dies by kill -9, on Redis nobody holds the lock until the 30-second lease ends; on PostgreSQL it stayed with the dead process until the server noticed the dropped connection, about a minute in my test. A lock isn't a guarantee, it's a lease.
Version note: The code in this post was built and run with .NET 10, DistributedLock.Postgres 1.3.1, DistributedLock.SqlServer 1.0.7, DistributedLock.Redis 1.1.1, Cronos 0.13.0 and Hangfire 1.8.25; the locks ran on PostgreSQL 17, SQL (Structured Query Language) Server 2025 and Redis 7.4.
Guess First
While Bookstore's API (Application Programming Interface) ran as a single pod, a BackgroundService produced the nightly sales report. For a campaign the API was scaled to 3 replicas. How many times did accounting receive the same report the next morning? You fix it with a distributed lock, and the pod holding the lock dies by kill -9 right then. When can the other two pods take the lock: immediately, a few seconds later, or much later? Does the answer depend on whether you keep the lock in Redis or in PostgreSQL?
Three Replicas, Three Reports
To shorten the experiment, instead of "every night at 03:00" I wrote a service that produces a report every 5 seconds, aligned to clock boundaries with Cronos:
public static class Schedule
{
// every 5 seconds in the demo instead of "every night at 03:00"
public static readonly CronExpression Report = CronExpression.Parse("*/5 * * * * *", CronFormat.IncludeSeconds);
public static async Task<DateTime> WaitForNextAsync(CancellationToken ct)
{
DateTime now = DateTime.UtcNow;
DateTime next = Report.GetNextOccurrence(now)!.Value;
await Task.Delay(next - now, ct);
return next;
}
}
public sealed class NaiveNightlyReportService(IConfiguration config, ILogger<NaiveNightlyReportService> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
DateTime slot = await Schedule.WaitForNextAsync(stoppingToken);
logger.LogInformation("[{Replica}] generating sales report for {Slot:HH:mm:ss}", config["Replica"], slot);
await Task.Delay(2000, stoppingToken);
}
}
}I started the same binary as three processes (--Replica api-1, api-2, api-3):
05:15:45.006 info: Bookstore.Reports.NaiveNightlyReportService[0] [api-1] generating sales report for 02:15:45
05:15:45.006 info: Bookstore.Reports.NaiveNightlyReportService[0] [api-2] generating sales report for 02:15:45
05:15:45.006 info: Bookstore.Reports.NaiveNightlyReportService[0] [api-3] generating sales report for 02:15:45
05:15:50.001 info: Bookstore.Reports.NaiveNightlyReportService[0] [api-1] generating sales report for 02:15:50
05:15:50.001 info: Bookstore.Reports.NaiveNightlyReportService[0] [api-2] generating sales report for 02:15:50
05:15:50.001 info: Bookstore.Reports.NaiveNightlyReportService[0] [api-3] generating sales report for 02:15:50Every slot three times. In a real nightly report that means three emails, three files or three rows in the database. The problem isn't in the code but in the assumption: a BackgroundService is per process, not per application. In Kubernetes, when the HPA (Horizontal Pod Autoscaler) raises the replica count automatically, this number changes during the day too.
DistributedLock: One API, Three Backends
The fix is to keep a "I'm doing this work right now" record somewhere all replicas can see. Instead of writing that by hand for each database, I used madelson's DistributedLock library; behind the same IDistributedLockProvider interface there are three different mechanisms:
string provider = builder.Configuration["LockProvider"] ?? "postgres";
builder.Services.AddSingleton<IDistributedLockProvider>(_ => provider switch
{
"postgres" => new PostgresDistributedSynchronizationProvider(builder.Configuration.GetConnectionString("Postgres")!),
"sqlserver" => new SqlDistributedSynchronizationProvider(builder.Configuration.GetConnectionString("SqlServer")!),
"redis" => new RedisDistributedSynchronizationProvider(
ConnectionMultiplexer.Connect(builder.Configuration.GetConnectionString("Redis")!).GetDatabase()),
_ => throw new InvalidOperationException($"Unknown lock provider {provider}")
});According to the library's documentation, the SQL Server provider uses SQL Server application locks (sp_getapplock). The PostgreSQL provider uses advisory locks; I looked at pg_locks while the lock was held:
select locktype, classid, objid, granted, pid from pg_locks where locktype='advisory';
locktype | classid | objid | granted | pid
----------+-----------+------------+---------+-----
advisory | 209205062 | 1137612306 | t | 153The Redis provider implements the RedLock algorithm documented by Redis: it writes a key with an expiry and extends it in the background. The key itself while the lock is held:
$ redis-cli --scan
bookstore:scheduler-leader
$ redis-cli pttl bookstore:scheduler-leader
22037
$ redis-cli get bookstore:scheduler-leader
Mac_26269_6c6b21936f6a4008b024b6140463cd07A warning: these packages bring their own driver versions transitively. In my project, Npgsql 8.0.6, Microsoft.Data.SqlClient 6.1.4 and StackExchange.Redis 2.7.33 came in; the System.Security.Cryptography.Pkcs brought by SqlClient again raised an NU1903 warning, and I updated it with an explicit reference as in the fifth post. If the rest of the app uses a newer driver, NuGet picks the highest version; check the versions with dotnet list package --include-transitive.
A Single Run with a Lock
public sealed class LockedNightlyReportService(
IDistributedLockProvider locks,
IConfiguration config,
ILogger<LockedNightlyReportService> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
DateTime slot = await Schedule.WaitForNextAsync(stoppingToken);
// timeout 0: if someone else holds the lock, give up without waiting
await using IDistributedSynchronizationHandle? handle =
await locks.TryAcquireLockAsync("bookstore:nightly-sales-report", TimeSpan.Zero, stoppingToken);
if (handle is null)
{
logger.LogInformation("[{Replica}] lock held elsewhere, skipping {Slot:HH:mm:ss}", config["Replica"], slot);
continue;
}
// if the lock is lost (connection dropped, lease expired), cancel the work too
using var linked = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken, handle.HandleLostToken);
logger.LogInformation("[{Replica}] generating sales report for {Slot:HH:mm:ss}", config["Replica"], slot);
await Task.Delay(2000, linked.Token);
}
}
}Passing TimeSpan.Zero to TryAcquireLockAsync means "don't wait; if someone else holds the lock, give up". I got the same result with all three backends:
# LockProvider=postgres
05:15:55.119 info: Bookstore.Reports.LockedNightlyReportService[0] [api-2] generating sales report for 02:15:55
05:15:55.119 info: Bookstore.Reports.LockedNightlyReportService[0] [api-3] lock held elsewhere, skipping 02:15:55
05:15:55.120 info: Bookstore.Reports.LockedNightlyReportService[0] [api-1] lock held elsewhere, skipping 02:15:55
# LockProvider=sqlserver
05:16:15.003 info: Bookstore.Reports.LockedNightlyReportService[0] [api-1] lock held elsewhere, skipping 02:16:15
05:16:15.003 info: Bookstore.Reports.LockedNightlyReportService[0] [api-2] generating sales report for 02:16:15
05:16:15.005 info: Bookstore.Reports.LockedNightlyReportService[0] [api-3] lock held elsewhere, skipping 02:16:15
# LockProvider=redis
05:16:20.015 info: Bookstore.Reports.LockedNightlyReportService[0] [api-1] generating sales report for 02:16:20
05:16:20.015 info: Bookstore.Reports.LockedNightlyReportService[0] [api-2] lock held elsewhere, skipping 02:16:20
05:16:20.019 info: Bookstore.Reports.LockedNightlyReportService[0] [api-3] lock held elsewhere, skipping 02:16:20One report per slot. But careful here: the lock guarantees "one at a time", not "one per slot". Because the report took 2 seconds, the other replicas always found the lock taken. If the report took 50 ms and the replicas' clocks were 100 ms apart, a late-waking replica could find the lock free and produce the report a second time for the same slot. Get one run per slot with a record inside the lock: with the "claim first" approach from the previous post, write a key such as sales-report:2026-11-08 to a unique table; the lock prevents concurrency, the record prevents repetition.
Leader Election: Holding the Lock for the Process's Lifetime
Instead of taking a separate lock for every job, you can elect one replica as leader and have it run all the scheduled work. The leader holds the lock for as long as the process lives; the others wait in AcquireLockAsync:
public sealed class LeaderElectedScheduler(
IDistributedLockProvider locks,
IConfiguration config,
ILogger<LeaderElectedScheduler> logger) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
// wait until we become leader; the lock is held as long as the process lives
await using IDistributedSynchronizationHandle handle =
await locks.AcquireLockAsync("bookstore:scheduler-leader", cancellationToken: stoppingToken);
logger.LogInformation("[{Replica}] became leader", config["Replica"]);
using var leadership = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken, handle.HandleLostToken);
try
{
while (!leadership.IsCancellationRequested)
{
DateTime slot = await Schedule.WaitForNextAsync(leadership.Token);
logger.LogInformation("[{Replica}] generating sales report for {Slot:HH:mm:ss}", config["Replica"], slot);
}
}
catch (OperationCanceledException) when (!stoppingToken.IsCancellationRequested)
{
logger.LogWarning("[{Replica}] lost leadership", config["Replica"]);
}
}
}
}After starting three replicas, PostgreSQL shows one lock granted and two waiting:
locktype | classid | objid | granted | pid
----------+-----------+------------+---------+-----
advisory | 209205062 | 1137612306 | f | 173
advisory | 209205062 | 1137612306 | t | 171
advisory | 209205062 | 1137612306 | f | 172The beauty of this approach is its simplicity: scheduled work runs in one place, on one replica, and no per-job coordination is needed. Its cost is just as simple: until the leader's death is noticed, no scheduled work runs. How long that takes is the topic of the next section.
When the Lock's Owner Dies: How Long Do You Wait?
I killed the leader with kill -9; no graceful shutdown, the lock isn't released. Redis first:
05:21:16.407 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-2] became leader
05:21:20.004 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-2] generating sales report for 02:21:20
05:21:24 kill -9 api-2
05:21:46.599 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-1] became leader
05:21:50.004 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-1] generating sales report for 02:21:50
api-2 became leader at 16.4, died at 24, and api-1 could only become leader at 46.6: exactly 30 seconds after the lock was taken. The defaults in the source code of DistributedLock's Redis provider explain this: the lock is written with a 30-second expiry, and while the leader lives it is extended at regular intervals (every 9 seconds by default). Because the leader died before the first extension, the lock dropped 30 seconds after it was first written. 5 report slots were lost in between. A shorter expiry narrows that gap, but at a price: if the leader delays an extension because of a GC (garbage collection) pause or a network blip, it loses the lock.
Then PostgreSQL. My expectation was that since an advisory lock is tied to the session, it would be released as soon as the process died:
05:18:56.084 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-1] became leader
05:19:00.003 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-1] generating sales report for 02:19:00
05:19:03 kill -9 api-1I waited 12 seconds and nobody became leader. In the database, the dead process's connection was still active:
select now()::time(0), pid, state, query from pg_stat_activity where datname='bookstore08' and pid<>pg_backend_pid();
02:19:31|171|active|SELECT pg_catalog.pg_sleep($1)
02:19:41|171|active|SELECT pg_catalog.pg_sleep($1)
02:19:51|171|active|SELECT pg_catalog.pg_sleep($1)The dead leader's connection is running pg_sleep. The library's source code makes the reason clear: when HandleLostToken is accessed, it runs a 1-minute pg_sleep query to monitor the connection. While a query runs, PostgreSQL by default doesn't check whether the client is still there; even if the process dies, the backend lives until the query finishes and keeps the advisory lock. With the connection being monitored, the lock was held for about a minute after its owner died: it was released somewhere between 02:19:51 and 02:20:01.
The client_connection_check_interval setting, which came with PostgreSQL 14, exists for exactly this case: it checks the client connection at intervals while a query runs, and its default is 0, i.e. off. I set it to 2 seconds in my own container and repeated the experiment:
ALTER SYSTEM SET client_connection_check_interval = '2s';
SELECT pg_reload_conf();
05:20:45.765 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-2] became leader
05:20:50.005 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-2] generating sales report for 02:20:50
05:20:53 kill -9 api-2
05:20:53.750 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-3] became leader
05:20:55.003 info: Bookstore.Reports.LeaderElectedScheduler[0] [api-3] generating sales report for 02:20:55api-3 became leader in the same second the leader died (20:53). This result comes from a setting, not a library; same code, same driver. The lesson is general: a distributed lock's promise of "released when the owner dies" depends on how fast the infrastructure notices the owner died, and you can't know that time without measuring it. SQL Server's sp_getapplock is tied to the session too; how fast the server notices a dropped connection depends on the TCP (Transmission Control Protocol) layer there as well. I didn't measure the kill -9 scenario on SQL Server for this post.
Leases, Renewal and Fencing Tokens
In the Redis example the lock is really a lease: ownership with a time limit. A lease is dangerous the other way too. The leader is alive but goes into a 40-second GC pause or drops off the network; the lease ends and another replica becomes leader. If the old leader wakes up still believing it's the leader, two leaders do the work at the same time.
The library's HandleLostToken is the first defense: it's cancelled when the lock is found to be lost. In the code we combine this token with stoppingToken and pass it to the work; when the lock goes, the work stops. But the word "found" matters: a process in a GC pause finds nothing out until it wakes up and runs the next line.
The definitive fix is for the protected resource itself to reject the old leader: every lock acquisition produces an increasing number (a fencing token), and the work attaches it to everything it writes; the resource rejects a write carrying a token smaller than the largest it has seen. That's not something a lock library can provide on its own; the resource being written has to check the token. The Distributed Systems Fundamentals series post "Distributed Locks, Leases and Fencing Tokens: Why a Lock Is Not Enough" covers the theory. In practice, a cheaper route is enough for most background jobs: the idempotency from the seventh post makes it harmless for two leaders to do the same work.
Hangfire's and Quartz's Own Solutions
If you use Hangfire or Quartz, part of this problem is already solved. We saw Quartz's clustering in the sixth post: a single node takes a trigger through a database lock. In Hangfire, a recurring job is a single record in storage; I connected three separate Hangfire servers to the same database and defined a recurring job that runs every minute:
app.Services.GetRequiredService<IRecurringJobManager>()
.AddOrUpdate<MinuteReportJob>("minute-report", j => j.RunAsync(), Cron.Minutely());
// three processes, the same SQL Server storage, 150 seconds
05:59:13.902 info: Bookstore.Jobs.MinuteReportJob[0] Minute report generated by pid 38931
06:00:13.999 info: Bookstore.Jobs.MinuteReportJob[0] Minute report generated by pid 38930Despite three servers, one job per minute; two different processes ran the two minutes. To stop two runs of the same method from running at the same time, Hangfire also has a DisableConcurrentExecution filter; I queued the job that rebuilds the search index twice:
public sealed class SearchIndexJobs(ILogger<SearchIndexJobs> logger)
{
// don't let two runs of the same method overlap; wait at most 10 s for the lock
[DisableConcurrentExecution(timeoutInSeconds: 10)]
public async Task RebuildAsync(string reason)
{
logger.LogInformation("Rebuilding book search index ({Reason})", reason);
await Task.Delay(3000);
logger.LogInformation("Search index rebuilt ({Reason})", reason);
}
}
jobs.Enqueue<SearchIndexJobs>(j => j.RebuildAsync("price import"));
jobs.Enqueue<SearchIndexJobs>(j => j.RebuildAsync("new books"));05:22:22.304 info: Bookstore.Jobs.SearchIndexJobs[0] Rebuilding book search index (price import)
05:22:25.306 info: Bookstore.Jobs.SearchIndexJobs[0] Search index rebuilt (price import)
05:22:25.316 info: Bookstore.Jobs.SearchIndexJobs[0] Rebuilding book search index (new books)
05:22:28.317 info: Bookstore.Jobs.SearchIndexJobs[0] Search index rebuilt (new books)A server with 5 workers could have taken both jobs at once, but the second waited for the first to finish. The filter takes a distributed lock on the storage; if it can't get it within timeoutInSeconds, the job fails and goes into retries. Note: this means "one at a time", not "once"; the second job still ran.
In Production
Measure the leader-loss time. Don't trust the assumption that "the work continues when the leader dies" without killing the leader's process in your own infrastructure and measuring how long a new leader takes. In the two experiments above that time ranged from 1 second to 60 seconds.
Treat the lock as part of the database connection pool. PostgreSQL and SQL Server locks keep a connection busy for as long as the lock is held. For leader election that's one connection; in a busy system taking a separate lock per job, it can exhaust the pool.
Think about scope when naming a lock. If bookstore:nightly-sales-report is the same in every environment and staging and production share the same Redis, a staging pod blocks production's report. Put the environment name in the key.
Common Pitfalls
Taking the lock without using/await using. Work that throws doesn't release the lock; everyone waits until the lease ends on Redis, or until the connection closes in the database.
Keeping the lock timeout shorter than the work. If the Redis expiry is shorter than the work and renewal falters for some reason, the lock passes to someone else while the work is still running. DistributedLock renews automatically; if you write your own lock, don't forget renewal.
Taking the lock for a "runs once" guarantee. A lock prevents concurrency; it doesn't prevent two consecutive runs for the same slot. Get one-per-slot with an idempotency record.
Ignoring HandleLostToken. Code that doesn't stop the work when the lock is lost widens the window in which two leaders run at once.
Assuming the death of the lock's owner is noticed instantly. In the PostgreSQL experiment, a dead process held the lock for about a minute.
A Small Experiment
★☆☆ Lower the Task.Delay(2000) in LockedNightlyReportService to 20 ms and add an artificial delay to one replica's clock (wait 100 ms after WaitForNextAsync). How many reports are produced for the same slot?
★★☆ Set up the Redis provider with a 5-second expiry (the options parameter of RedisDistributedSynchronizationProvider) and repeat the leader election experiment. How many slots are lost? Instead of killing the leader, pause it for 8 seconds (kill -STOP, then kill -CONT); do two leaders produce reports at the same time?
★★★ Add a fencing token column to the report table: the replica that becomes leader takes a token from a PostgreSQL sequence and checks WHERE token >= (SELECT MAX(token) ...) on every report write. In the kill -STOP scenario, is the old leader's write rejected?
In the next post we move to long-running work: splitting a 2-million-row price update into chunks instead of one transaction, a checkpoint table that remembers where it stopped, being cut off during a deploy and resuming where it left off, and how Hangfire keeps long jobs "invisible".