Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
Version Migrations for Background Jobs: BackgroundService Changes from .NET 6 to 10, .NET 11 and Quartz 3→4
.NET 6 started stopping the host on a BackgroundService failure, ShutdownTimeout went from 5 to 30 seconds in .NET 7, .NET 8 brought lifecycle hooks, and .NET 10 moved all of ExecuteAsync to the background: an exception before the first await crashes the process with code 134 on .NET 8 but is logged and exits with 0 on .NET 10. In Quartz 3→4 a simple job gave a single CS0535 error; the real work is packages, the schema migration and changed defaults.
Job Dashboards, Monitoring and Alerting: Hangfire Dashboard, Quartz 4 Dashboard and OpenTelemetry
The Hangfire Dashboard is open only to local requests by default, but a request coming through a proxy on the same machine counts as local too: a request through Docker got a 200. In production, put the dashboard behind an authorization policy and read-only mode; the Quartz 4 dashboard won't start at all without authorization. A "last successful run older than 26 hours" health check, Quartz 4's OpenTelemetry spans and metrics, and structured logs carrying the JobId complete the picture.
Long-Running and Batch Jobs: Chunking, Checkpoints and Surviving Deploy Interruptions
Doing a 2-million-row price update in a single UPDATE made a request updating the same row wait 1.9 seconds; with 20,000-row chunks it waited 4 ms. A job that commits each chunk in the same transaction as its checkpoint row stopped at row 800,000 during a deploy and carried on from there after restarting. Hangfire protects long jobs with a sliding invisibility timeout; if the process dies, the job moves to another worker once that window passes.
Jobs Across Multiple Instances: Distributed Locks, Leader Election and Double Execution
A BackgroundService in three replicas produces the nightly report three times; a lock taken through DistributedLock on SQL Server sp_getapplock, a PostgreSQL advisory lock or Redis brings that down to one. When the lock's owner dies by kill -9, on Redis nobody holds the lock until the 30-second lease ends; on PostgreSQL it stayed with the dead process until the server noticed the dropped connection, about a minute in my test. A lock isn't a guarantee, it's a lease.
Retry and Idempotency in Background Jobs: Living with At-Least-Once
A background job runs at least once and sometimes twice; retries make that even more likely. Hangfire retries 10 times at growing intervals by default and ExceptOn takes permanent errors out of retries; in Quartz 4 a retry policy is declared explicitly. "Check, then send" produces two emails from two concurrent runs; claiming first with a unique key brings it down to one, and an Idempotency-Key header does the same end to end with an external API.
Scheduling with Quartz.NET 4: Jobs, Triggers, Persistent JobStore and Clustering
In Quartz.NET 4, jobs and triggers are separate objects, IJob returns ValueTask and takes a CancellationToken, and DI and hosting live in the core Quartz package. Two nodes sharing one PostgreSQL database form a cluster: when the executing node dies by kill -9, the other notices about 15 seconds later and reruns the RequestRecovery job with recovering=true. At startup the code definition overwrites the JobDataMap stored in the database; IgnoreDuplicates prevents that.
Persistent Jobs with Hangfire: Fire-and-Forget, Delayed, Recurring and Continuations
Hangfire writes a method call to the database as a type name, a method name, parameter types and arguments serialized to JSON; it survives restarts and is retried 10 times by default. That's why only ids should be passed as arguments: a DbContext can't be serialized, and an entity is stored as a stale copy. Renaming a method sends old jobs still in the queue into retries with a method-not-found error.
Cron Expressions: Unix 5-Field, 6-Field with Seconds, Quartz Format and Time Zone/DST Pitfalls
Unix cron has 5 fields, Quartz wants 6–7 fields starting with seconds and counts weekdays as 1=Sunday; Quartz 4 rejects "0 0 * * *" with the default parser and accepts it with CronFormat.Unix. Across DST, a daily 02:30 job runs once at 03:00 in spring and only once in autumn, while interval jobs run twice in the repeated hour. You can write your own cron scheduler on TimeProvider and test it with FakeTimeProvider.
Worker Service and an In-Process Job Queue: Channel, PeriodicTimer and Windows Service/systemd
An in-process job queue built on a bounded Channel returns false from TryWrite when it is full; the endpoint turns that into a 503 with a Retry-After header and applies backpressure. For periodic work, PeriodicTimer doesn't pile up missed ticks: after a long run it fires at most once right away. AddWindowsService and AddSystemd only activate in their own environment, and the queue empties on restart.
IHostedService and BackgroundService in .NET: Lifecycle, Graceful Shutdown and the .NET 10 Behavior Change
In .NET 10 the whole of BackgroundService.ExecuteAsync runs in the background; code before the first await no longer blocks startup, and code that must run early moves to StartAsync or IHostedLifecycleService. On shutdown the host cancels stoppingToken and gives all stopping work a default 30-second ShutdownTimeout; work that ignores the token is cut off at the end of that window without even a log line.
Running Work Outside the Request: Choosing Between Fire-and-Forget, BackgroundService, Hangfire and Quartz.NET
Work started with Task.Run inside a controller loses its DbContext when the request scope ends, swallows its exception silently and dies halfway through a deploy; it has to be handed to something the host manages. Five questions drive the choice: can the work be lost on restart, is it scheduled, do you need retries and a dashboard, how many instances run; the answers point to BackgroundService, Hangfire, Quartz.NET or a platform scheduler.
Asynchronous Processing: Queues, Workers, Fan-Out and Backpressure
Heavy work that is not part of the response moves out of the request to a queue and workers; the API returns 202 Accepted and a status address. In a real experiment the synchronous invoice endpoint times out 52 of 100 concurrent requests yet still renders 100 PDFs, while the bounded-queue version answers in 9 ms and pushes back with 503 and Retry-After when full; asynchrony decouples the work's timing, it doesn't speed it up.