Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
Job Dashboards, Monitoring and Alerting: Hangfire Dashboard, Quartz 4 Dashboard and OpenTelemetry
The Hangfire Dashboard is open only to local requests by default, but a request coming through a proxy on the same machine counts as local too: a request through Docker got a 200. In production, put the dashboard behind an authorization policy and read-only mode; the Quartz 4 dashboard won't start at all without authorization. A "last successful run older than 26 hours" health check, Quartz 4's OpenTelemetry spans and metrics, and structured logs carrying the JobId complete the picture.
Retry and Idempotency in Background Jobs: Living with At-Least-Once
A background job runs at least once and sometimes twice; retries make that even more likely. Hangfire retries 10 times at growing intervals by default and ExceptOn takes permanent errors out of retries; in Quartz 4 a retry policy is declared explicitly. "Check, then send" produces two emails from two concurrent runs; claiming first with a unique key brings it down to one, and an Idempotency-Key header does the same end to end with an external API.
Persistent Jobs with Hangfire: Fire-and-Forget, Delayed, Recurring and Continuations
Hangfire writes a method call to the database as a type name, a method name, parameter types and arguments serialized to JSON; it survives restarts and is retried 10 times by default. That's why only ids should be passed as arguments: a DbContext can't be serialized, and an entity is stored as a stale copy. Renaming a method sends old jobs still in the queue into retries with a method-not-found error.
Running Work Outside the Request: Choosing Between Fire-and-Forget, BackgroundService, Hangfire and Quartz.NET
Work started with Task.Run inside a controller loses its DbContext when the request scope ends, swallows its exception silently and dies halfway through a deploy; it has to be handed to something the host manages. Five questions drive the choice: can the work be lost on restart, is it scheduled, do you need retries and a dashboard, how many instances run; the answers point to BackgroundService, Hangfire, Quartz.NET or a platform scheduler.