Job Dashboards, Monitoring and Alerting: Hangfire Dashboard, Quartz 4 Dashboard and OpenTelemetry

Version note: The code in this post was built and run with .NET 10, Hangfire 1.8.25, Quartz.NET 4.7.2 (including Quartz.Dashboard and Quartz.AspNetCore) and OpenTelemetry 1.20.1.

Guess First

The Hangfire Dashboard was added to Bookstore's API (Application Programming Interface): app.UseHangfireDashboard(), no other settings. The documentation says "open only to local requests by default", so the team feels safe. The app runs behind a reverse proxy on the same machine. If a request from the internet reaches /hangfire, does it get 401 or 200? And: what happens if you map Quartz 4's dashboard without configuring authorization?

The Hangfire Dashboard: Local Only by Default

builder.Services.AddHangfire(c => c
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSqlServerStorage(builder.Configuration.GetConnectionString("Hangfire")));
builder.Services.AddHangfireServer();

var app = builder.Build();
app.UseHangfireDashboard(); // default: /hangfire, local requests only
app.MapGet("/whoami", (HttpContext c) => $"remote={c.Connection.RemoteIpAddress} local={c.Connection.LocalIpAddress}");

According to Hangfire's documentation, the dashboard allows only local requests by default. "Local" means the request's remote IP (Internet Protocol) address is loopback or the same as the server's own address. I sent requests from my own machine and from a Docker container:

$ curl -s -o /dev/null -w "from 127.0.0.1: %{http_code}\n" http://127.0.0.1:17110/hangfire
from 127.0.0.1: 200

$ docker run --rm busybox:latest sh -c "wget -S -O /dev/null http://host.docker.internal:17110/hangfire 2>&1 | head -3"
Connecting to host.docker.internal:17110 (192.168.65.254:17110)
  HTTP/1.1 200 OK

$ docker run --rm busybox:latest sh -c "wget -q -O - http://host.docker.internal:17110/whoami"
remote=127.0.0.1 local=127.0.0.1

The container was on another network, yet its request got a 200. /whoami shows why: Docker's port forwarding opens the connection on the host from 127.0.0.1, and the app sees the request as local. The same goes for an nginx on the same machine, a sidecar or an SSH (Secure Shell) tunnel: the end of the connection reaching the app is local, while the request's real source is anywhere on the internet. "Local only" isn't a security boundary, just a sensible default for development. Don't rely on it in production.

Authorization in Production: Policies and Read-Only

Hangfire 1.8 offers an endpoint mapping that ties the dashboard to ASP.NET Core authorization policies. I put the dashboard at /ops/jobs, opened it to the Ops and Support roles, and gave change rights only to Ops:

builder.Services.AddAuthentication().AddBearerToken();
builder.Services.AddAuthorizationBuilder()
    .AddPolicy("JobsViewer", p => p.RequireRole("Ops", "Support"));

var app = builder.Build();
app.UseAuthentication();
app.UseAuthorization();

app.MapHangfireDashboardWithAuthorizationPolicy("JobsViewer", "/ops/jobs", new DashboardOptions
{
    // the Support role can look but can't retry/delete/trigger
    IsReadOnlyFunc = ctx => !ctx.GetHttpContext().User.IsInRole("Ops")
});

For the demo, an endpoint inside the same app issues tokens; in a real app, authentication comes from the company's identity provider. Four different requests:

no token:        GET  /ops/jobs -> 401
Customer token:  GET  /ops/jobs -> 403
Support token:   GET  /ops/jobs -> 200
Support token:   POST /ops/jobs/jobs/succeeded/requeue -> 401
Ops token:       POST /ops/jobs/jobs/succeeded/requeue -> 204

The request without a token got 401, the wrong role 403. Support can see the dashboard, but got 401 when trying to requeue a job; the same request returned 204 for Ops. Read-only mode looks like a small detail but it matters: pressing "Delete" or "Requeue" on the dashboard is an operation on production data. Not everyone who looks at the dashboard to debug needs that right.

Instead of a policy you can also write your own IDashboardAuthorizationFilter; you reach the HttpContext with DashboardContext.GetHttpContext() and check the user. The advantage of the endpoint mapping is that it uses the same authentication and policy infrastructure as the rest of your app. The Web Security (OWASP Top 10) series post that also covers IDOR (Insecure Direct Object Reference) flaws, "OWASP A01: Broken Access Control (1) — IDOR, Admin Endpoints and Authorization in ASP.NET Core", covers the general rules for protecting admin endpoints; a job dashboard is exactly an admin endpoint. (OWASP: Open Worldwide Application Security Project.)

The Quartz 4 Dashboard and HTTP API: No Start Without Authorization

Quartz 4.0 brought a Blazor dashboard and an HTTP (Hypertext Transfer Protocol) API. I tried to map them without configuring authorization:

builder.Services.AddQuartz();
builder.Services.AddQuartzDashboard();
var app = builder.Build();
app.MapQuartzDashboard();
app.Run();

fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.InvalidOperationException: The Quartz dashboard is mapped with no authorization: 16 endpoints answer anonymously, and they can schedule, pause and shut down every scheduler in this process - including a job whose type is named by a string the request carries. Quartz refuses to start rather than serve that by accident. Say which you meant:
        - app.MapQuartzDashboard().RequireAuthorization() authorizes its pages and its live-events hub;
        - services.AddQuartzDashboard(options => options.AuthorizationPolicy = "...") authorizes those and the static assets and the Blazor circuit with them;
        - app.MapQuartzDashboard().AllowAnonymous() serves it to anyone, deliberately.

The app didn't start at all. The error message gives the reason too: the dashboard and the HTTP API can stop schedulers and schedule a job by name, i.e. by a string the request carries. Quartz's README notes that, if the Quartz.Jobs package is in the app, this combination can reach NativeJob, which can run a process. Unlike Hangfire's "allow if local" default, Quartz 4 says "I won't start unless you tell me" and offers three explicit options: RequireAuthorization(), an authorization policy, or a deliberate AllowAnonymous(). That's good design; a required decision instead of an unsafe default.

A Health Check: The Dead Man's Switch

Dashboards show failed jobs. But the most dangerous job is one that never fails and never runs: its recurring definition was deleted, its cron expression is in the wrong time zone, the cluster can't elect a leader, the worker's queue list doesn't include its queue. To catch that, you have the job say "I ran" and treat the absence of that signal as an alarm; that's what's called a dead man's switch. Each job leaves a timestamp when it succeeds, and a health check checks how old those timestamps are:

// each job leaves a timestamp when it succeeds (kept in a table in production)
public sealed class JobHeartbeatStore(TimeProvider time)
{
    private readonly ConcurrentDictionary<string, DateTimeOffset> _lastSuccess = new();
    public void RecordSuccess(string job) => _lastSuccess[job] = time.GetUtcNow();
    public void Seed(string job, DateTimeOffset at) => _lastSuccess[job] = at;
    public DateTimeOffset? LastSuccess(string job) => _lastSuccess.TryGetValue(job, out var at) ? at : null;
}

// "dead man's switch": alarm if the job hasn't run, even without errors
public sealed class JobFreshnessHealthCheck(JobHeartbeatStore store, TimeProvider time, string job, TimeSpan maxAge)
    : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
    {
        DateTimeOffset? last = store.LastSuccess(job);
        if (last is null)
            return Task.FromResult(HealthCheckResult.Unhealthy($"{job} has never succeeded"));

        TimeSpan age = time.GetUtcNow() - last.Value;
        var data = new Dictionary<string, object> { ["lastSuccessUtc"] = last.Value.ToString("O"), ["ageHours"] = Math.Round(age.TotalHours, 1) };
        return Task.FromResult(age <= maxAge
            ? HealthCheckResult.Healthy($"{job} succeeded {age.TotalMinutes:F0} min ago", data)
            : HealthCheckResult.Unhealthy($"{job} last succeeded {age.TotalHours:F1} h ago (limit {maxAge.TotalHours} h)", data: data));
    }
}

[DisallowConcurrentExecution]
public sealed class StockSyncJob(JobHeartbeatStore heartbeats, ILogger<StockSyncJob> logger) : IJob
{
    public async ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken)
    {
        await Task.Delay(120, cancellationToken);
        logger.LogInformation("Stock synced for 312 books");
        heartbeats.RecordSuccess("stock-sync");
    }
}

The registration uses the same infrastructure as the ASP.NET Core Web API series post "Health Checks in ASP.NET Core Web API: Liveness, Readiness and Database Checks"; here we use a separate tag and endpoint for jobs:

builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddSingleton<JobHeartbeatStore>();
builder.Services.AddHealthChecks()
    .Add(new("stock-sync", sp => new JobFreshnessHealthCheck(
        sp.GetRequiredService<JobHeartbeatStore>(), TimeProvider.System, "stock-sync", TimeSpan.FromMinutes(15)),
        failureStatus: null, tags: ["jobs"]))
    .Add(new("nightly-sales-report", sp => new JobFreshnessHealthCheck(
        sp.GetRequiredService<JobHeartbeatStore>(), TimeProvider.System, "nightly-sales-report", TimeSpan.FromHours(26)),
        failureStatus: null, tags: ["jobs"]));

// ...

app.MapHealthChecks("/health/jobs", new HealthCheckOptions
{
    Predicate = r => r.Tags.Contains("jobs"),
    ResponseWriter = (ctx, report) => ctx.Response.WriteAsJsonAsync(new
    {
        status = report.Status.ToString(),
        checks = report.Entries.Select(e => new { name = e.Key, status = e.Value.Status.ToString(), e.Value.Description })
    })
});

The stock sync runs every 5 seconds and refreshes its timestamp; for the demo I set the nightly report to 27 hours ago:

$ curl -s -i http://127.0.0.1:17110/health/jobs
HTTP/1.1 503 Service Unavailable
{"status":"Unhealthy","checks":[{"name":"stock-sync","status":"Healthy","description":"stock-sync succeeded 0 min ago"},{"name":"nightly-sales-report","status":"Unhealthy","description":"nightly-sales-report last succeeded 27.0 h ago (limit 26 h)"}]}

A 503, plus which job raised the alarm and why. 26 hours means 2 hours of tolerance for a 24-hour job; pick the limit as the job's period plus a reasonable delay. Don't wire this endpoint to the Kubernetes liveness probe: restarting the pod because the report didn't run doesn't fix the problem, it hides it. Instead, call it periodically from a monitoring system and raise an alert. In production keep the timestamps in a table, not in memory; multiple replicas and restarts need to see the same timestamp.

OpenTelemetry: Quartz 4's Spans and Metrics

Quartz 4 publishes its own ActivitySource and Meter for OpenTelemetry; their names are constants in the QuartzInstrumentation class:

builder.Services.AddQuartz(q => q.ScheduleJob<StockSyncJob>(t => t
    .WithIdentity("stock-sync")
    .WithCronSchedule("0/5 * * ? * *")));
builder.Services.AddQuartzHostedService();
builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddSource(QuartzInstrumentation.ActivitySourceName).AddConsoleExporter())
    .WithMetrics(m => m.AddMeter(QuartzInstrumentation.MeterName).AddConsoleExporter((_, reader) =>
        reader.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 8000));

The span of one job run in the console exporter's output:

Activity.TraceId:            2bc730fb03f528b1174319c2b1568ed7
Activity.SpanId:             3991e390f6a583d3
Activity.TraceFlags:         Recorded
Activity.DisplayName:        Quartz.Job.Execute
Activity.Kind:               Internal
Activity.StartTime:          2026-11-08T02:30:05.0079340Z
Activity.Duration:           00:00:00.1230800
Activity.Tags:
    quartz.scheduler.name: QuartzScheduler
    quartz.scheduler.id: NON_CLUSTERED
    quartz.job.type: Bookstore.Api.StockSyncJob, p10
    quartz.fire.instance.id: 734269957552459
    quartz.trigger.group: DEFAULT
    quartz.trigger.name: stock-sync
    quartz.job.group: DEFAULT
    quartz.job.name: stock-sync
    quartz.job.result: succeeded
Instrumentation scope (ActivitySource):
    Name: Quartz
    Version: 4.7.2.0

The span carries the job's name, trigger, type and result (quartz.job.result: succeeded). Job store operations (Quartz.JobStore.AcquireNextTriggers, Quartz.JobStore.TriggeredJobComplete) come as separate spans too; with a database-backed store, that's where you see which step is slow. The metrics published in the same run:

Metric Name: quartz.job.execution.active, Description: Number of jobs currently running, Unit: {job}, Metric Type: LongSumNonMonotonic
Metric Name: quartz.job.execution.duration, Description: Elapsed time spent executing a job, Unit: s, Metric Type: Histogram
Metric Name: quartz.jobstore.operation.duration, Description: Elapsed time spent on a job store operation, Unit: s, Metric Type: Histogram
Metric Name: quartz.trigger.acquired, Description: Number of triggers acquired for firing, Unit: {trigger}, Metric Type: LongSum
Metric Name: quartz.trigger.acquisition.duration, Description: Elapsed time spent acquiring the next batch of triggers, Unit: s, Metric Type: Histogram

The most valuable for alerting are the quartz.job.execution.duration histogram (its quartz.job.result tag separates successes from failures) and quartz.job.execution.active. An important migration note: AddQuartzInstrumentation() from the community package OpenTelemetry.Instrumentation.Quartz, written for Quartz 3, doesn't work with 4.x. According to Quartz's XML (Extensible Markup Language) documentation, the old listener and activity sources this package listens to no longer produce data in 4.x; the app keeps running without an error and without seeing any spans. On 4.x use AddSource(QuartzInstrumentation.ActivitySourceName).

Hangfire has no built-in OpenTelemetry support; there's a community OpenTelemetry.Instrumentation.Hangfire package, but only beta versions (1.20.2-beta.1) have been published so far. I didn't try it for this post. Setting up OpenTelemetry, choosing a collector and a backend belong to the upcoming Observability series in the DevOps category.

Structured Logs: JobId and Attempt Count

When a job fails, the first question is "which job, which attempt?". If you add a parameter of type PerformContext to a job method, Hangfire fills it in at run time; with it you can add the job id and retry count to every log line as a scope:

public sealed class InvoiceJobs(ILogger<InvoiceJobs> logger)
{
    public Task SendAsync(int orderId, PerformContext? context)
    {
        // the JobId and attempt count are added to every log line as a scope
        using IDisposable? scope = logger.BeginScope(new Dictionary<string, object?>
        {
            ["JobId"] = context?.BackgroundJob.Id,
            ["RetryCount"] = context?.GetJobParameter<int>("RetryCount") ?? 0
        });
        logger.LogInformation("Sending invoice for order {OrderId}", orderId);
        return Task.CompletedTask;
    }
}

app.MapPost("/orders/{id:int}/invoice", (int id, IBackgroundJobClient jobs) =>
    jobs.Enqueue<InvoiceJobs>(j => j.SendAsync(id, null)));

The output with the JSON (JavaScript Object Notation) console formatter:

{"EventId":0,"LogLevel":"Information","Category":"Bookstore.Api.InvoiceJobs","Message":"Sending invoice for order 42","State":{"OrderId":42,"{OriginalFormat}":"Sending invoice for order {OrderId}"},"Scopes":[{"Message":"System.Collections.Generic.Dictionary`2[System.String,System.Object]","JobId":"1","RetryCount":0}]}

In your log backend you can now filter all lines of a job with JobId = "1", or only the retried jobs with RetryCount > 0. Which fields should go into logs, the cost of scopes, and joining logs with trace context are topics of the upcoming Logging in .NET series.

In Production: Alerting and Security

Three alerts are a good enough start for most systems. First, failed jobs: every job that exhausts its retries and lands in Failed is something a human needs to look at; in Hangfire watch the failed count, in Quartz the duration metric tagged with quartz.job.result. Second, queue growth: if the enqueued count keeps rising over a period, the workers can't keep up. Third, the freshness check above: jobs that don't fail but don't run. Don't alert on a retried job's first failure; that's what retries are for. The alert should come when the retries are exhausted or the success rate drops noticeably.

Don't leave the dashboard open to the internet. Even with authorization, keeping it on the internal network or behind a VPN (Virtual Private Network) shrinks the attack surface. The dashboard is an admin tool, not a screen for customers.

Remember the personal data in job arguments. In the fifth post we saw an order written to the job table with its email and address; the dashboard shows those arguments to everyone. Passing only ids as arguments solves both the staleness and the privacy problem. The same rule holds for logs: the Web Security (OWASP Top 10) series post "OWASP A09: Security Logging and Alerting Failures — What to Log, What Never to Log" covers what must never go into a log.

Common Pitfalls

Trusting the "local only" default. A proxy on the same machine, a sidecar or Docker's port forwarding makes every request look local.

Giving everyone write access. Not everyone looking at the dashboard needs to delete or requeue jobs; separate the roles with IsReadOnlyFunc.

Watching only errors. A job that never runs produces no errors; without a freshness check you notice weeks later.

Moving to Quartz 4 and keeping the old OpenTelemetry package. AddQuartzInstrumentation() silently produces nothing on 4.x.

Wiring the freshness health check to the liveness probe. Restarting the pod because the report didn't run turns the problem into a restart loop.

A Small Experiment

★☆☆ Protect the Hangfire Dashboard with an IDashboardAuthorizationFilter: allow only requests with the correct X-Ops-Key header. What does the request from the Docker container get now?

★★☆ Move JobHeartbeatStore to a database table and run two replicas. While only one replica runs the job, do both replicas' health checks give the same result?

★★★ Publish Quartz's quartz.job.execution.duration metric in Prometheus format and write an alert rule that fires when the failed run rate exceeds 10 percent over the last 15 minutes.

In the last post of the series we look back over time: what changed in BackgroundService and host behavior from .NET 6 to .NET 10, the exit code change .NET 11 brings, the real compiler errors of moving from Quartz 3 to 4, and the IHostedService + Timer pattern you'll meet when reading old code.