Observing Security Events in ASP.NET Core: Authentication, Authorization and Identity Metrics, Audit Trails and Alerts

Version note: The code in this post was built and run with .NET 10 (ASP.NET Core 10.0.13 runtime), using OpenTelemetry.Extensions.Hosting 1.20.1, OpenTelemetry.Exporter.Prometheus.AspNetCore 1.20.1-beta.1, Microsoft.Extensions.Diagnostics.Testing 10.11.0 and Prometheus 3.16.0.

Guess First

In Bookstore, wrong passwords are entered for Alan's account one after another and the account gets locked. During the same minutes Grace sends 46 requests to an admin endpoint she is not allowed to use. With .NET 10's built-in metrics, which of these two events can you see, and in how much detail? Does the metric tell you the name of the locked account? Does it tell you which policy rejected the request?

The answer to both is "no", and that is not a gap but a deliberate design. In this post we set up how the authorization, antiforgery, rate limiting and Kestrel limits from the earlier posts of the series show up in production: .NET 10's authentication, authorization and Identity metrics, collection with OpenTelemetry, Prometheus alert rules and an audit trail for the "who did what, when" question metrics cannot answer. What to write to the log and what never to write is the subject of the post "OWASP A09: Security Logging and Alerting Failures" in the Web Security series, which covers the OWASP (Open Worldwide Application Security Project) Top 10. We do not repeat it here.

.NET 10's Security Metrics

With .NET 10, ASP.NET Core publishes three new meters. The Microsoft.AspNetCore.Authentication meter produces the aspnetcore.authentication.authenticate.duration histogram and the challenges, forbids, sign_ins and sign_outs counters. The Microsoft.AspNetCore.Authorization meter has a single counter, aspnetcore.authorization.attempts, which tags the result as success or failure. The Microsoft.AspNetCore.Identity meter counts UserManager and SignInManager operations: password checks, sign-in results (success, failure, locked_out), user creation and update durations, token generation and verification. We add the Microsoft.AspNetCore.RateLimiting meter, available since .NET 8, to these.

Collecting with OpenTelemetry

Meters cost nothing if nobody listens; we tell the OpenTelemetry SDK (software development kit) which meters we want and expose an endpoint Prometheus can scrape:

builder.Services.AddOpenTelemetry().WithMetrics(metrics => metrics
    .AddMeter("Microsoft.AspNetCore.Authentication", "Microsoft.AspNetCore.Authorization",
        "Microsoft.AspNetCore.Identity", "Microsoft.AspNetCore.RateLimiting", SecurityMetrics.MeterName)
    .AddPrometheusExporter());

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();
app.UseRateLimiter();

app.MapPrometheusScrapingEndpoint();

The Prometheus exporter package is still in beta; in production you usually send to a collector over OTLP (OpenTelemetry Protocol) and forward from there to Prometheus or another backend. Also, do not leave the /metrics endpoint open to the internet; put it on a separate port or behind a network policy so only Prometheus can reach it. I played the scenario against the local app: three anonymous order requests, six wrong passwords for Alan, two role grants by Ada and three requests from Grace to the admin endpoint. The relevant lines from /metrics:

aspnetcore_authentication_challenges_total{otel_scope_name="Microsoft.AspNetCore.Authentication"} 3
aspnetcore_authentication_forbids_total{otel_scope_name="Microsoft.AspNetCore.Authentication"} 3
aspnetcore_authentication_sign_ins_total{otel_scope_name="Microsoft.AspNetCore.Authentication",aspnetcore_authentication_scheme="Identity.Application"} 2
aspnetcore_authorization_attempts_total{otel_scope_name="Microsoft.AspNetCore.Authorization",aspnetcore_authorization_result="failure",aspnetcore_user_is_authenticated="false"} 3
aspnetcore_authorization_attempts_total{otel_scope_name="Microsoft.AspNetCore.Authorization",aspnetcore_authorization_result="success",aspnetcore_user_is_authenticated="true"} 3
aspnetcore_authorization_attempts_total{otel_scope_name="Microsoft.AspNetCore.Authorization",aspnetcore_authorization_result="failure",aspnetcore_user_is_authenticated="true"} 3
aspnetcore_identity_sign_in_check_password_attempts_total{otel_scope_name="Microsoft.AspNetCore.Identity",aspnetcore_identity_sign_in_result="failure",aspnetcore_identity_user_type="Microsoft.AspNetCore.Identity.IdentityUser"} 4
aspnetcore_identity_sign_in_check_password_attempts_total{otel_scope_name="Microsoft.AspNetCore.Identity",aspnetcore_identity_sign_in_result="locked_out",aspnetcore_identity_user_type="Microsoft.AspNetCore.Identity.IdentityUser"} 2
aspnetcore_identity_sign_in_check_password_attempts_total{otel_scope_name="Microsoft.AspNetCore.Identity",aspnetcore_identity_sign_in_result="success",aspnetcore_identity_user_type="Microsoft.AspNetCore.Identity.IdentityUser"} 2
aspnetcore_rate_limiting_requests_total{otel_scope_name="Microsoft.AspNetCore.RateLimiting",aspnetcore_rate_limiting_policy="login",aspnetcore_rate_limiting_result="acquired"} 8
bookstore_audit_events_total{otel_scope_name="Bookstore.Security",bookstore_audit_action="role_granted"} 2

Alan's six attempts were counted as 4 failure and 2 locked_out: with MaxFailedAccessAttempts = 5, the fifth wrong password locked the account and that attempt also came back with locked_out. On the authorization side, anonymous rejections (401) and rejections of authenticated users (403) are separated by the aspnetcore_user_is_authenticated label; this split is very valuable when building alerts, because the second kind is almost always either a bug or someone probing permissions.

What Metrics Do Not Tell You

The output has no user name and no IP (Internet Protocol) address. That is deliberate: every distinct label value is a new time series, and an unbounded label like a user name brings a metrics system down within days; it also carries personal data into the metrics system. The answer to "which account was locked" lives in the log or the audit trail.

There are two more surprises. First, the metric has no policy name either. The table on Microsoft Learn lists the aspnetcore.authorization.policy attribute as present when a policy is used, but on an endpoint protected with RequireAuthorization("Admin") the middleware combines the policy first, so the name did not make it into the tag. Second, the attribute name in the documentation is user.is_authenticated, while the real output has aspnetcore.user.is_authenticated. I pinned both down with a test; differences like these between documentation and behavior should surface before you write the alert query, not after:

[Fact]
public async Task Forbidden_request_is_counted_as_authorization_failure()
{
    string db = Path.Combine(Path.GetTempPath(), $"bookstore-{Guid.NewGuid():N}.db");
    await using WebApplicationFactory<Program> factory = new WebApplicationFactory<Program>()
        .WithWebHostBuilder(b => b.UseSetting("ConnectionStrings:Bookstore", $"Data Source={db}"));
    HttpClient client = factory.CreateClient();
    IMeterFactory meterFactory = factory.Services.GetRequiredService<IMeterFactory>();
    using MetricCollector<long> attempts = new(meterFactory, "Microsoft.AspNetCore.Authorization", "aspnetcore.authorization.attempts");

    HttpResponseMessage login = await client.PostAsJsonAsync("/login",
        new LoginRequest("grace", "dev-only-Passw0rd!"), TestContext.Current.CancellationToken);
    HttpResponseMessage response = await client.GetAsync("/admin/audit/verify", TestContext.Current.CancellationToken);

    Assert.Equal(HttpStatusCode.OK, login.StatusCode);
    Assert.Equal(HttpStatusCode.Forbidden, response.StatusCode);
    CollectedMeasurement<long> measurement = Assert.Single(attempts.GetMeasurementSnapshot());
    Assert.Equal("failure", measurement.Tags["aspnetcore.authorization.result"]);
    // the endpoint policy is combined in the middleware, so its name does not reach the tag
    Assert.False(measurement.Tags.ContainsKey("aspnetcore.authorization.policy"));
    Assert.Equal(true, measurement.Tags["aspnetcore.user.is_authenticated"]);
}

MetricCollector<T> comes from the Microsoft.Extensions.Diagnostics.Testing package, and because it attaches to the app's own IMeterFactory, it collects only this test server's measurements. If you want to see which policy rejected a request, do it with your own log in the IAuthorizationMiddlewareResultHandler from the first post of the series.

An App-Specific Metric: Bookstore.Security

There are security events the framework does not know about: granting a role, an admin acting on behalf of another user (impersonation), issuing an API (Application Programming Interface) key. For these we write a small meter. As a label we use only the action name, which has a bounded set of values:

public sealed class SecurityMetrics
{
    public const string MeterName = "Bookstore.Security";

    private readonly Counter<long> _auditEvents;

    public SecurityMetrics(IMeterFactory meterFactory)
    {
        Meter meter = meterFactory.Create(MeterName);
        _auditEvents = meter.CreateCounter<long>("bookstore.audit.events", unit: "{event}",
            description: "Security-relevant actions written to the audit trail.");
    }

    public void AuditEventRecorded(string action) =>
        _auditEvents.Add(1, new KeyValuePair<string, object?>("bookstore.audit.action", action));
}

A meter created through IMeterFactory is tied to the lifetime of the DI (dependency injection) container and can be isolated in tests; opening a static field with new Meter(...) lets measurements bleed across tests.

Alert Rules

The metrics are being collected; now we decide which situations should wake a human. I ran the Prometheus rules in my own temporary container (Prometheus 3.16.0), scraping the local app every 5 seconds. The thresholds are small for the demo:

groups:
  - name: bookstore-security
    rules:
      - alert: AuthorizationFailureSpike
        expr: sum(rate(aspnetcore_authorization_attempts_total{aspnetcore_authorization_result="failure"}[1m])) > 0.5
        labels: { severity: warning }
        annotations:
          summary: "More than 30 authorization failures per minute"
      - alert: AccountLockedOut
        expr: sum(increase(aspnetcore_identity_sign_in_check_password_attempts_total{aspnetcore_identity_sign_in_result="locked_out"}[5m])) > 0
        labels: { severity: warning }
        annotations:
          summary: "Sign-in attempts against a locked-out account"
      - alert: LoginRateLimited
        expr: sum(increase(aspnetcore_rate_limiting_requests_total{aspnetcore_rate_limiting_result!="acquired"}[5m])) > 0
        labels: { severity: info }
        annotations:
          summary: "Login requests rejected by the rate limiter"
      - alert: PrivilegeGranted
        expr: sum by (bookstore_audit_action) (increase(bookstore_audit_events_total{bookstore_audit_action="role_granted"}[10m])) > 0
        labels: { severity: info }
        annotations:
          summary: "A role was granted; check the audit trail"

After the scenario, with Grace sending 46 requests to the admin endpoint over 25 seconds and Ada making a third role grant, these are the alerts the Prometheus API returned:

firing AuthorizationFailureSpike
firing AccountLockedOut
firing LoginRateLimited
firing PrivilegeGranted role_granted

I made an instructive mistake here. After the first two role grants the PrivilegeGranted alert did not fire. The counter came into existence after the app started, together with two grants, and the first value Prometheus saw was 2. increase() measures the difference between two samples; it cannot know what happened before the first one. For series that count rare events this is a serious problem: the first event may never produce an alert. The fix is to publish the counter with zero for the known actions at startup (Add(0, ...)), or to tie such events to a rule that alerts directly from the audit record instead of a metric.

There is no framework metric for antiforgery failures; adding a counter in the Bookstore.Security meter to the filter from the series' antiforgery post follows the same pattern. The Unprotect ticket failed from the Data Protection post, on the other hand, is a log line; tying it to a log-based alert rather than the metrics system is more appropriate.

Audit Trail: Who, What, When

A log tells you what the system did and gets deleted by rotation. An audit trail is the account of what a person did: who granted which permission to whom, which admin acted on behalf of which user. It is kept in a separate table with a separate retention period, and it is the first place you look during an incident investigation. In Bookstore a role grant is recorded like this:

admin.MapPost("/users/{userName}/roles/{role}", async (string userName, string role, ClaimsPrincipal actor,
    UserManager<IdentityUser> users, AuditTrail audit) =>
{
    IdentityUser? user = await users.FindByNameAsync(userName);
    if (user is null)
    {
        return Results.NotFound();
    }

    IdentityResult result = await users.AddToRoleAsync(user, role);
    if (!result.Succeeded)
    {
        return Results.ValidationProblem(result.Errors.ToDictionary(e => e.Code, e => new[] { e.Description }));
    }

    await audit.RecordAsync(actor.Identity!.Name!, "role_granted", userName, $"role={role}");
    return Results.NoContent();
});

The audit record is written after the operation is known to have succeeded; failed attempts are valuable too, but should be recorded under a separate action name (role_grant_failed). In a more robust setup the role change and the audit record are written in the same database transaction, so you never end up with one and not the other.

Immutability: A Hash Chain and Beyond

Someone with write access to the database can quietly change an audit record. If we chain each record's hash to the previous one, we cannot prevent the change, but we can detect it:

public sealed class AuditTrail(BookstoreDbContext db, TimeProvider clock, SecurityMetrics metrics)
{
    private const string Genesis = "GENESIS";

    public async Task RecordAsync(string actor, string action, string target, string? details = null,
        CancellationToken cancellationToken = default)
    {
        // assumes a single writer; ordered writes across instances need a lock on the database side
        string previous = await db.AuditEvents.OrderByDescending(e => e.Id).Select(e => e.Hash)
            .FirstOrDefaultAsync(cancellationToken) ?? Genesis;
        AuditEvent entry = new()
        {
            OccurredAt = clock.GetUtcNow(),
            Actor = actor,
            Action = action,
            Target = target,
            Details = details,
            PreviousHash = previous,
            Hash = ""
        };
        entry.Hash = ComputeHash(entry);
        db.AuditEvents.Add(entry);
        await db.SaveChangesAsync(cancellationToken);
        metrics.AuditEventRecorded(action);
    }

    public async Task<long?> FindFirstBrokenLinkAsync(CancellationToken cancellationToken = default)
    {
        string previous = Genesis;
        await foreach (AuditEvent entry in db.AuditEvents.AsNoTracking().OrderBy(e => e.Id)
                           .AsAsyncEnumerable().WithCancellation(cancellationToken))
        {
            if (entry.PreviousHash != previous || entry.Hash != ComputeHash(entry))
            {
                return entry.Id;
            }
            previous = entry.Hash;
        }
        return null;
    }

    private static string ComputeHash(AuditEvent e)
    {
        string canonical = string.Join('\n', e.PreviousHash, e.OccurredAt.ToUnixTimeMilliseconds(),
            e.Actor, e.Action, e.Target, e.Details ?? "");
        return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(canonical)));
    }
}

In the local database I manually changed the admin role granted to Alan into support, imitating someone covering their tracks:

$ sqlite3 Bookstore.Api/bookstore.db "select Id, Actor, Action, Target, Details, substr(PreviousHash,1,12), substr(Hash,1,12) from AuditEvents;"
1|ada|role_granted|grace|role=support|GENESIS|19F6040A12CF
2|ada|role_granted|alan|role=admin|19F6040A12CF|72361A39EA98
$ sqlite3 Bookstore.Api/bookstore.db "update AuditEvents set Details='role=support' where Id=2;"
$ curl -s -b run2/ada http://127.0.0.1:19350/admin/audit/verify
{"intact":false,"firstBrokenId":2}

Knowing the chain's limit matters. Someone with full rights on the database can recompute and rewrite the whole chain from the start; a hash chain only catches partial changes. Real immutability requires periodically writing the chain's latest hash somewhere else (a log system kept in a separate account, a blob storage with object lock), giving the app's database user only INSERT rights on the audit table, and streaming records into a WORM (write once, read many) store. Database features such as SQL (Structured Query Language) Server's ledger tables and the pgAudit extension for PostgreSQL also cover parts of this need.

Production Notes

Keep the number of alerts small. Every time a security alert fires someone should look; an alert that rings ten times a day is ignored within a week. Of the four alerts in this post, only two (AuthorizationFailureSpike and AccountLockedOut) should wake someone at night; the others should go to a channel as messages. Set thresholds not by guesswork but by looking at the distribution of normal traffic over two weeks.

A locked-account alert says little on its own; the real question is how many different accounts were locked. A user forgetting a password is normal; a hundred accounts locking within the same hour is credential stuffing. Since the metric has no account name, you answer this with a log query: count lockout logs per account.

The audit trail's retention should be longer than the log's, and deleting it should require a separate permission. Regulations such as KVKK (Kişisel Verilerin Korunması Kanunu, Turkey's personal data protection law) and GDPR (General Data Protection Regulation) require retention on one hand and keeping personal data to a minimum on the other; write the user's identifier to the audit record, not their email.

Common Pitfalls

Putting a user name, email or IP label on a metric. A cardinality explosion and personal data in the metrics system; this information belongs in the log and the audit trail.

increase() missing the first event on counters of rare events, as in the PrivilegeGranted example above; publish the counter with zero at startup.

Writing alerts based on the attribute name in the documentation. As with aspnetcore.user.is_authenticated in this post, verify the real label from the /metrics output or a test.

Writing the audit record before the operation. A failed operation then appears as "done"; write the record after the outcome is final and, if possible, in the same transaction.

A Small Experiment

★☆☆ Add a call to Add(0, ...) for role_granted at startup to SecurityMetrics, restart Prometheus and verify that the PrivilegeGranted alert fires on the first role grant.

★★☆ Move the ValidateAntiforgeryFilter from the antiforgery post into this project and count failures with a bookstore.antiforgery.failures counter in the Bookstore.Security meter. Write an alert rule for more than 10 failures per minute.

★★★ Write the audit chain's latest hash to a separate file as a "checkpoint" every 100 records and extend verification to compare against the checkpoints. Test how this catches an attacker who recomputes the whole chain.

This post completes the .NET 10 block of the series. In the last post we look at what .NET 11 brings to ASP.NET Core security: automatic CSRF (cross-site request forgery) protection based on Fetch Metadata, Device Bound Session Credentials and TLS (Transport Layer Security) channel binding.