Continuously Testing and Monitoring API Security: Authorization Regression, OpenAPI-Driven Scanning and Abuse Telemetry

Version note: The code in this post was run with .NET 10, OpenTelemetry.Extensions.Hosting 1.20.1, Microsoft.Extensions.Diagnostics.Testing 10.11.0 and Schemathesis 4.33.1; scanning was done only against the local lab.

Guess First

Over nine posts we added dozens of controls to the lab and wrote a test for each. Now think about this: what happens if tomorrow someone sends {"orderIds": null} to the POST /orders/cancel endpoint? None of our tests tried that input, because when writing tests we wrote the inputs that came to mind. And in production, how would you notice that a tenant is trying other tenants' order ids?

This post is something like the series' closing post: it places the defences we built into a test pyramid, runs a scanner that generates inputs from the OpenAPI document against our own lab and turns what it finds into regression tests, sets up telemetry that measures rejected requests per tenant, and finally produces a one-page checklist. The basics of what to write to logs and what never to write are in the A09 Security Logging and Alerting Failures post of the Web Security (OWASP Top 10) series.

The Test Pyramid: Who Catches What

The lab's test project now has 186 tests and they all run in about 2.6 seconds. The distribution shows the shape of the pyramid:

• Unit: the cancellation rules of OrderAuthorizationHandler, AdminPortFilter and similar pure logic. No host, no database, milliseconds.
• Integration: the authorization matrix (89 rows, 96 tests with the generator), inventory tests, property level, resource limits, sensitive flows, outbound requests, configuration. With WebApplicationFactory and in-memory SQLite.
• Scanning: a tool that generates inputs automatically from the OpenAPI document, only against our own environment. It finds inputs we didn't think of.
• Production telemetry: not a test, but it asks the same question of live traffic: who is rejected, where, and how often?

Unit tests at the base let you try every combination of the rules without expensive setup. The test for the cancellation rule from the second post:

// The base of the pyramid: the handler's rules without a host, in milliseconds
public sealed class OrderAuthorizationHandlerTests
{
    private static async Task<bool> CanCancelAsync(string userName, string ownerName, OrderStatus status, string? tenant = null)
    {
        AppUser user = SeedData.UserNamed(userName);
        AppUser owner = SeedData.UserNamed(ownerName);
        ClaimsPrincipal principal = new(new ClaimsIdentity(
        [
            new Claim(LabClaims.Subject, user.Id.ToString()),
            new Claim(LabClaims.Tenant, tenant ?? user.TenantId),
            new Claim(LabClaims.Role, user.Role)
        ], "Test", LabClaims.Subject, LabClaims.Role));
        Order order = new() { Id = Guid.CreateVersion7(), TenantId = owner.TenantId, CustomerId = owner.Id,
            Status = status, ShippingAddress = "x" };
        AuthorizationHandlerContext context = new([OrderOperations.Cancel], principal, order);

        await new OrderAuthorizationHandler().HandleAsync(context);
        return context.HasSucceeded;
    }

    [Theory]
    [InlineData("alan", "alan", OrderStatus.Pending, true)]
    [InlineData("alan", "alan", OrderStatus.Paid, false)]
    [InlineData("alan", "grace", OrderStatus.Pending, false)]
    [InlineData("margaret", "alan", OrderStatus.Paid, true)]
    [InlineData("margaret", "alan", OrderStatus.Shipped, false)]
    [InlineData("ada", "alan", OrderStatus.Shipped, true)]
    [InlineData("ada", "alan", OrderStatus.Cancelled, false)]
    [InlineData("edsger", "alan", OrderStatus.Pending, false)]
    public async Task Cancel_rules(string user, string owner, OrderStatus status, bool expected)
    {
        Assert.Equal(expected, await CanCancelAsync(user, owner, status));
    }
}

A unit test proves the rule is written correctly, but not that the rule is called in the right place. You can test the handler perfectly and forget to add the AuthorizeAsync call to the endpoint. That's why the weight of the pyramid sits in the integration layer: the matrix and inventory tests ask how each endpoint really behaves. What we call authorization regression is exactly this: proving on every build that yesterday's decision still holds today.

OpenAPI-Driven Scanning: Only Against Your Own Environment

The matrix tests the scenarios we thought of. For the ones we didn't, we use a tool that reads the OpenAPI document and generates inputs, both schema-conforming and not, for every operation. In the lab I installed Schemathesis (an open-source, schema-based API (Application Programming Interface) testing tool) into a Python environment in the working folder. The API scan mode of OWASP ZAP (Zed Attack Proxy) also starts from the OpenAPI document in the same way; in this post I ran only Schemathesis.

An important rule: these tools are run only against an environment you own and that is set aside for testing. I ran it against the local lab, in the Development environment and with Margaret's token. In CI (Continuous Integration) the target should be your own staging environment; never production, a partner's sandbox or a third-party API. A scan sends thousands of requests, creates data and triggers rate limits.

$ schemathesis run http://localhost:19520/openapi/v1.json -H "Authorization: Bearer $MARGARET" --max-examples 15 --no-color

The summary of the first run:

Failures:
  ❌ ST202 Schema-valid request rejected: 3
  ❌ ST103 Undocumented status code: 26

Errors:
  🚫 Schema Error: 1

Test cases:
  301 generated, 28 found 29 unique failures

No server errors at all, but every one of the 26 operations has an "undocumented status code". The reason is in the contract: according to the document every operation returns only 200, while the API returns 400, 401, 403, 404 and 429. Three operations also have a "schema-valid request rejected" finding: according to the document pageSize can be any integer, while the API rejects anything above 100. The scanner showed us the gap between contract and behaviour. Its security meaning is this: a team writing a client by trusting the document only learns about the limits in production. One more interesting detail: since it ran in the Development environment, some 400 responses contained exception details. That's one more reason to scan a staging environment with production settings; the API8 (Security Misconfiguration) tests in the ninth post guard this separately.

I fixed the contract in two steps. An OpenAPI operation transformer that adds the standard security responses to every operation, and range information on the list parameters:

// Security responses every operation can return are written into the contract; scanners and clients don't treat them as unexpected
public sealed class StandardResponsesTransformer : IOpenApiOperationTransformer
{
    private static readonly (string Code, string Description)[] Standard =
    [
        ("400", "The request is invalid."),
        ("401", "Authentication is required."),
        ("403", "The caller is not allowed to perform this operation."),
        ("404", "The resource does not exist or is outside the caller's scope."),
        ("429", "A rate limit or quota was exceeded.")
    ];

    public Task TransformAsync(OpenApiOperation operation, OpenApiOperationTransformerContext context,
        CancellationToken cancellationToken)
    {
        operation.Responses ??= [];
        foreach ((string code, string description) in Standard)
        {
            operation.Responses.TryAdd(code, new OpenApiResponse { Description = description });
        }
        return Task.CompletedTask;
    }
}

public sealed record OrderListQuery(
    [property: Range(1, 10_000)] int? Page,
    [property: Range(1, ResourceLimits.MaxPageSize)] int? PageSize,
    [property: AllowedValues("createdAt", "-createdAt", "status", null)] string? Sort,
    OrderStatus? Status);

The Range attributes appeared in the document as minimum and maximum; AllowedValues didn't, so sort is still a free string in the document. These attributes are here only for the contract: validation is still done by the allowlist in the endpoint.

From Finding to Regression Test

Once the contract was fixed, the scanner went deeper, and the second run brought the really valuable finding:

Failures:
  ❌ ST101 Server error: 2
  ❌ ST202 Schema-valid request rejected: 3
  ❌ ST103 Undocumented status code: 4

    curl -X POST -H 'Authorization: [Filtered]' -H 'Content-Type: application/json' -d '{"email": null}' http://localhost:19520/account/password-reset

    curl -X POST -H 'Authorization: [Filtered]' -H 'Content-Type: application/json' -d '{"orderIds": null}' http://localhost:19520/orders/cancel

Both endpoints returned 500: one a NullReferenceException, the other an ArgumentNullException inside Distinct(). In both records the fields are declared non-nullable (string Email, IReadOnlyList<Guid> OrderIds), but System.Text.Json doesn't look at nullable annotations by default and accepts null. Adding if (request.Email is null) one by one will be forgotten in the next endpoint. The systematic fix is turning on two settings that came with .NET 9 in the JSON (JavaScript Object Notation) options:

// The root cause of the two 500s the scanner found: null in a non-nullable field; now a 400 at binding
builder.Services.ConfigureHttpJsonOptions(options =>
{
    options.SerializerOptions.RespectNullableAnnotations = true;
    options.SerializerOptions.RespectRequiredConstructorParameters = true;
});

Every finding becomes a regression test, so the bug doesn't come back even if the scanner never runs again:

// The 500s Schemathesis found: every finding becomes a regression test
public sealed class ScanRegressionTests(OrdersApiFactory factory) : IClassFixture<OrdersApiFactory>
{
    [Theory]
    [InlineData("/account/password-reset", "anonymous", """{"email": null}""")]
    [InlineData("/orders/cancel", "margaret", """{"orderIds": null}""")]
    public async Task Null_for_a_non_nullable_field_is_a_400_not_a_500(string path, string actor, string body)
    {
        using HttpResponseMessage response = await factory.CreateClientFor(actor).PostAsync(path,
            new StringContent(body, Encoding.UTF8, "application/json"), TestContext.Current.CancellationToken);

        Assert.Equal(HttpStatusCode.BadRequest, response.StatusCode);
    }
}

In the third run there were no server errors left:

Failures:
  ❌ ST202 Schema-valid request rejected: 3
  ❌ ST103 Undocumented status code: 1

Errors:
  🚫 Schema Error: 1

Test cases:
  738 generated, 4 found 4 unique failures

The remaining findings are ones I accept deliberately: cross-field rules that can't be expressed in the schema (the product of page and page size, the report range not exceeding 92 days), the sort allowlist not reflected in the document, and the {version} parameter of the versioned route not being defined in the document, which we saw in the ninth post. Listing these explicitly as findings to ignore in CI stops a new finding from getting lost in the noise.

Abuse Telemetry: What to Measure

Tests guard known behaviour; telemetry shows the unusual in live traffic. For API security the most valuable signals are rejected requests: 401 (no identity or an invalid one), 403 (identity but no permission), 404 (sometimes an out-of-scope record) and 429 (a limit). I wrote a small middleware that counts them by route, tenant and status code, and a counter added to the ScopeProbe from the second post:

// Counts rejected requests by route, tenant and status code; a user id or IP never becomes a metric tag
public sealed class SecurityMetrics
{
    public const string MeterName = "Bookstore.Orders.Security";
    private readonly Counter<long> _rejected;
    private readonly Counter<long> _outOfScope;

    public SecurityMetrics(IMeterFactory meterFactory)
    {
        Meter meter = meterFactory.Create(MeterName);
        _rejected = meter.CreateCounter<long>("bookstore.security.rejected_requests", "{request}",
            "Requests rejected with 401, 403, 404 or 429.");
        _outOfScope = meter.CreateCounter<long>("bookstore.security.out_of_scope_lookups", "{lookup}",
            "Lookups of records that exist outside the caller's scope.");
    }

    public void Rejected(string route, string tenant, int statusCode) =>
        _rejected.Add(1, new("http.route", route), new("tenant", tenant), new("http.response.status_code", statusCode));

    public void OutOfScope(string resource, string tenant) =>
        _outOfScope.Add(1, new("resource", resource), new("tenant", tenant));
}

The tag choice is deliberate. Route, tenant and status code take a limited number of values; metric systems keep a separate series for each tag combination, and unbounded values such as a user id or IP (Internet Protocol) address both blow up the cost and carry personal data into the metric infrastructure. The answer to "who" lives in the log, with a keyId or user id; the metric only says "where and how much". Unmatched paths are collected into a single (unmatched) value; otherwise someone scanning would create a new series for every URL (Uniform Resource Locator) they try.

ASP.NET Core has its own meters too: counters for rate limiting since .NET 8, and for authorization that came with .NET 10. I collect all three together with OpenTelemetry. In the lab I turned on the console exporter and, as Alan, requested two out-of-scope orders, one order that never existed and one forbidden endpoint, and as Barbara three report requests that exceeded the quota:

Metric Name: bookstore.security.rejected_requests, Description: Requests rejected with 401, 403, 404 or 429., Unit: {request}, Metric Type: LongSum
Instrumentation scope (Meter):
	Name: Bookstore.Orders.Security
(2026-11-08T06:35:55.0891600Z, 2026-11-08T06:36:00.0301960Z] http.response.status_code: 404 http.route: /orders/{orderId:guid} tenant: north
Value: 3
(2026-11-08T06:35:55.0891600Z, 2026-11-08T06:36:00.0301960Z] http.response.status_code: 403 http.route: /support/orders/{orderId:guid} tenant: north
Value: 1
(2026-11-08T06:35:55.0891600Z, 2026-11-08T06:36:00.0301960Z] http.response.status_code: 429 http.route: /reports/daily-sales tenant: south
Value: 1

Metric Name: bookstore.security.out_of_scope_lookups, Description: Lookups of records that exist outside the caller's scope., Unit: {lookup}, Metric Type: LongSum
Instrumentation scope (Meter):
	Name: Bookstore.Orders.Security
(2026-11-08T06:35:55.0891870Z, 2026-11-08T06:36:00.0301980Z] resource: Order tenant: north
Value: 2

Metric Name: aspnetcore.authorization.attempts, Description: The total number of authorization attempts., Unit: {attempt}, Metric Type: LongSum
Instrumentation scope (Meter):
	Name: Microsoft.AspNetCore.Authorization
(2026-11-08T06:35:58.7851260Z, 2026-11-08T06:36:00.0301980Z] aspnetcore.authorization.result: success aspnetcore.user.is_authenticated: true
Value: 6
(2026-11-08T06:35:58.7851260Z, 2026-11-08T06:36:00.0301980Z] aspnetcore.authorization.result: failure aspnetcore.user.is_authenticated: true
Value: 1

Two of the three 404s also show up in out_of_scope_lookups: those are records that exist but belong to another tenant. The third is an id that never existed. The ratio of the two counters is a good signal for alerts: if most of a tenant's 404s go to out-of-scope records, someone is collecting ids or an integration is asking for the wrong tenant's data. These metrics are tested too; there's a test that checks with MetricCollector<T> that the counter increases with the right tags.

The alerts I set up are of this kind: the number of 403s in a tenant over 5 minutes is far above the average for the same hour over the last 7 days; out_of_scope_lookups is non-zero (it should almost never happen in normal traffic); the 429 rate on a route suddenly rises. Using deviation from the tenant's own history instead of an absolute threshold makes it possible to run the same rule for large and small tenants.

The Role and Limits of the Gateway and the WAF

Where do the API gateway and the WAF (Web Application Firewall) fit in this picture? The gateway is the right place for connection- and IP-level limits, TLS (Transport Layer Security) termination, coarse authentication and the route inventory. The WAF is good at cutting known attack signatures, malformed requests and obvious injection attempts.

What neither knows is your business logic. A WAF can't know why Alan's GET /orders/0197f001-...-03 request should be rejected: the request is syntactically flawless, the token is valid, the id is a valid GUID (Globally Unique Identifier). The first six items of this series, i.e. BOLA (Broken Object Level Authorization), BOPLA (Broken Object Property Level Authorization), BFLA (Broken Function Level Authorization) and sensitive business flows, are solved only in code that knows the application's data. We'll build the gateway in detail in the upcoming API Gateway and YARP (Yet Another Reverse Proxy) series; the practical rule here: every check at the gateway is a layer added in front of the application's, not a replacement for it.

A One-Page API Security Checklist

All the series' controls in one table. Each row is a risk, the defence in the lab and the test or evidence guarding it:

Production Notes

Run scanning in CI not on every pull request but after deploying to staging and nightly; a short run (a few examples per operation) on every deploy, a long one at night. The scanning account should run in its own tenant with fake data, and that tenant's quota should be set deliberately; otherwise rate limits turn most of the scan into 429s. In the third run, Schemathesis's "mostly returned 429" warning was saying exactly that.

Telemetry itself is an attack surface. The data going to the metric and log infrastructure must not contain tokens, API keys, signatures, emails or addresses. A tracing or access log setup that records the whole URL also records everything in the query string; that's one of the reasons we rejected API keys in the query string in the third post.

Alerts need an owner. If the person who gets the "403 rate increased" alert doesn't know what to do, the alert gets muted a week later. Write a short runbook for each alert: which dashboard to look at, which log query to run, who talks to the tenant.

Common Pitfalls

The first pitfall is fixing a scanner finding once. If the finding doesn't become a regression test, it comes back in the next refactor and nobody notices until the scanner runs again.

The second is running a scanning tool against an environment you don't own. A partner's sandbox, a shared test environment or production aren't suitable for scanning; it carries both legal and operational risk.

The third is putting a user id, an IP or a raw URL into a metric. Cardinality explodes, costs go up and personal data leaks into a system outside the logging policies.

The fourth is relaxing in-app authorization because there's a gateway or WAF. Every request that passes the gateway must pass the same checks in the application; an internal network call that bypasses the gateway or a wrong route rule breaks that assumption overnight.

A Small Experiment

★☆☆ Start the lab in Development and run Schemathesis with Alan's token. How does it differ from the result you got with Margaret, and why?

★★☆ Design an alert rule for bookstore.security.rejected_requests: a 5-minute window per tenant, deviation from the same hour over the last 7 days. Write it as a metric query and trigger it in the lab with fake traffic.

★★★ Give Schemathesis a configuration file: let it exclude the known and accepted findings explicitly (the ST202 cross-field rules, the {version} schema error) and break CI when a new finding appears. Keep the list of accepted findings in a file that goes through code review.

In the last post of the series we look back at the list: what changed in the OWASP API Security Top 10 from the 2019 edition to 2023, how to translate old pentest reports into today's categories, and how we'll update this series when a new edition arrives.