Responding to an Application Security Incident: Detection, Scoping, Preserving Evidence, Communication and Notification

Version note: The code in this post was built and run with .NET 10; the tests ran with xUnit v3 4.0.3. The incident described is a scenario: the audit records are synthetic data generated with the demo-data command, and the output of the tools and tests is real.

Guess First

12 November, 08:40. Nevin Hanım, the procurement officer of Lakeside School, calls Hakan, the institutional sales lead of Bookstore, the online bookstore used as the shared example across this site's series: someone from Hillcrest University mentioned the school's Bookstore orders in a meeting; they had seen the list. Hakan passes it to the team. Emre finds the cause in ten minutes: the GET /api/institutions/{institutionId}/orders endpoint added for the institution portal in release 2 does not compare the institution id in the URL with the caller's institution. The fix is a single if.

Guess: if Emre deploys the fix right away, cleans up the noisy logs produced during the investigation and says "fixed", what is lost? The answer: almost everything. Who saw which orders of which institution, and when, can no longer be known; the only sentence for the customer is a guess; nothing is left to rely on for a possible legal notification. In this post we talk about responding to an application security incident: detection, preserving evidence, deriving the scope from records, fixing, communication and notification.

Why Is a Security Incident Different?

Severity levels, the incident commander role, status updates and postmortems were covered in the incident management and postmortem posts of the SRE and Production Incidents series; Bookstore uses the same process. A security incident adds four differences on top of it. The first is evidence: in an outage the goal is to bring the service back as fast as possible; in a security incident the answer to "what happened?" matters as much as the fix, and a hasty change can erase that answer. The second is the other side: an outage has a bug, a security incident sometimes has someone who is still in the system. Closing an account, blocking an IP or announcing to everyone can warn them and give them time to cover their tracks. The third is the law: if personal data was affected, the notification deadline under KVKK has started running, so the person responsible for personal data and legal are informed in the first hour. The fourth is scope: the list of affected users must come from records, not from estimates.

The third revision of NIST's incident response guide, SP 800-61r3, was published on 3 April 2025. The four-phase life cycle of the previous revision (preparation; detection and analysis; containment, eradication and recovery; post-incident activity) is no longer the document's main model. Rev. 3 organises its recommendations around the six functions of CSF 2.0: Govern, Identify and Protect cover preparation; Detect, Respond and Recover cover the response itself. The document also says organisations can use whichever life cycle model suits them. The practical lesson for this post: the response starts before the incident. A team with no audit log, no runbook and no rehearsal tries to invent them on the day of the incident.

Detection: Noticing Before the Customer Does

This incident was detected by a customer's phone call; that is the worst channel for detection. Yet the signal was in the system. The endpoint wrote an audit record on every call: who, from which institution, asked for which institution's data, which order ids were returned, what the outcome was, and which trace it belongs to. A single allowed row where the caller's institution differs from the requested institution is reason for an alert. In Bookstore this alert was set up after the incident; how to set it up is in the post on observing security events in the ASP.NET Core Security series. The class that holds the audit record is short; it stores identifiers only, never the order contents:

public sealed class AccessAuditEntry
{
    public long Id { get; set; }
    public DateTime OccurredAtUtc { get; set; }
    public required string UserId { get; set; }
    public int CallerInstitutionId { get; set; }
    public required string Action { get; set; }
    public int ResourceInstitutionId { get; set; }
    public string ResourceIds { get; set; } = "";
    public required string Outcome { get; set; }
    public string? TraceId { get; set; }
}

The bug itself is of the kind covered systematically in the BOLA post of the API Security series. Running the regression test against the code before the fix:

failed Bookstore.Api.Tests.InstitutionOrdersIncidentTests.Another_institutions_order_list_is_not_found_and_the_attempt_is_audited (3ms)
  Assert.Equal() Failure: Values differ
  Expected: NotFound
  Actual:   OK
Test run summary: Failed!
  total: 3
  failed: 1
  succeeded: 2

Preserving Evidence

The most important rule of the first hour: copy before you change anything. Bookstore has a small tool in the repo for this. The preserve command uses SQLite's online backup to take a consistent copy even while the API is running, exports the audit table row by row as JSON, and writes the SHA-256 hash of every file into a manifest. If you use SQL Server in production, the same step is a database backup plus an export of the time window from the log and trace store; the principle does not change. The audit records here are synthetic; they were generated for the exercise with the demo-data command:

dotnet run --project tools/Bookstore.IncidentTools -- demo-data incident/bookstore.db
dotnet run --project tools/Bookstore.IncidentTools -- preserve incident/bookstore.db incident/evidence

23 synthetic audit rows written to bookstore.db
copied database, exported 23 audit rows
57ca1efb32b48bb5f66ba418c42f2ab6413ed63d8b597289eca480e7aa4cd91b  bookstore-evidence.db
bb8e8b639d38dddc9734e9323b8925575f6b8d9d5599a7d9456bfdc71e8fa219  access-audit.jsonl

The copy itself is a database file, and SQLite may create -wal and -shm files next to it when it opens it. That is why the tool switches the copy's journal mode to DELETE; the evidence folder keeps a single, self-contained file:

// SQLite online backup: a consistent copy even while the API is writing.
await using (SqliteConnection source = new($"Data Source={dbPath};Mode=ReadOnly"))
await using (SqliteConnection target = new($"Data Source={copy};Pooling=False"))
{
    await source.OpenAsync();
    await target.OpenAsync();
    source.BackupDatabase(target);
    // A single self-contained file: no -wal/-shm side files next to the evidence.
    await using SqliteCommand journal = target.CreateCommand();
    journal.CommandText = "PRAGMA journal_mode=DELETE;";
    await journal.ExecuteNonQueryAsync();
}

Nothing is written to the evidence folder again. Analysis happens on a copy, and when the work is done the hash of the evidence is compared with the manifest. Who took which file and when is written to the incident log in UTC; this is called the chain of custody and it decides the value of the evidence in a legal process. Extending log retention is part of preserving evidence too: logs that would be deleted after 14 days must not be deleted before the incident is closed.

Deriving the Scope from Records

"Probably only Lakeside was affected" is not an answer. The list of affected records comes from the audit log, and this query is tested as part of the code too. The query counts only allowed rows: a denied request returned no data. It groups the rows where the caller's institution differs from the data's institution by the institution whose data was read and by the user who read it:

public static async Task<IReadOnlyList<CrossInstitutionRead>> CrossInstitutionReadsAsync(
    BookstoreContext db, DateTime fromUtc, DateTime toUtc, CancellationToken ct)
{
    List<AccessAuditEntry> rows = await db.AccessAuditEntries.AsNoTracking()
        .Where(e => e.Action == "institution.orders.list" && e.Outcome == "allowed"
            && e.CallerInstitutionId != e.ResourceInstitutionId
            && e.OccurredAtUtc >= fromUtc && e.OccurredAtUtc < toUtc)
        .OrderBy(e => e.OccurredAtUtc)
        .ToListAsync(ct);

    return
    [
        .. rows.GroupBy(e => (e.ResourceInstitutionId, e.UserId, e.CallerInstitutionId))
            .Select(g => new CrossInstitutionRead(
                g.Key.ResourceInstitutionId, g.Key.UserId, g.Key.CallerInstitutionId, g.Count(),
                g.First().OccurredAtUtc, g.Last().OccurredAtUtc,
                [.. g.SelectMany(e => e.ResourceIds.Split(',', StringSplitOptions.RemoveEmptyEntries)).Select(int.Parse).Distinct().Order()]))
            .OrderBy(r => r.ResourceInstitutionId).ThenBy(r => r.UserId)
    ];
}

The command runs against the copy taken for analysis, not against the evidence itself; afterwards the hash of the evidence is checked again with PowerShell:

Copy-Item incident/evidence/bookstore-evidence.db incident/analysis.db
dotnet run --project tools/Bookstore.IncidentTools -- affected incident/analysis.db 2026-11-11T00:00:00Z 2026-11-12T00:00:00Z
(Get-FileHash -Algorithm SHA256 incident/evidence/bookstore-evidence.db).Hash.ToLower()

victim  reader  reader-inst  requests  first (UTC)          last (UTC)           orders
     1  alan              2         2  2026-11-11 14:03:00  2026-11-11 14:04:00  101,102,103,104
     1  edsger            2         1  2026-11-11 15:41:00  2026-11-11 15:41:00  101,102,103,104
affected institutions: 1, affected orders: 4
57ca1efb32b48bb5f66ba418c42f2ab6413ed63d8b597289eca480e7aa4cd91b

The result lets you write a precise sentence: on 11 November, between 14:03 and 15:41, two users from Hillcrest University saw the list of four orders of Lakeside School. The hash is the same as in the manifest, so the analysis did not touch the evidence. One more thing needs to be written down just as clearly: the window that could not be checked. If audit records started with release 2, when the endpoint went live, there is nothing before that anyway; but if log retention had been shorter, the sentence "we cannot know anything before this date" would have to go into the text for the customer. The TraceId on each row lets you go to the trace of the same request if needed and see which other calls it made.

The query's test uses synthetic records with one row reading the caller's own institution, two cross-institution reads and one denied request, and verifies that the result contains only the cross-institution reads that really returned data, with the right order ids. A query you will run on the day of an incident should not run for the first time on that day.

The Fix and the Regression Test

The fix was deployed after the evidence was preserved. The rule is simple: the institution comes from the token, not from the URL. On a mismatch the request gets 404 and the attempt is written to the audit as denied; this also meets ASVS requirement v5.0.0-16.3.2, which asks for failed authorization attempts to be logged:

private static async Task<IResult> ListOrders(
    int institutionId, ClaimsPrincipal user, BookstoreContext db, AccessAudit audit, CancellationToken ct)
{
    if (!InstitutionalCaller.TryRead(user, out InstitutionalCaller? caller))
    {
        return TypedResults.Forbid();
    }
    // INC-2026-11-12: the institution in the route must be the caller's own; the token decides, not the URL.
    if (institutionId != caller.InstitutionId)
    {
        await audit.WriteAsync(caller, "institution.orders.list", institutionId, [], "denied", ct);
        return TypedResults.NotFound();
    }
    List<InstitutionOrderView> orders = await db.Orders.AsNoTracking()
        .Where(o => o.InstitutionId == institutionId)
        .OrderBy(o => o.Id)
        .Select(o => new InstitutionOrderView(o.Id, o.Status.ToString(), o.PlacedAtUtc, o.Total))
        .ToListAsync(ct);
    await audit.WriteAsync(caller, "institution.orders.list", institutionId, orders.Select(o => o.Id), "allowed", ct);
    return TypedResults.Ok(orders);
}

[Fact]
[Trait("incident", "INC-2026-11-12")]
[Trait("asvs", "v5.0.0-8.4.1")]
[Trait("asvs", "v5.0.0-16.3.2")]
public async Task Another_institutions_order_list_is_not_found_and_the_attempt_is_audited()
{
    HttpResponseMessage response = await factory.CreateClientAs(TestUsers.Alan).GetAsync("/api/institutions/1/orders", ct);

    Assert.Equal(HttpStatusCode.NotFound, response.StatusCode);
    AccessAuditEntry? entry = await LastAuditEntryAsync("alan");
    Assert.Equal(("denied", 2, 1), (entry?.Outcome, entry?.CallerInstitutionId, entry?.ResourceInstitutionId));
}

Test run summary: Passed!
  total: 28
  failed: 0
  succeeded: 28
  skipped: 0
  duration: 1s 035ms

The new tests show up in the requirements file too: v5.0.0-8.4.1, which asks multi-tenant applications for cross-tenant controls, was added to the list, and v5.0.0-16.3.2, planned since the third post, is now test. T13 was added to the threat model; its mitigation field notes "found by the incident, not by this model". The root cause points there too: the PR that added the endpoint did not tick the threat model box from the first post, and the new flow never made it into the model. What needs fixing is not only the code but the review habit that asks about that box.

Communication and Notification

The message to Lakeside School is clear because it comes from the records: what happened, which data (the number, status, date and total of four orders), who saw it (two users of another institution), when, what was done, and whom to contact with questions. Hillcrest University gets a separate message saying that its users accessed this data and asking for it to be deleted; the wording does not blame the users, because the records show a URL changed out of curiosity, not an attack.

Legal notification is not an engineering decision. In Bookstore the person responsible for personal data assesses whether the affected data is personal data; in this incident the returned fields were order information belonging to an institution and contained no personal data, but the confidentiality clause in the contract still required notifying the customer. Had personal data been affected, the 72-hour notification deadline from the KVKK Board's 2019 decision would have applied; its legal framework and preparation are in the data breach post of the Data Privacy and KVKK series. Engineering's job in this process is to inform the responsible person in the first hour and give them a precise list of affected records.

The sentence on the status page is kept short: "We are investigating unauthorised access to institutional order lists; we are contacting affected customers directly." Exploit steps, the URL pattern or people's names do not go on the status page, into tickets or into general channels. The same separation applies to the postmortem: the timeline, the root cause and the actions are shared with the team; names of individuals, raw evidence and exploit details of similar issues not yet fixed are not. Bookstore's incident record lives in the repo as docs/security/incidents/INC-2026-11-12.md and contains only this shareable part.

Runbook and Tabletop Exercise

None of the steps in this post should be thought up on the day of the incident. Bookstore's security runbook is a short document added to the general incident process; the first hour:

## First hour

1. Open a private incident channel; invite only who is needed. No details in public channels or tickets.
2. Name the roles: incident commander, technical lead, scribe, and a communications owner.
   Call the data protection contact (KVKK) and legal early: if personal data may be affected, the 72-hour
   clock for notifying the authority may already be running.
3. Preserve evidence before you change anything:
   `dotnet run --project tools/Bookstore.IncidentTools -- preserve <db> <evidence-dir>`,
   export the logs and traces of the time window, record who did what and when (UTC) in the incident log.
4. Contain without tipping off: prefer a quiet fix or a feature flag over deleting accounts or data.
   Do not rotate or delete anything that is evidence until it is preserved.

The way to put the runbook to the test is a tabletop exercise: the team sits around a table (or on a video call), reads a scenario card and answers in turn "whom do we call first, which command do we run first, what do we tell the customer?". No system changes; the goal is to find the gaps in the runbook. In Bookstore a 60-minute exercise is held every six months. The scenario in this post is now one of the cards in the runbook too.

In Production

The retention period of the audit log must be longer than the time it takes to notice an incident. This incident was noticed after a day; many leaks are noticed months later. Retention has to be balanced against the data minimisation principle of KVKK; writing identifiers rather than content into the audit record makes that balance easier.

The audit table should be somewhere the application itself cannot delete from. If the application's database user can only insert into the audit table, someone who takes over the application cannot erase their tracks. In production this is done with a separate schema permission or by streaming logs to a store outside the application; the details are in the post of the ASP.NET Core Security series that covers the audit trail.

Clocks must be in UTC and in sync across servers. Ordering records written in different time zones or with drifting clocks is one of the biggest time sinks on the day of an incident.

Common Pitfalls

Deleting the logs and saying "fixed" is the biggest mistake; it is usually done not out of bad intent but out of a "let's clean up" reflex. Deleted logs also delete the only way to determine who was affected, and turn every sentence the organisation can say into a guess.

The second pitfall is deciding who was affected by guessing. "Probably only one institution" and, the other way round, "let's play it safe and write to everyone" are both wrong: the first may leave out someone who was really affected, the second needlessly worries hundreds of customers who were not and dilutes the real message.

The third is starting the fix before preserving evidence, and the fourth is telling legal and the person responsible for personal data on the last day. Both save minutes and cost days.

A Small Experiment

★☆☆ Add a row to the data generated with demo-data where an institution reads its own data but the result is denied, and run the affected command again. Why does the output not change?

★★☆ Design an alert for every allowed audit row where the caller's institution differs from the data's institution: which query, how often, sent to whom? How much earlier would this alert have caught this incident?

★★★ Play the second of the three tabletop cards in the runbook (a leaked webhook secret) with your team. Compare this runbook with the steps in the leak response post of the Configuration and Secret Management series; are there overlapping or missing steps?

This post completes the series. In the first post we started with a DFD and four questions; in the second we found threats with STRIDE and turned them into tests, in the third we tied decisions to ASVS 5.0 requirements, in the fourth we talked about receiving reports from outside, in the fifth about pentest findings and in this post about an incident. What they all have in common is that security does not live in one tool or one person but in small, recorded habits the Bookstore team repeats every sprint. The root cause of this incident was not a missing tool either, but an unticked box.