Working with Penetration Testers: Scope, Rules of Engagement, Reading Findings, Retesting and After the Report

Version note: The code in this post was built and run with .NET 10; the tests ran with xUnit v3 4.0.3. The pentest described is a scenario: the findings were reproduced on a local copy of Bookstore on 127.0.0.1, and the regression tests and logs are real output.

Guess First

Bookstore, the online bookstore used as the shared example across this site's series, is about to sell Institutional Orders to Lakeside School. The school's procurement officer, Nevin Hanım, wants one thing before signing: the report of a pentest done by an independent company. The team's first reaction: "The pipeline already runs DAST with ZAP; let's send that report."

Guess: what can a pentest find in an API that passes ZAP cleanly every night? In this post's scenario the pentest brought four findings; two were real and important, one was informational and one was a false positive. No scanner could have found either of the real two, because both answered "is this business rule enough?", not "is this request valid?". In this post we talk about preparing for a pentest, writing the scope, reading the report and turning findings into tests.

Pentest, DAST, Bug Bounty and Red Team

All four rest on the idea of "someone looking from outside", but they answer different questions:

DAST with ZAP was covered in the sixth post of the DevSecOps series; we talked about bug bounties in the previous post. A red team tests the organisation, not the application: it checks whether the attack was noticed, and for a small team it is usually an early step. As for when a pentest is needed, Bookstore has three triggers: a contractual requirement from an institutional customer, an audit, and a big release that changes trust boundaries. Institutional Orders hit all three.

Pentest companies usually refer to public methodologies: OWASP's Web Security Testing Guide (the latest stable release is 4.2 from December 2020; 5.0 is in progress), NIST's SP 800-115 from September 2008, which is still current, or PTES. Asking which methodology they will use when you collect quotes makes it easier to understand, when you read the report, what was tested and what was not.

Scope and Rules of Engagement

The quality of a pentest is decided before the test starts, by the document that gets signed. In Bookstore the rules of engagement live in the repo under docs/security/pentest-2026-11/. The scope part:

| In scope | Notes |
|---|---|
| `https://staging-api.bookstore.example.com` | Bookstore.Api, same release as production |
| Institutional purchase requests, approval and ordering | Main focus |
| Customer checkout up to the payment redirect | |

| Out of scope | Why |
|---|---|
| Production (`bookstore.example.com`) | Real customer data and real orders |
| Payment provider's hosted page | Third party; needs the provider's own permission |
| Shipping carrier API | Third party |
| Denial-of-service by volume, social engineering, physical access | Not part of this engagement |

A few decisions need attention. The test environment runs the same release as production but holds only synthetic data; it is not a copy of production. The payment provider's page is out of scope because that system is not ours; testing it needs the provider's own permission. Test accounts were prepared per role, and the list deliberately includes an account with both the InstitutionBuyer and the InstitutionApprover role: the testers need it to test the four-eyes rule from the second post. Account passwords go through the company's secure channel, not by e-mail, and are reset after the test.

Rate limiting is a fine balance. Turn the limits off completely and a missing rate limit cannot be found; leave them on and the test hits a wall every minute. Bookstore's decision: limits stay on, the testers' source addresses get a higher limit on staging only, and testing with a lower limit is possible on request. The most important part of the document, however, is the "stop and report immediately" list: if real personal or payment data is seen on staging, if a critical finding such as access to another institution's orders turns up, or if the environment becomes unstable, the test stops and Bookstore gets a phone call the same day.

The Report the Institutional Customer Wants

Nevin Hanım's team will not read the whole report; three things matter to the school. An executive summary: who did the test, on which dates, with which scope, and the overall result. The list of findings with the final status of each. And, after the retest, a short letter signed by the company: "these findings were fixed and verified, these were accepted". For the Bookstore team the real value is in the detailed part: reproduction steps and a CVSS 4.0 score for every finding. Sharing the detailed report with the customer under a non-disclosure agreement, or not sharing it at all, is common practice; the executive summary and the retest letter are enough for most customers.

Reading Findings: Severity Disputes, False Positives and Root Causes

When the report arrived, Emre moved the findings into a table. The report's severity and the team's severity sit in separate columns; when they differ, the reason is written down:

| Id | Report severity | Our severity | Finding | Root cause group | Decision | Regression test | Retest |
|---|---|---|---|---|---|---|---|
| F-01 | Medium | Medium | Splitting a large purchase into requests under the unit limit avoids approval | Documented limit not enforced (ASVS 2.1.3 / 2.3.2) | Fix: rolling 24 h unit total | `PentestRegressionTests` F-01 | Passed |
| F-02 | High | Medium | No limit on `POST /api/orders`; scripted orders reserve stock | Documented limit not enforced (ASVS 2.4.1) | Fix: fixed-window rate limit per client | `PentestRegressionTests` F-02 | Passed |
| F-03 | Info | Info | `Server: Kestrel` header reveals the web server | None (no exploit path) | Accept; listed as out of scope in SECURITY.md | None | n/a |
| F-04 | High | False positive | Possible SQL injection in `GET /api/books?search=` (response time differs with `'`) | None: EF Core parameters; `%` and `_` are escaped | Dispute with evidence (query log + test) | `PentestRegressionTests` F-04 | Accepted by tester |

For F-02 the report says High and the team says Medium. The dispute does not claim the finding is not real; it argues about the impact. Scripted orders use up stock reservations, but they leak no data, cost no money, and the reservations can be cancelled. This is the same as threat T03 from the second post, which was also scored Medium there. Different severity, same decision: it gets fixed. A severity discussion sets the priority, not whether to fix.

F-04, on the other hand, reports a suspected SQL injection: adding a single quote to the search parameter changes the response time. Disputing such a finding needs evidence; "we use EF Core" is not enough. The Bookstore team ran the API locally with EF Core command logging and sent the same request:

curl -s "http://127.0.0.1:52865/api/books?search=austen%27%20OR%20%271%27%3D%271"

{"items":[],"total":0,"page":1,"pageSize":12}

info: Microsoft.EntityFrameworkCore.Database.Command[20101]
      Executed DbCommand (1ms) [Parameters=[@pattern='?' (Size = 19), @term='?' (Size = 17)], CommandType='Text', CommandTimeout='30']
      SELECT COUNT(*)
      FROM "Books" AS "b"
      INNER JOIN "Authors" AS "a" ON "b"."AuthorId" = "a"."Id"
      WHERE "b"."Title" LIKE @pattern ESCAPE '\' OR "a"."Name" LIKE @pattern ESCAPE '\' OR "b"."Isbn" = @term

The input never enters the SQL text; it travels as the @pattern and @term parameters, and the difference in response time comes from the difference in the number of results. Still, the log was a reminder of one more thing: because the search uses LIKE, if the % and _ characters were not escaped, a user could pull the whole catalog in one request with a wildcard. In Bookstore these characters are already escaped; to make that permanent, the finding was closed with a regression test that searches for SQL syntax and a wildcard as plain text. Seeing the evidence, the tester accepted the finding as a false positive. Injection itself is covered in the injection post of the Web Security (OWASP Top 10) series.

Grouping findings by root cause before fixing them one by one is how you find copies of the same mistake elsewhere. F-01 and F-02 share a root cause: a limit written in docs/security/authorization-rules.md is not enforced in the code. So besides the fix, another task was opened: find the code and the test for every limit in the document. The story of F-01 is instructive too: it is threat T12, which we accepted on purpose in the second post. The acceptance rested on the assumption that it "shows up in the daily report"; the pentest report showed that the institutional customer did not find that assumption enough, and Selin changed the decision. An accepted risk is a risk the customer will see too.

A Regression Test for Every Finding

A finding is not closed until a test that reproduces it is written. The tests are tagged with the finding number, so before the retest dotnet test --filter-trait "pentest=F-01" runs only that finding's tests. For F-01 the test came first. In the same unit, if two requests that are each under the limit pass it within 24 hours, the second one must wait for approval:

[Fact]
[Trait("pentest", "F-01")]
[Trait("threat", "T12")]
[Trait("asvs", "v5.0.0-2.3.2")]
public async Task Requests_split_under_the_limit_need_approval_once_the_unit_total_passes_it()
{
    // Science Department of Lakeside School: limit 5,000.00 TRY
    HttpClient margaret = factory.CreateClientAs(new TestUser("margaret", 1, 3, "InstitutionBuyer"));

    PurchaseRequestView first = await CreateAsync(margaret, Frankenstein, 20);    // 3,958.20
    PurchaseRequestView second = await CreateAsync(margaret, Frankenstein, 20);   // 7,916.40 in 24 hours

    Assert.Equal("Approved", first.Status);
    Assert.Equal("PendingApproval", second.Status);
}

failed Bookstore.Api.Tests.PentestRegressionTests.Requests_split_under_the_limit_need_approval_once_the_unit_total_passes_it (219ms)
  Assert.Equal() Failure: Strings differ
  Expected: "PendingApproval"
  Actual:   "Approved"
Test run summary: Failed!
  total: 1
  failed: 1
  succeeded: 0

The fix moves the approval rule from the total of a single request to the total of the unit's auto-approved requests in the last 24 hours. Because SQLite cannot sum decimal values in SQL through EF Core, the totals are pulled into a list and added up in memory; the list is small. On SQL Server SumAsync would do:

private async Task ApplyApprovalRuleAsync(PurchaseRequest request, CancellationToken ct)
{
    InstitutionUnit unit = await db.InstitutionUnits.SingleAsync(u => u.Id == request.UnitId, ct);
    // Pentest F-01: the limit applies to the unit's requests approved by the limit in the last 24 hours, not to one request.
    DateTime since = clock.GetUtcNow().UtcDateTime.AddHours(-24);
    List<decimal> recentTotals = await db.PurchaseRequests
        .Where(r => r.UnitId == request.UnitId && r.Id != request.Id
            && r.ApprovedBy == AutoApprover && r.CreatedAtUtc >= since)
        .Select(r => r.Total)
        .ToListAsync(ct);   // SQLite cannot SUM decimals in SQL; the list is small
    bool needsApproval = recentTotals.Sum() + request.Total > unit.ApprovalLimit;
    request.Status = needsApproval ? PurchaseRequestStatus.PendingApproval : PurchaseRequestStatus.Approved;
    request.ApprovedBy = needsApproval ? null : AutoApprover;
    request.ApprovedAtUtc = needsApproval ? null : clock.GetUtcNow().UtcDateTime;
    request.ApprovedLinesHash = needsApproval ? null : request.ComputeLinesHash();
}

For F-02, ASP.NET Core's rate limiter was attached to the order endpoint. Algorithms and choosing partition keys are covered in detail in the rate limiting post of the ASP.NET Core Security series; here 10 requests per minute per client address are enough:

// Pentest F-02 / ASVS v5.0.0-2.4.1: placing orders is limited per client address.
int placeOrderLimit = builder.Configuration.GetValue("RateLimiting:PlaceOrder:PermitLimit", 10);
builder.Services.AddRateLimiter(options =>
{
    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    options.AddPolicy(OrderingModule.PlaceOrderPolicy, context => RateLimitPartition.GetFixedWindowLimiter(
        context.Connection.RemoteIpAddress?.ToString() ?? "unknown",
        _ => new FixedWindowRateLimiterOptions { PermitLimit = placeOrderLimit, Window = TimeSpan.FromMinutes(1) }));
});

Its test runs on a separate host that lowers the limit to 3; the fourth request must get 429. That matters so the other tests do not share the same limit:

[Fact]
[Trait("pentest", "F-02")]
[Trait("threat", "T03")]
[Trait("asvs", "v5.0.0-2.4.1")]
public async Task Placing_orders_is_rate_limited_per_client()
{
    HttpClient client = factory
        .WithWebHostBuilder(b => b.UseSetting("RateLimiting:PlaceOrder:PermitLimit", "3"))
        .CreateClient();

    List<HttpStatusCode> codes = [];
    for (int i = 0; i < 4; i++)
    {
        HttpResponseMessage response = await client.PostAsJsonAsync("/api/orders",
            new PlaceOrderRequest(Guid.NewGuid(), "Ada Lovelace", "ada@example.com"), ct);
        codes.Add(response.StatusCode);
    }

    Assert.Equal([HttpStatusCode.NotFound, HttpStatusCode.NotFound, HttpStatusCode.NotFound, HttpStatusCode.TooManyRequests], codes);
}

v5.0.0-2.4.1 was planned in the third post; it is now test in the requirements file, and the traceability test confirms it. The result of the whole suite:

Test run summary: Passed!
  total: 25
  failed: 0
  succeeded: 25
  skipped: 0
  duration: 1s 135ms

Continuous regression testing for authorization findings and OpenAPI-based scanning are covered in the last post of the API Security series; the tests here are the kind that go into that setup.

Retest and After the Report

The retest happens after the fixes reach production, within the 30 days written in the rules of engagement. The tester tries every finding again with their own steps; because Bookstore's regression tests are an automated copy of those steps, the retest brings no surprises. The letter signed after the retest goes to Nevin Hanım.

Once the report is closed, two jobs remain, and both are the ones skipped most. The first is updating the threat model: T03 and T12 are now Mitigated, their mitigation fields name the tests, and row R-02 in the risk register is closed. To check whether the model is current:

jq -r '[.detail.diagrams[].cells[].data.threats[]?] | group_by(.status)[] | "\(.[0].status)\t\(length)"' threat-model/bookstore-checkout.json

Mitigated	7
Open	5

The second is going back to the process: for everything the pentest found, ask "why didn't we find this?". F-01 was in the model but accepted; F-02 was in the model but waiting in the backlog. Both were visible inside the process; what was missing was prioritisation. This is what the root cause analysis and process improvement practices of NIST's SSDF look like in a small team.

In Production

During the pentest window, staging logs and alerts look different from normal: unauthorised access attempts, malformed input, 429s. So that these signals are not confused with an incident, the test window is announced to the on-call team in advance and the testers' source addresses are labelled in the alert rules. Labelling is not silencing; a real attack the tester never saw could happen in the same window.

When the test ends, test account passwords are reset, accounts are disabled if no longer needed, and every access given to the testers is revoked. If staging does not use a separate identity provider from production, this step is even more critical.

A pentest is a snapshot. Every data flow that changes in the first release after the report date falls outside the report's scope. That is why in Bookstore the pentest is tied to a yearly cycle and to big releases; the time in between is covered by the threat model, regression tests and pipeline scans.

Common Pitfalls

Testing production without notice is the most dangerous mistake. A test against production that nobody was told about can leave the team fighting what looks like a real attack, expose real customer data, or even disrupt the service; that is why rules of engagement exist. If the tester asks "shall we try production too?", the answer is a separate, written permission.

The second pitfall is closing only the "critical" findings and archiving the report. Medium and Low findings often come back unchanged in the next pentest and show the customer that the team does not care about findings. Every finding is either fixed or accepted with a written decision; there is nothing in between.

The third is disputing every finding. A dispute without evidence costs trust; a dispute with evidence, like F-04, helps the tester too. The fourth is treating the pentest report as the security program itself: the report is a measurement of how well the team's own processes work.

A Small Experiment

★☆☆ Change the decision for F-03: find the setting that turns off Kestrel's Server header, apply it and write a test that verifies the header is gone. Does this change the finding's severity?

★★☆ For every limit in docs/security/authorization-rules.md, find its code and its test. Is any limit left without a test? Turn this sweep, suggested by the root cause group, into a checklist.

★★★ Write the rules of engagement for Bookstore's next pentest: this time the scope is back office. Which test accounts are needed, which data must be synthetic, and what should the stop conditions be?

A pentest is a controlled test we run so that we find what we want found. In the last post of the series we talk about what is outside our control: a security incident in production where one institution's orders are read by another, and responding to it through detection, scoping, preserving evidence, communication and notification.