Real-World Bug Hunts 1: Backend — Culture, Time Zones, Sockets and Mutable Keys

Version note: The code in this post was built and run with .NET 10, Entity Framework Core (EF Core) 10.0.14, xUnit v3 (xunit.v3 4.0.2) and Microsoft.Extensions.TimeProvider.Testing 10.11.0; container output comes from Docker 29.2 and the mcr.microsoft.com/dotnet/aspnet:10.0 image.

Guess First

An endpoint in the Bookstore API (Application Programming Interface) calls a method that emails an order confirmation, and that method is async void. The mail server returns an error. Does the user get 201 Created? Does the process stay up? And what happens when the same code runs in a container where the app is PID (process identifier) 1, the container's first process? One more: can two keys in a Dictionary be equal to each other according to Equals?

The last two posts of the series apply the method we built over the first eleven posts to real cases. This one covers six backend cases, all following the same template: symptom, first wrong hypothesis, narrowing, root cause, fix, and the test that proves it. I won't re-explain topics whose mechanics are covered in other series (the Turkish I, Dictionary key equality, async void, N+1); the focus here is how you find the bug in production.

Case 1: A Role Check That Breaks Under the Turkish Culture

Symptom: admin users routed to the newly added second server get 403 on the admin screen; the same user routed to the first server is fine. First wrong hypothesis: "The role claim in the token is corrupted, or the authorization cache is stale." The hypothesis collapses because the same token gives different results on two servers: same input, different output. The difference is not in the input but in the environment.

To narrow it down we went back to the environment-differences checklist from the second post in the series and added each server's CultureInfo.CurrentCulture to the health endpoint. The first server is en-US, the new one is tr-TR. On the code side the suspicious line turned up right away:

namespace Bookstore.Hunts;

public static class RoleCheck
{
    // buggy: uppercases according to the current culture
    public static bool IsAdminBuggy(string role) => role.ToUpper() == "ADMIN";

    public static bool IsAdmin(string role) =>
        string.Equals(role, "admin", StringComparison.OrdinalIgnoreCase);
}

Real output under three cultures:

en-US     "admin".ToUpper() = ADMIN buggy=True fixed=True
tr-TR     "admin".ToUpper() = ADMİN buggy=False fixed=True
invariant "admin".ToUpper() = ADMIN buggy=True fixed=True

Root cause: in Turkish the uppercase of "i" is the dotted "İ". The mechanics are in the C# Fundamentals post "Strings in C#: Interpolation, StringBuilder and the Turkish I Trap"; what matters here is that this bug shows up only on a machine with a different culture. The fix is to make the comparison culture-independent: for text that means something to the machine, such as a role name or a protocol value, use StringComparison.OrdinalIgnoreCase. The test runs the same check under three cultures:

[Theory]
[InlineData("en-US")]
[InlineData("tr-TR")]
[InlineData("")]
public void Admin_role_is_recognized_in_every_culture(string cultureName)
{
    CultureInfo original = CultureInfo.CurrentCulture;
    try
    {
        CultureInfo.CurrentCulture = new CultureInfo(cultureName);
        Assert.True(RoleCheck.IsAdmin("admin"));
    }
    finally
    {
        CultureInfo.CurrentCulture = original;
    }
}

I first pointed the test at the buggy method and ran it; the red result names exactly one culture:

failed Bookstore.Hunts.Tests.RoleCheckTests.Admin_role_is_recognized_in_every_culture(cultureName: "tr-TR") (1ms)
  Assert.True() Failure
  Expected: True
  Actual:   False

To close this class of bug for good we turned on the analyzers. According to Microsoft Learn, CA1311 ("Specify a culture or use an invariant version") is not enabled by default in .NET 10, and CA1862 is enabled only as a suggestion, so a normal build says nothing. After adding an .editorconfig to the project, the build reports two warnings (messages shortened):

[*.cs]
dotnet_diagnostic.CA1311.severity = warning
dotnet_diagnostic.CA1862.severity = warning

src/Bookstore.Hunts/RoleCheck.cs(5,53): warning CA1862: Prefer using 'string.Equals(string, StringComparison)' to perform a case-insensitive comparison, ...
src/Bookstore.Hunts/RoleCheck.cs(5,58): warning CA1311: Specify a culture or use an invariant version to avoid implicit dependency on current culture

Case 2: A Daily Sales Report Shifted by Three Hours

Symptom: the finance team says Bookstore's "yesterday's sales" report doesn't match the payment provider's end-of-day settlement. The difference shows up every day, but it's small. First wrong hypothesis: "The payment provider confirms some payments late, so they slip into the next day." To test it we listed the mismatched orders one by one. They all had one thing in common: they were placed between 00:00 and 03:00 Turkey time. Confirmation delays don't produce such a regular time window; three hours is the offset between UTC (Coordinated Universal Time) and Turkey.

Orders were written correctly with DateTime.UtcNow. The report's day boundaries, however, came from another era:

public static class DailySalesReportBuggy
{
    public static (DateTime From, DateTime To) Yesterday()
    {
        // buggy: the server's local midnight is compared against UTC values
        DateTime today = DateTime.Today;
        return (today.AddDays(-1), today);
    }
}

Root cause: DateTime carries no offset. A value read from the database has Kind set to Unspecified, and the >= comparison looks only at ticks. Comparing local midnight with UTC values means the report actually sums the "UTC day", that is, 03:00 to 03:00 Turkey time. I placed five orders at, in Turkey time, yesterday 01:10, 10:15 and 23:40 and today 00:20 and 02:50, then ran the report:

TZ=Europe/Istanbul DateTime.Now=2026-11-08 12:47 DateTime.UtcNow=2026-11-08 09:47
buggy: [2026-11-07 00:00, 2026-11-08 00:00) -> orders 2,3,4,5
fixed: [2026-11-06 21:00, 2026-11-07 21:00) -> orders 1,2,3

The buggy report misses yesterday's 01:10 order and books two of today's orders under yesterday. Running the same program with TZ=UTC didn't change the buggy range, because at that hour the date is the same in both time zones. If the job runs at 01:00 at night in a container whose time zone is UTC, though, DateTime.Today returns the day before Turkey's current day and the shift grows to a full day. That's why the test pins the hour at which the job really runs.

The fix has two parts. First, compute the day boundaries in the business rule's time zone and convert them to UTC; second, take the clock from outside through TimeProvider, so the test can set up any moment it wants:

public sealed class DailySalesReport(TimeProvider clock)
{
    private static readonly TimeZoneInfo Istanbul =
        TimeZoneInfo.FindSystemTimeZoneById("Europe/Istanbul");

    public (DateTime FromUtc, DateTime ToUtc) Yesterday()
    {
        DateTime todayLocal = TimeZoneInfo.ConvertTime(clock.GetUtcNow(), Istanbul).Date;
        DateTime fromLocal = todayLocal.AddDays(-1);
        return (ToUtc(fromLocal), ToUtc(todayLocal));
    }

    private static DateTime ToUtc(DateTime local) =>
        TimeZoneInfo.ConvertTimeToUtc(DateTime.SpecifyKind(local, DateTimeKind.Unspecified), Istanbul);
}

[Fact]
public void Yesterday_uses_istanbul_day_boundaries_when_job_runs_after_midnight()
{
    // the job runs at 01:00 on 8 November, Turkey time
    FakeTimeProvider clock = new(new DateTimeOffset(2026, 11, 7, 22, 0, 0, TimeSpan.Zero));
    DailySalesReport report = new(clock);

    (DateTime fromUtc, DateTime toUtc) = report.Yesterday();

    Assert.Equal(new DateTime(2026, 11, 6, 21, 0, 0, DateTimeKind.Utc), fromUtc);
    Assert.Equal(new DateTime(2026, 11, 7, 21, 0, 0, DateTimeKind.Utc), toUtc);
}

The other classic date shift, a date without a time zone in JSON (JavaScript Object Notation) moving by a day in the browser, is a case in the last post of the series.

Case 3: new HttpClient() and Socket Exhaustion

Symptom: on the afternoon of a sale day, calls to the stock service intermittently fail with SocketException. First wrong hypothesis: "The stock service can't handle the load." But the stock service's own metrics are calm; the error happens on our side, before a connection is established. Narrowing started with looking at the state of the TCP (Transmission Control Protocol) connections the process opens.

for (int i = 0; i < 1000; i++)
{
    // buggy: a new HttpClient and a new connection pool for every request
    using HttpClient client = new();
    await client.GetStringAsync("http://127.0.0.1:5501/stock/42");
}

I ran this loop against my own Kestrel and right after it counted the connections on that port with netstat. Then I repeated the same 1000 requests with a single shared client (waiting in between for the TIME_WAIT entries to clear):

$ ./demo sockets new
mode=new requests=1000 elapsed=243 ms
$ netstat -an -p tcp | grep '127.0.0.1.5501' | awk '{print $NF}' | sort | uniq -c
1000 TIME_WAIT

$ ./demo sockets shared
mode=shared requests=1000 elapsed=120 ms
$ netstat -an -p tcp | grep '127.0.0.1.5501' | awk '{print $NF}' | sort | uniq -c
   1 TIME_WAIT

Root cause: every HttpClient comes with its own handler and connection pool; when it's disposed the connection closes, and the side that closes the connection keeps that port in TIME_WAIT for a while. On this Mac net.inet.tcp.msl is 15000 ms; when I measured, the TIME_WAIT entries disappeared between 30 and 35 seconds, roughly two MSLs (Maximum Segment Lifetime). The ephemeral port range is 49152–65535, which is 16,384 ports. Back-of-the-envelope: open more than 16,384 / 30 ≈ 546 new connections per second to the same target and you run out of ports. That's exactly what happened on the sale day.

The fix is one of the two options in Microsoft's HttpClient guidelines: either a long-lived (static or singleton) client with PooledConnectionLifetime, or short-lived clients produced by IHttpClientFactory. The first looks like this:

// a single instance; connections are recycled every 5 minutes, so DNS changes are picked up too
HttpClient shared = new(new SocketsHttpHandler { PooledConnectionLifetime = TimeSpan.FromMinutes(5) });

The PooledConnectionLifetime detail matters: the same guidelines say HttpClient resolves the DNS (Domain Name System) entry only when a connection is created and ignores the TTL (time to live). If you apply the "always use the same client" fix without this setting, you trade socket exhaustion for a new bug where the client keeps going to the old address after the service's IP (Internet Protocol) address changes. For this case, catching new HttpClient() calls at build time is a more effective regression check than a unit test: you can ban that constructor with BannedApiAnalyzers, which we used in the eleventh post of the series (I didn't run that step here).

Case 4: The Key That Got Lost in a Dictionary

Symptom: in some orders the same book appears as two separate lines, each with quantity 1. First wrong hypothesis: "The frontend sends the same request twice." The network log showed a single request. We traced the cart code step by step:

public sealed record MutableCartLineKey
{
    public required string Isbn { get; set; }
    public required string Format { get; set; }
}

Dictionary<MutableCartLineKey, int> lines = [];
MutableCartLineKey added = new() { Isbn = "978-0-13-468599-1", Format = "paperback" };
lines[added] = 1;

// buggy: the key is mutated while it sits in the Dictionary (ISBN normalization)
added.Isbn = added.Isbn.Replace("-", "");
Console.WriteLine($"ContainsKey(added) = {lines.ContainsKey(added)}");
Console.WriteLine($"Keys.Any(k => k.Equals(added)) = {lines.Keys.Any(k => k.Equals(added))}");

MutableCartLineKey scanned = new() { Isbn = "9780134685991", Format = "paperback" };
if (lines.TryGetValue(scanned, out int quantity))
{
    lines[scanned] = quantity + 1;
}
else
{
    lines[scanned] = 1;
}

Console.WriteLine($"lines.Count = {lines.Count}");
Console.WriteLine($"keys equal = {lines.Keys.First().Equals(lines.Keys.Last())}");

ContainsKey(added) = False
Keys.Any(k => k.Equals(added)) = True
lines.Count = 2
keys equal = True

The first two lines are this case's diagnostic technique: the hash lookup (ContainsKey) can't find the key, the linear search (Keys.Any) can. That pair is the shortest way to say "the key changed after it was added to the Dictionary". Careful: lines.Keys.Contains(added) is useless here, because KeyCollection.Contains calls ContainsKey internally too. The last line shows a Dictionary holding two keys that are equal to each other, a state that should be impossible. The ISBN (International Standard Book Number) normalization ran after the key's hash was computed, and the key stayed in the wrong bucket.

The root cause is combining a record's value equality with properties that have a set. The mechanics are in the C# Collections post "C# Dictionary and HashSet: TryGetValue, Key Equality and Comparers". The fix is to make the key immutable and to normalize in exactly one place, when the key is created:

public readonly record struct CartLineKey
{
    public string Isbn { get; }
    public string Format { get; }

    private CartLineKey(string isbn, string format) => (Isbn, Format) = (isbn, format);

    public static CartLineKey Create(string isbn, string format) =>
        new(isbn.Replace("-", "").Trim(), format.Trim().ToLowerInvariant());
}

[Fact]
public void Same_book_scanned_with_and_without_hyphens_is_one_line()
{
    Dictionary<CartLineKey, int> lines = [];
    lines[CartLineKey.Create("978-0-13-468599-1", "paperback")] = 1;
    CartLineKey scanned = CartLineKey.Create("9780134685991", "Paperback");

    lines[scanned] = lines.GetValueOrDefault(scanned) + 1;

    Assert.Single(lines);
    Assert.Equal(2, lines[scanned]);
}

Case 5: The async void That Crashed the Process

Symptom: the API process restarts a few times a day. The global exception handler's log has nothing, and the last log line each time is a successful order request. First wrong hypothesis: "There's a memory leak and the container is being killed for OOM (Out Of Memory)." The cheap way to test that is the exit code: the OOM killer kills the process with SIGKILL, the exit code is 137, and Docker reports OOMKilled=true. Ours was 134, that is SIGABRT: the process had terminated itself.

public sealed class OrderPlacedNotifier
{
    // buggy: async void, there is no caller to catch the exception
    public async void OnOrderPlacedBuggy(int orderId)
    {
        await Task.Delay(100);
        throw new SmtpException($"Mailbox unavailable while sending confirmation for order {orderId}");
    }
}

app.MapPost("/orders/{id:int}", (int id, OrderPlacedNotifier notifier) =>
{
    notifier.OnOrderPlacedBuggy(id);
    return Results.Created($"/orders/{id}", new { id });
});

On the client side everything looks fine:

$ curl -s -i -X POST http://127.0.0.1:5502/orders/1042
HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Date: Sun, 08 Nov 2026 09:51:58 GMT
Server: Kestrel
Location: /orders/1042
Transfer-Encoding: chunked

{"id":1042}

Meanwhile, 100 ms later, this lands on the process's stderr (file path shortened):

Unhandled exception. System.Net.Mail.SmtpException: Mailbox unavailable while sending confirmation for order 1042
   at Bookstore.Hunts.OrderPlacedNotifier.OnOrderPlacedBuggy(Int32 orderId) in .../src/Bookstore.Hunts/OrderPlacedNotifier.cs:line 10
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__124_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
exit code: 134

The bottom of the stack trace is a signature: Task.<>c.<ThrowAsync> and QueueUserWorkItemCallback. The exception wasn't thrown in a request pipeline but in a work item queued to the thread pool; since ASP.NET Core has no SynchronizationContext, an async void method's exception goes there and terminates the process. The middleware and UseExceptionHandler never hear about it. The mechanics are in the C# Async post "C# async void and Fire-and-Forget: Lost Exceptions".

In a container a surprising detail came up. I ran the same app in the aspnet:10.0 image with dotnet directly as PID 1. The exception was printed, but the container didn't exit:

$ docker inspect --format 'Status={{.State.Status}} ExitCode={{.State.ExitCode}}' debugwork-crash-12
Status=running ExitCode=0
$ curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:5503/orders/1043
201
$ docker stats --no-stream --format '{{.CPUPerc}}' debugwork-crash-12
100.08%

After printing "Unhandled exception" the process stayed up, burning one CPU (Central Processing Unit) core at 100%, and kept answering new requests with 201. According to Linux's kill(2) and pid_namespaces(7) man pages, PID 1 only receives signals for which it has installed a handler; what I saw is consistent with SIGABRT failing to terminate the process for that reason. When I started the same container with docker run --init (so that PID 1 is a small init process), the result was back to what you'd expect:

Status=exited ExitCode=134 OOMKilled=false

I observed this in a Linux container on Docker Desktop; if your app is PID 1 in your own orchestrator, run the same experiment there. A half-dead process that passes its health check is much harder to find than one that crashes cleanly.

The fix: the method becomes async Task and is wired to something that can observe the exception; work that shouldn't affect the request's outcome, like email, goes to a background queue where it's logged inside a try/catch. The test checks that the exception is now visible to the caller:

[Fact]
public async Task Notification_failure_is_observable_by_the_caller()
{
    OrderPlacedNotifier notifier = new();

    await Assert.ThrowsAsync<SmtpException>(() => notifier.OnOrderPlacedAsync(1042));
}

To catch this class of bug at build time I added the Microsoft.VisualStudio.Threading.Analyzers 18.12.1 package; the build produced this warning (URL part shortened):

src/Bookstore.Hunts/OrderPlacedNotifier.cs(7,23): warning VSTHRD100: Avoid "async void" methods, because any exceptions not handled by the method will crash the process

Case 6: N+1 and Tracking-Driven Slowness in EF Core

Symptom: the order list in the admin panel gets slower as the number of orders grows. First wrong hypothesis: "An index is missing on the Orders table." But the database's slow query log doesn't contain a single slow query. To narrow it down we changed the question: if the queries aren't slow, how many are there? Using EF Core's LogTo we counted only the executed commands:

public static async Task<List<OrderListItem>> ListBuggyAsync(BookstoreDbContext db) =>
    (await db.Orders.ToListAsync())
        // buggy: the lazy loading proxy sends a separate query per Customer
        .Select(o => new OrderListItem(o.Id, o.Customer.Name, o.Total))
        .ToList();

public static Task<List<OrderListItem>> ListAsync(BookstoreDbContext db) =>
    db.Orders
        .AsNoTracking()
        .Select(o => new OrderListItem(o.Id, o.Customer.Name, o.Total))
        .ToListAsync();

With 2000 orders and 400 customers in an in-memory SQLite database, measured after a warm-up round:

buggy: rows=2000 sql commands=401 tracked=2400 elapsed=34 ms
fixed: rows=2000 sql commands=1 tracked=0 elapsed=1 ms

401 queries: one for the orders and 400, one per customer. N here isn't the number of orders but the number of distinct customers; thanks to identity resolution the same customer isn't queried twice. That's why a test environment with 3 customers shows "only 4 queries" and nobody notices. The change tracker has also accumulated 2400 entities, an unnecessary cost for a read-only list. In-memory SQLite has no network latency; on a real database server every round trip adds network latency, and 401 round trips go from a few milliseconds to seconds. The mechanics are in the EF Core post "EF Core Performance Basics: N+1, Projection, AsNoTracking and Compiled Queries".

The fix is projection: produce the DTO (Data Transfer Object) with a single JOIN on the SQL (Structured Query Language) side and turn off tracking. The regression test looks at the number of queries, not the duration; duration varies by machine, the query count doesn't:

[Fact]
public async Task Order_list_runs_a_single_sql_command()
{
    // ... two customers and two orders seeded, the commands counter is incremented via LogTo
    using BookstoreDbContext db = new(options);
    commands = 0;
    List<OrderListItem> list = await OrderQueries.ListAsync(db);

    Assert.Equal(2, list.Count);
    Assert.Equal(1, commands);
}

All the tests together:

Test run summary: Passed!
  total: 7
  failed: 0
  succeeded: 7
  skipped: 0
  duration: 1s 317ms

In Production

In five of the six cases the first hypothesis said the bug lived in another system: the token, the payment provider, the stock service, the frontend, the database. Each time, a cheap observation refuted it: the same token on two servers, the time distribution of the mismatched orders, netstat output, a single network request, an exit code, a query count. In production, eliminating hypotheses with one- or two-minute observations like these before reaching for expensive tools saves hours.

Make environment information visible. Culture, time zone, runtime version and which instance answered should sit on a health or diagnostics endpoint (behind authorization). In Cases 1 and 2 the path to the root cause started with exactly this information. In containers, set the TZ and LANG variables explicitly in the Dockerfile; don't rely on the image's defaults.

Watch exit codes and restart counts. Telling 134 from 137 answers "memory or exception?" at a glance. To capture the moment the process crashes, the automatic crash dump settings from the ninth post of the series apply here too. If your app is PID 1 in a container, use --init or your orchestrator's equivalent.

Compare counters by count, not by duration: connection count, TIME_WAIT count, query count, tracked entity count. Duration is noisy and changes under load; a count describes the code's behavior directly and can become a test.

Common Pitfalls

Fixing the bug without writing its test. In all six cases the fix was one line; the valuable part is the test and the analyzer rule that keep the same mistake from coming back. The three levels from the eleventh post of the series (hotfix, regression test, a change that makes this class of bug impossible) become concrete here: OrdinalIgnoreCase, a regression test and CA1311.

Fixing it with a second bug. Making HttpClient static and forgetting PooledConnectionLifetime brings in a DNS bug; replacing DateTime.Now with DateTime.UtcNow without computing the day boundary in the business rule's time zone shifts the report to the UTC day instead.

Trusting the test environment's small data. N+1 is 4 queries with 3 customers; socket exhaustion never shows with 10 requests; a time zone bug never turns red in a test that doesn't run after midnight. Pick test inputs based on production's boundary conditions.

Not reading the bottom of the stack trace. In Case 5 the answer wasn't in the top line but in the ThrowAsync and QueueUserWorkItemCallback frames below it. As we did when reading async frames in the fourth post of the series, don't close a stack trace after looking only at its first line.

A Small Experiment

★☆☆ Add the .editorconfig from Case 1 to your own project and build. How many CA1311 warnings come out? Classify each one as "for the machine or for a human": the machine ones get Ordinal/ToUpperInvariant, the ones shown to users get an explicit culture.

★★☆ Rewrite the test from Case 2 for a moment when the job runs at 23:30 Turkey time. Does the result change? Then change the buggy version to take a TimeProvider, try the same two moments with it, and work out how many orders shift at which hour.

★★★ Reproduce Case 3 on your own machine: change the loop to send a fixed number of requests per second instead of 1000 in a row, and measure the value at which the TIME_WAIT count levels off. Check whether your number matches "request rate × TIME_WAIT duration".

All the cases in this post happened inside the server. In the last post of the series the scene moves to the boundary between the browser and the server: a white screen after deploy, a CORS (Cross-Origin Resource Sharing) preflight that fails only in production, a birth date that shifts by a day, floating point in money calculations and memory that grows from an observable that's never unsubscribed; and at the end, the series' one-page debugging checklist.