Race Conditions vs Data Races: Check-Then-Act, Read-Modify-Write and Atomicity

Version note: The code in this post was built and run with .NET 10 (C# 14); tests use xUnit v3 4.0.3. The outputs were taken on ARM64; statements about x64 are based on the .NET memory model spec and Intel's documentation.

Guess First

Bookstore, the online bookstore used as the shared example across this site's series, has a counter for the number of books sold: _sold++. Eight threads increment this counter a million times each. What is the result? Something close to eight million, or much less?

Second question: Dune has 100 copies in stock. Eight threads try fifty reservations each at the same time, and the code checks if (Available > 0) every time. In the worst case, how many books are sold? If we write the same code with a ConcurrentDictionary, is the problem gone? Last question: if one thread writes a decimal price while another reads it, can the reader see a value that was never written? In the previous post we used a simulation to see an increment lost in eighteen of twenty orders; in this post we produce the same bugs with real threads and give them names.

Two Definitions: Data Race and Race Condition

These two terms are often confused, but they describe different things. A data race is a technical definition about memory: two threads access the same memory location with no synchronization between them, and at least one of them writes. By synchronization we mean mechanisms that establish an order between accesses, such as a lock, Interlocked or Volatile. You can detect a data race by looking at the code itself, without knowing the result.

A race condition is about correctness: whether the program works correctly depends on the timing or the order of events. So a race condition is a semantic bug; to detect it you need to know what the program is supposed to do. That is also the main idea of John Regehr's 2011 post "Race Condition vs. Data Race": the two overlap, but one can exist without the other. In this post we see examples in both directions.

The distinction has different consequences in different languages. In C++ a data race is undefined behavior: the compiler may do anything. .NET is gentler: the memory model spec says a read sees a value that was written to that location; it does not make up a value from nothing. That does not make data races harmless; which value is seen is undefined, and as we will see below, a value can even be split. The details of the memory model are the subject of the sixth post of the series.

Atomicity Violation: count++ Is Three Steps

The first bug class is the atomicity violation: an operation we think of as one step is actually several, and another thread gets in between. The best-known example is read-modify-write:

public sealed class SoldCounter
{
    private int _sold;

    public int Sold => _sold;

    // Data race: read, add and write are three separate steps
    public void RecordSale() => _sold++;

    public void RecordSaleAtomic() => Interlocked.Increment(ref _sold);
}

In the experiments I use a small helper, RaceRunner.RunTogether, that releases the threads at the same moment with a Barrier, so the work really overlaps:

dotnet run -c Release --project src/Bookstore.ConcurrencyLab -- lost-update 8 1000000

expected=8,000,000 plain=1,451,805 atomic=8,000,000

Only 1.45 million of the eight million increments survived; 82% were lost. With two threads, 1,099,348 of two million remained. When _sold++ is compiled, it becomes separate machine instructions that read the value into a register, add one and write it back. If two cores read the same old value, both write the same new value and a sale is lost. This is not specific to ARM64: according to Intel's software developer's manual, on x64 too an increment instruction that operates on memory is not atomic with respect to other processors without the lock prefix. Interlocked.Increment turns the three steps into one with the hardware's atomic instruction, and the result is exactly eight million every time.

Check-Then-Act: The Stock Check

The second class is check-then-act: looking at a condition and acting on it. Between the check and the action another thread can change the state, and the action then relies on a stale check:

public sealed class StockItem(string isbn, int available)
{
    public string Isbn { get; } = isbn;
    public int Available { get; private set; } = available;
    private readonly Lock _gate = new();

    // Check-then-act: the check may be stale when the act runs
    public bool TryReserve()
    {
        if (Available > 0)
        {
            Available--;
            return true;
        }
        return false;
    }

    // Check and act in the same critical section
    public bool TryReserveLocked()
    {
        lock (_gate)
        {
            if (Available > 0)
            {
                Available--;
                return true;
            }
            return false;
        }
    }
}

dotnet run -c Release --project src/Bookstore.ConcurrencyLab -- oversell 2000

rounds=2000 StockItem oversold in 372 rounds (worst: 350 sold of 100)
rounds=2000 ConcurrentDictionary oversold in 1403 rounds

In 372 of two thousand rounds more than 100 books were sold; 350 in the worst round. In a second run I saw 407 rounds and a worst case of 400, meaning every attempted reservation succeeded. Two bugs stack up here: two threads can both see Available > 0 at the same time (check-then-act), and because Available-- is itself a read-modify-write, decrements are lost too (atomicity violation). In the locked version the check and the action sit in a single critical section, so none of the two hundred rounds in the tests oversells.

A Race Condition Without a Data Race

Now let's try Ayşe's suggestion from the team: keep the stock in a ConcurrentDictionary, then it is thread-safe:

// Every single call is thread-safe; the sequence of calls is not
public bool TryReserve(string isbn)
{
    if (_available[isbn] > 0)
    {
        _available[isbn] = _available[isbn] - 1;
        return true;
    }
    return false;
}

This code has no data race: every read and write goes through the ConcurrentDictionary's own synchronization. Yet according to the output above, 1403 of two thousand rounds oversold, worse than the first version. The reason is that other threads get in between three separate calls, each safe on its own. The race condition lives at the level of the contract: the rule "stock never drops below zero" is protected not by a single call but by the sequence of calls.

The fix is to reduce the check and the update to a single atomic step. TryUpdate changes the value only if the old value we expect is still there; if not, we try again:

// One atomic compare-and-update decides the whole step
public bool TryReserveAtomic(string isbn)
{
    while (true)
    {
        int current = _available[isbn];
        if (current == 0)
            return false;
        if (_available.TryUpdate(isbn, current - 1, current))
            return true;
    }
}

This loop is exactly the compare-and-swap idea; we look at its hardware side in the third post and at lock-free algorithms in the seventh. We covered ConcurrentDictionary APIs such as AddOrUpdate and GetOrAdd in the synchronization post of the C# Asynchronous Programming series.

The opposite direction is possible too: code with a data race but no race condition. The workers in the report job write true to a shared _hadActivity bool field without synchronization whenever they find work, and the field is only read after all workers have finished with Join. Several threads write to the same location without a lock, so by definition there is a data race; but they all write the same value, and the read comes after the order that Join establishes. The result does not depend on timing. I would still write such code with Volatile.Write, so that the next developer does not read it as "data races are harmless" and try the same with a counter.

Order Violations and Lazy Initialization

The third class is the order violation: A has to happen before B, but the code does not enforce it. In the 2008 study in which Shan Lu and colleagues examined 105 real concurrency bugs, about a third of the non-deadlock bugs were of this kind. A class that tracks shipments starts a background thread in its constructor but assigns the dictionary it uses only afterwards:

// Order violation: the worker thread may run before _carriers is assigned
public ShipmentTracker()
{
    _worker = new Thread(Poll);
    _worker.Start();
    _carriers = new() { ["TR"] = "Yurtici", ["DE"] = "DHL" };
}

dotnet run -c Release --project src/Bookstore.ConcurrencyLab -- init-order 10000

runs=10000 worker saw an uninitialized field in 5429 runs

In more than half of ten thousand runs, the worker ran into a null field. This rate varies a lot between machines: in an environment where starting a thread is slow, the constructor almost always finishes first and the bug can hide for years. The fix is to express the order in code: start the thread after the object is fully built, or move it to a Start method.

The lazy initialization race is a special case of check-then-act:

// Lazy initialization race: two threads can both see null
public static SearchIndex Get()
{
    if (_index is null)
        _index = new SearchIndex();
    return _index;
}

dotnet run -c Release --project src/Bookstore.ConcurrencyLab -- lazy-init 10000

rounds=10000 index created more than once in 165 rounds

In 165 of ten thousand rounds where four threads asked at the same time, the search index was built more than once. If the index is cheap, this is only waste; but if the constructor locks a file, opens a connection or increments a counter, the result is broken. The right tool is Lazy<T>; we will see why it is right, and the subtleties of double-checked locking, in the memory model post.

Torn Reads: Half Old, Half New

The last class is the sneakiest. According to the .NET memory model spec, reads and writes of properly aligned primitive types and references are atomic up to pointer size: in a 64-bit process long and double are read in one piece, in a 32-bit process there is no such guarantee. The spec makes no atomicity promise for decimal or your own structs. So if one thread writes a 16-byte decimal while another reads it, the reader can see a number whose halves come from the old and the new value. This is called a torn read. I built an experiment where a writer alternates between two valid values and a reader counts values that were never written:

Thread reader = new(() =>
{
    while (!Volatile.Read(ref done))
    {
        decimal seen = box._price;
        if (seen != 1.00m && seen != 99_999.99m)
            torn++;
    }
});

I ran the same experiment for PriceRange(Min, Max), 16 bytes made of two long values, and for ShelfPrice(List, Campaign), 32 bytes made of two decimal values. Every written value has Min ≤ Max and a campaign price ≤ the list price; the reader counts values that break this invariant:

dotnet run -c Release --project src/Bookstore.ConcurrencyLab -- torn 50000000

decimal: 0 torn reads in 50,000,000 writes
PriceRange: 0 reads with Min > Max in 50,000,000 writes
ShelfPrice: 1,959,926 reads with Campaign > List in 50,000,000 writes

For the two 16-byte types I saw no tearing in 50 million writes; for the 32-byte ShelfPrice, about two million reads saw a campaign price above the list price, a price pair that was never written. Looking at the JIT's output with DOTNET_JitDisasm, on ARM64 the 16 bytes are copied with a single ldp/stp instruction pair and the 32 bytes with two separate pairs. The Arm architecture makes such a pair visible in one piece only under certain conditions (the processor's FEAT_LSE2 feature and 16-byte alignment). So seeing zero for decimal is an observation for this processor and this JIT output, not a guarantee: the spec promises nothing, and on x64 the guarantee also only covers up to 8 bytes.

The practical consequence of torn reads: if you share a value made of several fields (a price range, an amount and its currency, a coordinate) without a lock, the reader can see a combination that never existed, even if you write each field correctly. The fix is either a lock or keeping the value as an immutable class and swapping its reference; a reference assignment is atomic, and the reader sees either the whole old object or the whole new one.

Thinking in Invariants, and the Range of Fixes

The common language of all these examples is the invariant: a rule that must always hold. "Sold + remaining = initial stock", "campaign price ≤ list price", "the index is built only once". The most productive way to analyse a race is to ask: "which invariant looks broken, between which two steps, to which thread?" If the code contains steps that temporarily break an invariant (stock was reduced but the sale is not recorded yet), those steps must run together in a way other threads cannot see. That is exactly the idea of a critical section, and the subject of the next post.

Ordered from the safest, not from the most expensive, the fixes are: Do not share: each worker works on its own data and results are merged at the end (the report in the first post worked like this). Immutable data: the value never changes; a change produces a new object and its reference is swapped atomically. Atomic operation: a single field is updated with a compare-and-swap such as Interlocked or TryUpdate. Lock: the general way to protect several fields or a compound check-and-act sequence. The APIs themselves are in the synchronization post of the C# Asynchronous Programming series; in this series we focus on which one protects which invariant and why.

In Production

A race you do not see in tests shows up under production load. In our experiments we released eight threads in the same microsecond with a barrier; a typical unit test runs on one thread, or with threads far apart in time, and sees nothing. In production hundreds of requests hit the same stock record in the same second; at the start of a campaign this happens all at once. In concurrency tests, start threads together with a barrier, repeat the round thousands of times, and verify the invariant rather than a single result: "sold ≤ 100" and "sold + remaining = 100".

Race conditions are not only in memory. The same check-then-act shows up in the database as "SELECT first, then UPDATE", and across two HTTP requests as "check the balance first, then spend". We covered such business logic flaws (double spending, coupon reuse) in the A06 post of the Web Security (OWASP Top 10) series; the fix there is the same: reduce the check and the action to a single atomic step, for example a conditional UPDATE ... WHERE Available > 0.

In the logs, the trace of a race is inconsistency. Negative stock, a report whose totals do not add up, a record saying "this order was shipped twice" is not just a bug report but evidence that an invariant was broken. Before writing a job that silently repairs such inconsistencies, try to find the interleaving that produces them.

Common Pitfalls

Assuming code is automatically thread-safe once it uses a ConcurrentDictionary. In the experiment the concurrent collection gave the worst result: 1403 of two thousand rounds oversold. A thread-safe collection only protects its own individual operations, not your compound operation.

Saying "it never happened to me". The order violation experiment failed 54% of the time on this machine; in an environment where thread start is slower, the same code can look bug-free for years. Having worked does not show that code is correct; you can only show that by reasoning about orders.

Guessing atomicity from the size of the type. Do not rely on a guarantee the spec does not give just because the 16-byte decimal did not tear on this ARM64 processor; the same code can break with long in a 32-bit process, or with decimal on another processor.

Protecting each field separately. Writing the List and Campaign prices separately with Interlocked does not protect the invariant; they are a pair that must change together. The unit of protection is set by the invariant, not by the field.

A Small Experiment

★☆☆ Run the lost-update command with 1, 2, 4 and 8 threads and put the share of lost increments into a table. Why does the share grow with the thread count?

★★☆ Protect the ShelfPrice experiment with a lock, then instead of the lock make ShelfPrice a sealed record class and update the field with a reference assignment. Verify that the number of reads breaking the invariant is zero in both, and explain which one never makes the reader wait.

★★★ Add a TryReserveBoth(isbn1, isbn2) method to CatalogStock that reserves two books in one go: if both are available, both are reduced; if one is missing, neither is. Show why you cannot write this correctly with TryUpdate, and solve it with a lock. Write a stress test that verifies the invariant.

In this post we named the races and saw that the essence of the fix is "reduce the check and the action to a single step other threads cannot see". In the next post we go into the theory behind it: the three requirements of the critical section problem, attempts to solve it with only reads and writes, why Peterson's algorithm breaks on this ARM64 machine, and where compare-and-swap, the semaphore and the monitor come from.