Read Concern, Write Concern and Read Preference: Tuning Consistency in MongoDB

Version note: The examples in this post were run with MongoDB 9.0.8 Community Server, mongosh 2.6, .NET 10 and MongoDB.Driver 3.14. The scripts gave the same result on MongoDB 8.0.38; the only difference is the name of one error code, covered in the relevant section.

Guess First

In Bookstore, the online bookstore used as the shared example across this site's series, the payment service marks an order Paid, then the customer is redirected to the order page. To take load off the primary, the order page reads from a secondary. This post's lab is a three-member replica set, and the two secondaries are behind at that moment. The payment service marks order 379 as paid and creates order 380, both with w: 1:

// The payment service: order 379 becomes Paid, order 380 is created, both with w:1
orders.updateOne({ _id: 379 }, { $set: { status: "Paid" } }, { writeConcern: { w: 1 } });

show("primary, read concern local:", id => orders.findOne({ _id: id }));
show("secondary:", id => orders.find({ _id: id }).readPref("secondary").next() ?? null);
show("primary, read concern majority:", id => orders.find({ _id: id }).readConcern("majority").next() ?? null);

Three reads, two orders. Which one sees what? The third line reads from the primary; it should see the current value too, right?

primary, read concern local:       379 Paid, 380 Paid
secondary:                         379 Pending, 380 not found
primary, read concern majority:    379 Pending, 380 not found

The page reading from a secondary tells the customer "order not found"; 379 still looks unpaid. The third line shows the less known side: even when you read from the primary, if you ask for readConcern: "majority" you do not see writes that have not reached a majority. This post covers the three settings that decide consistency in MongoDB: when a write counts as "done" (write concern), which data a read sees (read concern) and which member a read goes to (read preference). The theory itself, consistency models, is in the "Consistency Models: Linearizability, Sequential, Causal and Eventual Consistency" post of the Distributed Systems Fundamentals series; here we look at how MongoDB applies these ideas and what they cost.

The Lab: a Three-Member Replica Set in One Container

The lab from post 2 has a single member; without secondaries none of this post's output would exist. In a real deployment each member runs on a separate machine, preferably in a separate zone. In the lab the goal is to see the behavior, so we run three mongod processes in one container and give each a 0.25 GB WiredTiger cache; total memory stays around 1 GB, and together with post 2's container two containers run on the machine. The container's start script:

# 12 - three mongod processes in one container: a 3-member replica set with a small footprint (lab only)
for port in 27021 27022 27023; do
    mkdir -p "$base/$port"
    mongod --replSet rs3 --port "$port" --bind_ip_all --dbpath "$base/$port" \
        --keyFile "$base/keyfile" --wiredTigerCacheSizeGB 0.25 \
        --logpath "$base/$port.log" --pidfilepath "$base/$port.pid" --fork
done

compose.rs3.yaml publishes the three ports only on 127.0.0.1 and uses the admin password from post 2's .env file. The members are named localhost:27021–27023; as in post 2, these names resolve both inside the container and from the application on Windows. In PowerShell:

docker compose -f compose.rs3.yaml up -d
docker compose -f compose.rs3.yaml exec rs3 bash /lab/scripts/12-rs3-init.sh
docker compose -f compose.rs3.yaml exec rs3 mongosh "mongodb://admin@localhost:27021,localhost:27022,localhost:27023/?authSource=admin&replicaSet=rs3" --quiet /lab/scripts/12-consistency.js -p

localhost:27021 PRIMARY
localhost:27022 SECONDARY
localhost:27023 SECONDARY
books: 77

The init script initiates the replica set, creates the admin and bookstore_app users through the localhost exception and loads the Bookstore data with mongoimport. To make lag visible, 12-consistency.js locks both secondaries with fsyncLock: a locked member accepts no writes and cannot apply the oplog, but it still answers reads. It is an exaggerated version of a secondary falling behind under heavy load. At the end the script releases the locks and undoes its changes.

Write Concern: When Is a Write Done?

Write concern says on how many members the server must see a write before acknowledging it. w: 1 is the primary's acknowledgment alone, w: "majority" means a majority of the voting members have durably written the change to their oplog. What applies if you say nothing?

default write concern: {"w":"majority","wtimeout":0} source: implicit

The default on replica sets is majority; source: implicit shows that nobody set it explicitly and it is the server's own default. With three members a majority is two, the primary plus one secondary. j: true asks for the write to reach the journal before the acknowledgment; whether majority covers the journal as well is decided by the replica set's writeConcernMajorityJournalDefault setting. The real trap is in the timeout:

// w:"majority" with a timeout: the error does not undo the write
attempt("w: majority, wtimeout 2000:", () => {
    const r = orders.updateOne({ _id: 378 }, { $set: { note: "checked" } }, { writeConcern: { w: "majority", wtimeout: 2000 } });
    return JSON.stringify(r);
});
print("order 378 note on the primary:".padEnd(34), orders.findOne({ _id: 378 }).note);

w: majority, wtimeout 2000:        WriteConcernTimeout: waiting for replication timed out
order 378 note on the primary:     checked

The application got an error, but the write is on the primary, and the secondaries will receive it once the lock is released. A write concern error does not mean "the write did not happen"; it means "I could not give you the guarantee you asked for in time". If the application treats this as a failure and lets the user try again, the same payment can be written twice; the operation must be idempotent, or you must decide by reading the result back. Difference in 8.0: in the same situation 8.0.38 returned the name WriteConcernFailed, 9.0.8 WriteConcernTimeout; if you have error handling code that looks at the name rather than the code, check it when upgrading.

The price of w: 1 is durability. If the primary acknowledges the write and crashes before passing it on to the secondaries, the new primary has never seen this write; when the old primary comes back, the write is undone by a rollback. The user saw "your payment was received", and the database has no payment.

Read Concern: Which Data Does a Read See?

Read concern decides which level of durability a read sees. local (the default) is the latest data the member has, including writes that may be undone by a rollback. majority is only data that has reached a majority and so cannot be undone; this is why the third line of "Guess First" showed Pending: the primary knew about Paid, but that write was not on a majority yet. snapshot is used in transactions and for reading a consistent view at a point in time with atClusterTime. linearizable is the strongest: it guarantees that the read sees all majority writes acknowledged before it started, and to do that it confirms with the secondaries that the primary can still acknowledge w: "majority" writes:

read concern linearizable:         MaxTimeMSExpired: operation exceeded time limit

While the secondaries were locked, the confirmation never came and the read was cut off at the maxTimeMS limit; without maxTimeMS it could have waited forever. linearizable only makes sense on the primary, for queries that return a single document, and it adds a round trip to a majority to every read. It is for the rare reads that need "definitely the latest value", such as a stock count; not for an order list.

Read Preference: Which Member Does a Read Go To?

To limit how stale secondary reads can be there is maxStalenessSeconds, but the documentation says it cannot be smaller than 90 seconds: because drivers check members' last write time at infrequent intervals, the estimate is coarse. So there is no setting for "at most a few hundred milliseconds stale". Reading from secondaries has one more misconception: every secondary applies all of the primary's writes itself, so secondaries do not share the write load, they only spread reads. If the primary is struggling with writes, moving reads to secondaries does not solve it.

Causal Consistency: Reading Your Own Writes

What the order page really needs is not "the latest data for everyone" but "seeing the payment this customer just made": read-your-writes. MongoDB provides this with a causally consistent session. After each operation the session keeps the timestamp the server returned (operationTime) and adds it to the next read as afterClusterTime; if the secondary has not caught up to that point, it waits instead of answering. While locked:

// Causal consistency: a read in the same session waits for the session's own write
const session = db.getMongo().startSession({ causalConsistency: true });
const sessionOrders = session.getDatabase("bookstore").orders;
sessionOrders.updateOne({ _id: 379 }, { $set: { status: "Shipped" } }, { writeConcern: { w: 1 } });
attempt("causal session, secondary read:", () => sessionOrders.find({ _id: 379 }).readPref("secondary").maxTimeMS(2000).next().status);

causal session, secondary read:    MaxTimeMSExpired: Error waiting for optime
read concern linearizable:         MaxTimeMSExpired: operation exceeded time limit
--- secondaries caught up
secondary:                         379 Shipped, 380 Paid
primary, read concern majority:    379 Shipped, 380 Paid
causal session, secondary read:    Shipped

While locked, the session did not give a stale answer, it waited; once the lock was released, the same read returned Shipped. In real life secondaries are not locked but a few milliseconds behind. I measured what that means from .NET: the read-your-writes command writes a number to order 379 200 times and reads it back from a secondary right away, counting a read when the value is older than the one written. No locks this time:

IMongoCollection<BsonDocument> secondary = reader.WithReadPreference(ReadPreference.Secondary);

using IClientSessionHandle session = await client.StartSessionAsync(new ClientSessionOptions { CausalConsistency = true });
await RunAsync("w:majority, secondary, causal session",
    orders.WithWriteConcern(WriteConcern.WMajority), orders.WithReadConcern(ReadConcern.Majority), session);

    await writer.UpdateOneAsync(session, order379, update);
    read = await secondary.Find(session, order379).FirstAsync();

w:1, secondary                           stale reads: 59 of 200
w:majority, secondary                    stale reads: 3 of 200
w:majority, secondary, causal session    stale reads: 0 of 200

The second run gave 51, 5 and 0; on 8.0.38 56, 2 and 0. The numbers change from machine to machine, the order does not. With w: 1, roughly one read in four is stale. w: "majority" narrows the gap but does not close it: a majority is the primary plus one secondary, and the read can land on the other secondary. With a causal session, zero. The documentation states clearly why this combination matters: for the four causal consistency guarantees (reading your own writes among them) to hold in situations such as a network partition too, reads must use majority read concern and writes majority write concern. A session with w: 1 and local also works on a normal day, but gives no guarantee during a failover. In .NET the session must be carried through the request; in ASP.NET Core this means keeping it as a scoped service, and if the payment and the order page are different requests, storing the last operationTime (session.OperationTime) in the customer's session and passing it to the new session with AdvanceOperationTime.

The Limits of Multi-Document Transactions

The script's second line reads two server parameters related to transactions:

transaction lifetime: 60 s, max open transactions: 10000

transactionLifetimeLimitSeconds defaults to 60: a transaction that runs longer is aborted in the background. maxConcurrentMultiDocumentTransactions appears as new in the 9.0 release notes: when the number of transactions open at the same time reaches 10000, the server rejects new ones with a TooManyOpenTransactions error; 0 removes the limit. Difference in 8.0: the parameter exists on 8.0.38 as well and returned the same value; the 8.0 parameter documentation lists it as added in 8.0.30. So the limit applies to the latest 8.0 patches too. The typical reason for approaching the limit is many transactions left open: transactions waiting without a commit or an abort fill both this counter and the WiredTiger cache. Transactions also read only from the primary in terms of read preference; moving reads to secondaries does not lighten the transaction load. The rule from post 10 applies: keep transactions short, the body free of side effects, and first check whether the model fits in a single document.

In Production

Set these deliberately and in one place. The connection string has the w, readConcernLevel and readPreference options; in .NET you can also change them per collection with WithWriteConcern, WithReadConcern and WithReadPreference. As a rule, keep the defaults (majority writes, primary reads) and make the exceptions in places with a name in the code: "reports use secondaryPreferred", "log writes use w: 1".

Monitor replication lag. rs.printSecondaryReplicationInfo() shows how far each secondary is behind the primary; every screen reading from a secondary passes this number on to the user as staleness. If the lag grows, the cause is usually the secondary's disk or heavy reads running on it; in post 13 we look at a hidden analytics member for reports.

Common Pitfalls

Lowering the write concern and calling it "faster". w: 1 reduces write latency because it does not wait for a secondary; the time saved is data that can be lost in a failover. Unacceptable for payments, orders and stock; acceptable perhaps for a click counter or a view log. Make the decision with measurements and in writing.

Using secondaries for reporting without limits. When a heavy aggregation fills a secondary's CPU and cache, it applies the oplog more slowly; the lag grows, and because w: "majority" writes wait for that secondary, writes on the primary slow down too. On top of that, that secondary may become primary in an election. For reports, the hidden member from post 13 or the materialized view from post 8 is safer.

Taking a write concern timeout for a rollback. The 378 example above: an error came back, the data was written. Read the state before retrying, or make the operation idempotent.

A Small Experiment

★☆☆ Lock only one secondary in 12-consistency.js. How do the w: "majority" and readConcern("majority") lines change? Why?

★★☆ Add a fourth scenario with ReadPreference.Nearest to the read-your-writes command and compare the number of stale reads. Then try the causal session with w: 1 and local; why is the result still zero on a normal day?

★★★ Let payment and the order page be separate HTTP requests: in the first request write session.OperationTime to a cookie, in the second open a new causal session, advance it with AdvanceOperationTime and read from a secondary. Does the number of stale reads stay at zero?

In the next post we look inside the replica set in the same three-member lab: we kill the primary and measure how long the election takes, compute the oplog window, and then move on to sharding and see how the choice of shard key distributes the orders.