System Design Case Study: Chat and Notification System

Version note: The code in this post was built and run with C# 14 and .NET 10; the gateway routing example ran against Redis 7.4 with StackExchange.Redis 3.4.

Guess First

The Reader Community is getting chat for book clubs. The first version is simple: the ASP.NET Core app keeps WebSocket connections in an in-memory dictionary and, when a message arrives, finds the recipient's connection and writes to it. It works perfectly in development. In production the app scales to two instances behind a load balancer; Ada's connection lands on the first instance and Grace's on the second. To order messages, the timestamp sent by the client is used.

Two questions. When Ada writes a message to Grace, does Grace see it? And if Grace's phone clock is two minutes behind, in what order do the messages in the conversation appear?

Requirements and Capacity Estimation

Functional requirements: one-to-one and group chat (a book club has at most a few hundred people); messages reach an online recipient immediately; an offline recipient gets a push notification and sees what she missed when she opens the app; sent, delivered and read states for messages; who's online (presence). Non-functional requirements: messages must not be lost; messages in a conversation must appear in the same order for everyone; delivery to an online recipient under 500 ms at p99 (the time 99% of requests stay under); message history is permanent. Out of scope: end-to-end encryption, voice and video calls, file sharing.

Assumptions: 5 million DAU (Daily Active Users), 40 messages per person per day, 1 KB per stored message, 30% of DAU connected at the same time, a gateway node carrying 100 thousand concurrent connections.

const double dau = 5_000_000, messagesPerUser = 40, bytesPerMessage = 1_000, peak = 3, connectionsPerNode = 100_000;
double sendsPerSec = dau * messagesPerUser / 86_400;

messages/s 2,315 (peak 6,944), storage/day 200 GB, per year 73 TB
concurrent connections if 30% of DAU are online: 1,500,000 -> 15 gateway nodes at 100,000 each

The message rate (2,300 per second, 7,000 at peak) isn't scary on its own. Two numbers are decisive: a message history that grows by 73 TB a year and is never deleted, and 1.5 million concurrent, long-lived connections. The second sets this system apart from the ones so far: an HTTP (Hypertext Transfer Protocol) request lasts milliseconds and ends; a WebSocket connection stays open for hours and is tied to one server. Memory per connection, the server's file descriptor limit and what happens to connections during a deploy are all part of the capacity plan.

API and Data Model

There are two channels between client and server. The WebSocket channel is for real-time traffic: the client sends send (clientMsgId, conversationId, text); after storing the message the server replies with ack (clientMsgId, seq) and pushes message (conversationId, seq, from, text) to the other members; the client reports delivered and read (conversationId, seq) for messages it received and read. A regular HTTP API (Application Programming Interface) serves history: GET /api/conversations/{id}/messages?afterSeq=120&limit=50. A client that opens the app or loses its connection fetches everything after the last seq it saw from here.

The data model: Conversations and Members (with LastDeliveredSeq and LastReadSeq per member), and Messages. The message table is the part of this system that grows, and its access pattern is clear: "the last 50 messages of this conversation" and "everything after 120 in this conversation". So the partition key is the conversation ID plus a time bucket, and the sort key is seq. The design described on Discord's engineering blog follows the same idea: according to the 2017 post "How Discord Stores Billions of Messages", messages were stored with a partition key made of the channel ID and a fixed time window; the 2023 post "How Discord Stores Trillions of Messages" describes the hot partition problems in a 177-node Cassandra cluster and the move to a 72-node ScyllaDB cluster. The time bucket keeps a very active conversation from growing a single partition forever.

High-Level Design

There are two kinds of servers. Gateway nodes are stateful: they hold WebSocket connections and do nothing else. The chat service is stateless: it assigns a sequence number to a message, stores it, finds where the recipients are and forwards it. What ties them together is the presence record (presence:{user} → which gateway) and a pub/sub channel each gateway listens on. If the recipient isn't on any gateway, the message goes to the push worker. The answer to the first Guess First question is in this diagram too: in the single-instance design the connections were in a dictionary; with two instances, Grace's connection isn't in the memory of the instance processing Ada's message, and the message is silently lost.

Deep Dive: WebSockets and Scaling Stateful Servers

WebSocket was standardized in 2011 as RFC (Request for Comments) 6455. It starts with an HTTP request (Upgrade: websocket) and then becomes a two-way channel over the same TCP (Transmission Control Protocol) connection. In the series' "From One Server to Millions of Users: Vertical and Horizontal Scaling and the Stateless Tier" post we talked about making the app tier stateless; WebSocket is the deliberate exception to that rule. The connection itself is state. What you do is confine the state to the smallest possible area: the gateway holds only the connection, while business logic and data stay in the stateless chat service.

That has practical consequences. The load balancer must support long-lived connections, and its idle timeout must be longer than the WebSocket ping/pong interval; otherwise a quiet conversation's connection gets cut in between. Deploying a gateway means closing the tens of thousands of connections on it; if all the disconnected clients try to reconnect in the same second, the remaining nodes face a thundering herd. That's why connections are drained gradually during a deploy (connection draining) and the client's reconnect logic uses jittered backoff, as in the series' "Reliability Patterns: Timeouts, Retries, Backoff with Jitter, Circuit Breakers and Load Shedding" post.

I set up routing with two gateways and one Redis. Each gateway writes the presence record of a connecting user with a 30-second TTL (Time To Live) and listens on its own channel; the chat service assigns the message a sequence number and stores it, then finds each recipient's gateway and publishes to that channel:

public async Task<ChatMessage> SendAsync(string conversationId, string from, string[] members, string text)
{
    IDatabase db = redis.GetDatabase();
    long seq = await db.StringIncrementAsync($"conv:{{{conversationId}}}:seq");      // order decided by the server
    ChatMessage message = new(conversationId, seq, from, text);
    await db.ListRightPushAsync($"conv:{{{conversationId}}}:log", JsonSerializer.Serialize(message));  // stored before delivery

    foreach (string to in members.Where(m => m != from))
    {
        RedisValue gateway = await db.StringGetAsync($"presence:{to}");
        if (gateway.HasValue)
            await redis.GetSubscriber().PublishAsync(RedisChannel.Literal($"gw:{gateway}"),
                JsonSerializer.Serialize(new Envelope(to, message)));
        else
            await db.ListLeftPushAsync("push:queue", $"{to}: new message from {from}");     // offline: push notification
    }
    return message;
}

Ada on gw-1, Grace on gw-2, Alan offline; two messages:

  gw-2 -> grace : #1 ada: Chapter 3 tonight?
  gw-1 -> ada   : #2 grace: Yes, 9 pm
  push queue    : alan: new message from ada | alan: new message from grace
  presence      : ada=gw-1 (ttl 30 s), alan=offline

Even though Ada's message arrived at gw-1, it reached Grace on gw-2; two push notifications were queued for Alan. Redis pub/sub is "fire and forget": if nobody is listening on the channel at that moment, the message is lost. That's not a problem in this design, because the message was stored before it was published; a message that couldn't be delivered comes from the history API when the client reconnects. Pub/sub is only the fast path here; durability lives in the message store. In the .NET world the ready-made version of this structure is SignalR with a Redis backplane, or a managed SignalR service; the details are the subject of the upcoming Real-time Applications series in the Backend category.

Deep Dive: Message Ordering and Delivery

The answer to the second Guess First question: in a conversation ordered by client timestamps, Grace's replies, with her clock two minutes behind, show up before the messages she's replying to. Client clocks can't be trusted. A single place must decide order: a counter per conversation (the INCR above), or a log written to a single partition per conversation. The main idea of the Messaging and Event-Driven Architecture series post "Message Ordering and Partitioning: Where Ordering Guarantees Begin and End" holds here too: global order is expensive and unnecessary; order per conversation is enough.

Even with the server deciding order, the network may deliver messages late, twice or not at all: when a connection drops, the gateway's buffered messages are gone, and a reconnecting client receives some of them both from push and from history. seq makes all three cases solvable on the client side:

public sealed class ClientInbox
{
    private readonly SortedDictionary<long, ChatMessage> _pending = [];
    public long LastShown { get; private set; }
    public List<ChatMessage> Shown { get; } = [];

    public string Receive(ChatMessage m)
    {
        if (m.Seq <= LastShown || _pending.ContainsKey(m.Seq)) return $"seq {m.Seq}: duplicate, dropped";
        _pending[m.Seq] = m;
        while (_pending.Remove(LastShown + 1, out ChatMessage? next))
        {
            Shown.Add(next);
            LastShown = next.Seq;
        }
        return _pending.Count == 0
            ? $"seq {m.Seq}: shown up to {LastShown}"
            : $"seq {m.Seq}: buffered, waiting for {LastShown + 1}";
    }
}

Let 6 arrive early, 4 arrive twice, and 5 get lost:

  seq 1: shown up to 1
  seq 2: shown up to 2
  seq 3: shown up to 3
  seq 4: shown up to 4
  seq 6: buffered, waiting for 5
  seq 4: duplicate, dropped
  seq 7: buffered, waiting for 5
  gap detected: fetch seq 5 from history
  seq 5: shown up to 7
  shown: 1,2,3,4,5,6,7

The duplicate was dropped, the early arrivals were held back and the gap was noticed; the client asked history for 5 with afterSeq=4, and the conversation completed in the right order. The sender-side counterpart is clientMsgId: when the client resends a message it got no ack for, the server recognizes the same clientMsgId and returns the old seq instead of assigning a new one. That's the chat version of the idea in the Solution Patterns series post "Idempotency Keys: Not Creating the Same Order Twice on Double Clicks and Retries".

Delivery states get cheap through seq as well. "Sent" is the moment the server stores the message and returns the ack. For "delivered" and "read", instead of a separate flag per message, two numbers per member are enough: LastDeliveredSeq and LastReadSeq. "Grace has read up to 42" means 42 and everything before it has been read; in a group of 200, that's two numbers per member instead of 200 rows per message.

Deep Dive: Offline Users, Push and Presence

The way to reach an offline user on mobile is the platform's push service: APNs (Apple Push Notification service) for Apple devices, FCM (Firebase Cloud Messaging) for Android. The push worker reads from the queue and calls them; since the platform services have their own rate limits and error codes, this work is done asynchronously with its own retry and backoff logic, not on the request path. A push notification isn't the message carrier, it's a wake-up call: it contains a short preview, and when the user opens the app, messages come from the history API. That way a lost or late push doesn't cause data loss. Since a user can have more than one device, presence and push are tracked per device, not per user.

The presence record has a TTL: the client sends a heartbeat every 15–20 seconds and the gateway renews the record; when a connection dies silently, the record drops off on its own in 30 seconds. The real cost is spreading presence. Telling all of a user's friends the moment she comes online means a fan-out on every connect and disconnect; the fan-out problem from the series' "System Design Case Study: News Feed and Timeline" post is here too. Most products relax this: presence is sent only to members of the open conversation, or the client periodically asks about the people on screen; nobody notices if "last seen" is a few seconds late.

Deep Dive: Group Chat and Message Storage

In a book club of a few hundred people, fan-out can happen at write time: each message means as many presence lookups and publishes as there are members. A more efficient version publishes per gateway rather than per user: if 40 members of the conversation are on the same gateway, one publish is enough and the gateway distributes the message locally to 40 connections. In channels with thousands or hundreds of thousands of members (an author's announcement channel, say), instead of delivering to everyone, gateways subscribe to the channel and deliver only to clients that have it open at that moment; the rest read from history the next time they open it. That's the chat counterpart of the hybrid model in the news feed.

On the storage side there's data growing by 73 TB a year, almost never updated, always read by conversation and time range. Through the decision lens of the series' "Scaling the Data Layer: Replication, Sharding and SQL vs NoSQL" post (SQL being Structured Query Language), this access pattern fits a wide-column store (Cassandra, ScyllaDB) well; in a relational database it can also be done with sharding by conversation ID and time-based partitioning. The lesson from Discord's 2023 post is independent of the store: a very popular channel keeps a single partition hot. Discord put a data service layer in front of the database that collapses identical concurrent queries into one (request coalescing); the database-side version of the request coalescing in the series' "Caching Strategies: Cache-Aside, Write-Through, Write-Behind and Cache Stampede" post.

Trade-offs

Common Pitfalls

Keeping connections in instance memory and scaling out

On the second instance, messages are silently lost. Where a connection lives must be in a shared record (presence), and messages routed there.

Ordering by client clock

Phone clocks drift. The server decides order; the client only displays it.

Delivering first and storing second

If delivery succeeds and storage fails, the message shows up for the recipient but isn't in history. Store first, then deliver.

Treating push as a data channel

Push can be late and can get lost. It's a wake-up call; messages come from history when the app opens.

Dropping every connection at once during a deploy

Tens of thousands of clients reconnecting in the same second take down the remaining nodes. Drain gradually, and use jittered backoff on the client.

A Small Experiment

★☆☆ Redo the capacity estimate with 20 million DAU and 50% of users connected at once. How many gateway nodes do you need? When one node goes down, how much load do reconnecting clients put on the others?

★★☆ Add clientMsgId to ChatService.SendAsync: a message arriving a second time with the same clientMsgId shouldn't get a new seq, it should return the first message's seq. Choose a way to do this atomically in Redis and test it.

★★★ Change routing to publish per gateway rather than per user: in a group of 200 members spread over 15 gateways, how many publishes happen per message?

So far we've designed systems with predictable traffic. In the series' last post we look at a scenario where traffic jumps a hundredfold in a second and stock is limited: flash sales and concert ticketing.