Resilience in SignalR: Automatic Reconnect, Stateful Reconnect, Streaming and MessagePack

Version note: The code in this post was built and run with .NET 10, Microsoft.AspNetCore.SignalR.Protocols.MessagePack 10.0.14, Angular 22.2, and @microsoft/signalr and @microsoft/signalr-protocol-msgpack 10.0.11.

Guess First

You turned on AllowStatefulReconnects on the server and withStatefulReconnect() on the client. The promise of stateful reconnect is that messages missed during a drop are sent again. The user's Wi-Fi goes away for 2 seconds and comes back, and the server sends 7 messages in the meantime. Does the user receive those 7 messages?

I found the answer in the JavaScript client's source code, and it was not what I expected. In this post we treat a drop not as an error but as a normal event that happens on mobile networks, when laptop lids close, and on every deploy: reconnecting, message loss, streaming, MessagePack and slow clients.

withAutomaticReconnect and a Custom Retry Policy

By default the SignalR client does nothing when the connection drops; the connection falls to Disconnected and stays there. withAutomaticReconnect() without parameters tries four times, waiting 0, 2, 10 and 30 seconds, and then gives up. For a server deploy that takes 42 seconds this may not be enough, and thousands of clients reconnecting in the same seconds also strain the server. You can write your own policy:

// 0, 1, 2, 4, 8 ... s (at most 10) + 20% jitter; give up after 60 s
const retryPolicy: IRetryPolicy = {
  nextRetryDelayInMilliseconds(context: RetryContext): number | null {
    if (context.elapsedMilliseconds > 60_000) {
      return null;
    }
    if (context.previousRetryCount === 0) {
      return 0;
    }
    const base = Math.min(1000 * 2 ** (context.previousRetryCount - 1), 10_000);
    return base * (0.8 + Math.random() * 0.4);
  },
};

Returning null means "give up"; at that point the connection becomes Disconnected and onclose is called. Making the first attempt without waiting closes short drops quickly; jitter (a ±20% random deviation) keeps all clients from arriving in the same millisecond when the server restarts. RetryContext also gives the reason for the drop through retryReason; if you got a 401, for example, sending the user to the login page makes more sense than retrying.

After Reconnecting: UI State and Rebuilding Groups

@Injectable({ providedIn: 'root' })
export class InventoryService {
  private readonly connection: HubConnection = new HubConnectionBuilder()
    .withUrl('http://localhost:17607/hubs/inventory')
    .withAutomaticReconnect(retryPolicy)
    .withStatefulReconnect()
    .configureLogging(LogLevel.Warning)
    .build();

  readonly status = signal<ConnectionStatus>('disconnected');

  constructor() {
    this.connection.onreconnecting(() => this.status.set('reconnecting'));
    this.connection.onreconnected(() => {
      this.status.set('connected');
      // new connection id: rejoin groups, refresh missed state over HTTP
    });
    this.connection.onclose(() => this.status.set('disconnected'));
  }

  async start(): Promise<void> {
    this.status.set('connecting');
    await this.connection.start();
    this.status.set('connected');
  }
}

Three callbacks, three UI (User Interface) states. In onreconnecting, show the user a "connection lost, retrying" banner and disable buttons that rely on live data; in onreconnected, remove the banner; in onclose, offer a "reload the page" option. When I killed the server with kill -9 while the page was open in the browser, the screen changed to "Status: reconnecting", and a few seconds after restarting the server it went back to "Status: connected".

The body of onreconnected is too important to leave empty. Remember from the earlier posts: the reconnected connection gets a new connection id and is not a member of any group. This callback is the right place to rejoin the watched groups and refresh the state missed during the drop with an ordinary HTTP (Hypertext Transfer Protocol) request.

Stateful Reconnect: Preventing Message Loss on Short Drops

app.MapHub<InventoryHub>("/hubs/inventory", options =>
    options.AllowStatefulReconnects = true);

With stateful reconnect, introduced in .NET 8, both sides keep the messages they send in a buffer until the other side acknowledges them (100,000 bytes by default). If the transport drops, the client reconnects with the same connection id and unacknowledged messages are sent again. To test this, I connected the clients to the server through a small TCP (Transmission Control Protocol) proxy that lets me cut every connection at will. Two clients: one with only withAutomaticReconnect, the other additionally with withStatefulReconnect. While the server sent 100 sequence-numbered messages 20 ms apart, I cut the connections at the proxy (the proxy kept accepting new connections immediately):

 0.1s connected: plain=RxgL6AdcA25VXXQF8CxyHg stateful=DuibFD7nk5g5o_XeTfsIRw
 1.1s --- network drop (0 ms)
 1.1s plain    reconnecting: WebSocket closed with status code: 1006 (no reason given).
 1.1s --- network back
 1.1s plain    reconnected, connectionId EdAbrbm6U8IxYYbHxnsBSg
plain    id EdAbrbm6U8IxYYbHxnsBSg, received 99/100, missing: 48
stateful id DuibFD7nk5g5o_XeTfsIRw, received 100/100, missing: none

The plain client got a new connection id and lost message 48, sent in the few milliseconds while it was reconnecting. On the stateful client, onreconnecting never fired, the connection id stayed the same and all 100 messages arrived. The application code did not even notice the drop.

Now the opening question. I repeated the same test with the proxy refusing new connections for 2 seconds, with 20 messages 300 ms apart:

 0.1s connected: plain=Yg65VctRomzFyldfz_yI0w stateful=hRxCTixP1e3Cq2wPbelQNA
 1.1s --- network drop (2 s)
 1.1s plain    reconnecting: WebSocket closed with status code: 1006 (no reason given).
 1.1s stateful reconnecting: WebSocket closed with status code: 1006 (no reason given).
 3.1s --- network back
 3.7s stateful reconnected, connectionId SpdjDYTFBTzrdkZ02NG1cA
 3.8s plain    reconnected, connectionId Hmh1dlyTJvUy5iZ3w1QKKw
plain    ids now Hmh1dlyTJvUy5iZ3w1QKKw, received: 1,2,3,4,14,15,16,17,18,19,20
stateful ids now SpdjDYTFBTzrdkZ02NG1cA, received: 1,2,3,4,14,15,16,17,18,19,20

Both clients lost messages 5 to 13, and both got a new connection id. I found the reason in HttpConnection.js of @microsoft/signalr 10.0.11: when the transport closes, stateful reconnect tries to reconnect to the same connection only once, immediately; if that attempt fails, it stops the connection and the job is left to withAutomaticReconnect, i.e. a brand-new connection. The answer to the opening question: no. Stateful reconnect is for short transport drops that recover instantly (a proxy cutting a connection, a cell handover on a mobile network), not for outages lasting seconds. If you need to make up for missed messages in every case, you need an application-level sequence number and a "give me everything after this number" call, just like Last-Event-ID in SSE (Server-Sent Events).

Stateful reconnect only works on the WebSocket transport, and because it keeps a buffer on both sides it has a per-connection memory cost; it can be tuned with StatefulReconnectBufferSize.

Server-to-Client Streaming

public sealed record StockLevel(int BookId, string Title, int Available, DateTimeOffset At);

public sealed class InventoryHub(ILogger<InventoryHub> logger) : Hub
{
    // server -> client stream: cancellationToken fires when the client unsubscribes
    public async IAsyncEnumerable<StockLevel> WatchStock(
        int bookId, int count, [EnumeratorCancellation] CancellationToken cancellationToken)
    {
        int sent = 0;
        try
        {
            for (int i = 0; i < count; i++)
            {
                yield return new StockLevel(bookId, "Clean Code", 100 - i, DateTimeOffset.UtcNow);
                sent++;
                await Task.Delay(100, cancellationToken);
            }
        }
        finally
        {
            logger.LogInformation("WatchStock({BookId}) ended after {Sent} items, cancelled: {Cancelled}",
                bookId, sent, cancellationToken.IsCancellationRequested);
        }
    }
}

When a hub method returns IAsyncEnumerable<T> (the IAsyncEnumerable post of the C# Async series) or ChannelReader<T>, SignalR sends it as a stream: every item is a separate message, followed by a completion message. The difference from Clients.All.SendAsync is that a stream belongs to a request: the client starts it and the client can end it. When the client releases the subscription, the token marked with [EnumeratorCancellation] fires. On the Angular side we wrap the stream in an Observable:

watchStock(bookId: number, count: number): Observable<StockLevel> {
  return new Observable<StockLevel>((subscriber) => {
    const stream = this.connection.stream<StockLevel>('WatchStock', bookId, count).subscribe({
      next: (level) => subscriber.next(level),
      error: (err) => subscriber.error(err),
      complete: () => subscriber.complete(),
    });
    // when the RxJS subscription is released, the stream on the server is cancelled too
    return () => stream.dispose();
  });
}

In the browser, the component watching a 30-item stream showed "Clean Code available: 71 (updates: 30)". When I started a 1,000-item stream from the Node client and released the subscription after 350 ms:

disposed after 4 items

info: Bookstore.Realtime.InventoryHub[0]
      WatchStock(7) ended after 4 items, cancelled: True

The server stopped after the fourth item. When the component is destroyed, takeUntilDestroyed releases the subscription, dispose() sends a cancel message to the server, and the server does not bother producing the remaining 996 items. If you skip this, you both produce and send data the client will never see.

Client-to-Server Streaming

// client -> server stream: ISBN scans from a warehouse handheld
public async Task<int> UploadScans(IAsyncEnumerable<string> isbns)
{
    int count = 0;
    await foreach (string isbn in isbns)
    {
        count++;
        logger.LogInformation("Scanned {Isbn}", isbn);
    }
    return count;
}

const scans = new signalR.Subject();
const upload = c.invoke('UploadScans', scans);
for (const isbn of ['9780132350884', '9780201485677', '9780134494166']) { scans.next(isbn); await wait(50); }
scans.complete();
console.log('UploadScans ->', await upload);

UploadScans -> 3

For a method that takes an IAsyncEnumerable<T> or ChannelReader<T> parameter, the client passes a Subject; every next() goes to the server as an item, complete() ends the loop, and the promise from invoke resolves with the method's return value. It suits streaming the ISBNs (International Standard Book Number) scanned by a warehouse handheld within a single call, without sending one HTTP request each. You also do not hit the 32 KB message limit from the previous post, because each item is a separate message.

The MessagePack Protocol: When?

builder.Services.AddSignalR()
    .AddMessagePackProtocol();

One line on the server; the client chooses it with withHubProtocol(new MessagePackHubProtocol()), and the server supports both protocols at once. I measured the same 50-item stream with both protocols, counting the bytes going from server to client at the proxy:

json        50 items, 6511 bytes server->client, first item: {"bookId":7,"title":"Clean Code","available":100,"at":"2026-11-08T02:44:51.40816+00:00"}
messagepack 50 items, 3009 bytes server->client, first item: {"BookId":7,"Title":"Clean Code","Available":100,"At":["2026-11-08T02:44:56.497Z",0]}

The bytes dropped to less than half. But look at the first item: the property names arrived in PascalCase (BookId, Available) and the DateTimeOffset turned into an array of a date and an offset. Because the JSON (JavaScript Object Notation) protocol uses camelCase, TypeScript code that reads level.available starts reading undefined the moment you switch protocols. With MessagePack, property names travel as they are in C# and date types are encoded differently; if you switch, you have to change the client's interfaces and date conversions together.

When is it worth it? Thousands of small messages per second (live prices, game state, telemetry), binary data (a byte array becomes base64 in JSON and grows by a third), and scenarios where traffic on mobile networks costs money. Where it is not worth it: an app sending a few notifications a minute. There, ease of debugging (being able to read the message in the browser's network tab) is worth more than a few hundred bytes. Also note that a SignalR server compiled with Native AOT (Ahead-of-Time) supports only the JSON protocol.

Slow Clients and Backpressure

Every connection has a send buffer on the server. When a slow client on a mobile network cannot read messages, the buffer fills up. The limits SignalR documents determine what happens: TransportMaxBufferSize (64 KB by default) is the most data the server buffers for a connection, and TransportSendTimeout (10 seconds by default) is the longest it waits for a single send to complete; if that is exceeded, the connection is closed. So a slow client does not consume memory forever; at some point its connection is cut and it has to reconnect and fetch the current state.

In streams, backpressure works naturally through await: the method producing the IAsyncEnumerable produces the next item once the previous one could be sent. If you use ChannelReader, pick a bounded channel; writing to an unbounded channel with a fast producer and a slow consumer silently fills memory. For streams coming from the client, StreamBufferCapacity (10 by default) limits the number of items the server buffers before processing; when the limit is reached, processing of new messages from that connection waits until the server consumes the items.

The practical rule on the broadcast side (Clients.All): send a fast-changing value not on every change but at fixed intervals, with only its latest state. Sending a stock count that changes 50 times a second every 200 ms makes no difference to the user, but cuts the buffer of slow clients to a fifth.

Production Notes

Plan for the reconnect storm. When a server shuts down during a deploy, every client connected to it tries to reconnect at once. A 0 ms first attempt is good, but add jitter to the later ones; and verify with a load test that negotiate and authentication on the server can take that load.

Track connection state as a metric: on the client, send the counts of onreconnecting and onclose to telemetry. "5% of users reconnect every 10 minutes" is the clearest signal of a proxy idle timeout or a load balancer setting.

When you turn on stateful reconnect, update your memory estimate: a buffer of up to 100,000 bytes per connection means around 1 GB of extra memory in the worst case for 10,000 connections.

Common Pitfalls

Not adding withAutomaticReconnect and assuming the connection recovers by itself. The default behaviour is to stay Disconnected after the first drop.

Settling for the default four attempts. A 42-second deploy can exhaust all attempts of the default policy; write a longer policy with jitter.

Taking stateful reconnect as a "no message loss" guarantee. It only works for drops that recover instantly; with an outage of a few seconds you get a new connection and lost messages.

Not rebuilding groups and current state in onreconnected. The new connection is in no group.

Not releasing stream subscriptions. The server keeps producing and sending after the component is closed.

Switching to MessagePack without updating property names and date fields on the client. The code compiles and the data arrives as undefined.

A Small Experiment

★☆☆ In the Angular app, use withAutomaticReconnect() without parameters, stop the server, and measure the time between attempts in the browser console. What happens after the fourth attempt?

★★☆ Add a "watched books" list to InventoryService and restart WatchStock for each of them in onreconnected. Verify by restarting the server that the stream continues by itself.

★★★ Add application-level recovery to the sequence-numbered broadcast: the client keeps the last number it received, calls a hub method named GetMissedSince(lastSeq) in onreconnected, and the server returns the missing ones from a buffer holding the last 1,000 messages. Verify that messages 5–13 now arrive in the 2-second drop test.

In the next post we leave the limits of a single server: why half the messages go missing when a second server is added, sticky sessions, the Redis backplane, Azure SignalR Service, and monitoring SignalR in production.