Prerequisites for Zero Downtime: Graceful Shutdown, Readiness, Stateless Apps and Backward Compatibility

Version note: The examples in this post were run with .NET 10 (SDK 10.0.402), Angular 22.2, Nginx 1.30 and Docker 29.

Guess First

A request that takes five seconds arrives at the Bookstore API (Application Programming Interface). One second after it starts, you send docker stop to the container. What does the client see: a 200, a 502, or a dropped connection? And if you run the same experiment against an app whose HostOptions.ShutdownTimeout is lowered to two seconds, does the answer change?

The second question is about Angular. A user opened the page in the morning, you deployed a new version at noon and deleted the old JavaScript files. Without refreshing the tab, the user clicks the "Books" link. What happens on screen? Write both answers down; we will see both with real runs.

Is the App Ready Before You Pick a Strategy?

In the first post of the series we put rolling, blue-green and canary on a map. They all share an assumption: an instance that is shutting down finishes the work it already has, a new instance tells the truth about being ready for traffic, a user gets the same result whichever instance they land on, and the old and new versions can run side by side for a while. If any of these four assumptions fails, even the most expensive blue-green setup shows errors to users. In this post we measure all four on Bookstore: graceful shutdown, readiness, being stateless and backward compatibility.

Graceful Shutdown: What Happens to In-Flight Requests?

Let's try the defaults first. The /api/slow?ms=5000 endpoint waits five seconds and then responds. We send the request and stop the container one second later:

docker run -d --name depwork-api-a --stop-timeout 40 -p 18610:8080 depwork-bookstore-api:1.0.0
curl -s -w ' -> %{http_code} in %{time_total}s\n' "localhost:18610/api/slow?ms=5000" &
sleep 1
docker stop depwork-api-a

What the client sees:

{"version":"1.0.0","instance":"ee49ca47b125","waitedMs":5000} -> 200 in 5.078387s

The shutdown sequence in the app log:

info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
info: Bookstore.Api.DrainOnShutdown[0]
      SIGTERM received, readiness is now Unhealthy, draining for 0s

The request completed, docker stop returned in four seconds and the exit code was 0. Behind the scenes Docker sends SIGTERM (the terminate signal) to the main process in the container. The .NET generic host catches it and starts StopAsync. Kestrel first stops accepting new connections, then waits for the requests it already has. How long it waits is set by HostOptions.ShutdownTimeout; the default has been 30 seconds since .NET 6 (TimeSpan.FromSeconds(30) in the dotnet/runtime source).

Now the same experiment with a two-second ShutdownTimeout. We also turn on Kestrel's debug logs:

docker run -d --name depwork-api-a --stop-timeout 40 -p 18610:8080 \
    -e Shutdown__TimeoutSeconds=2 \
    -e Logging__LogLevel__Microsoft.AspNetCore=Debug \
    depwork-bookstore-api:1.0.0

 -> 000 in 3.030439s

dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[16]
      Some connections failed to close gracefully during server shutdown.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[8]
      Connection id "0HNPT90568R56" sending RST because: "The connection was aborted because the server is shutting down and request processing didn't complete within the time specified by HostOptions.ShutdownTimeout."
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:18610/api/slow?ms=5000 - 499 - - 3035.1442ms

The client saw the connection drop after 3 seconds (curl's 000 means "no HTTP (Hypertext Transfer Protocol) response"). Kestrel waited two seconds, then sent an RST (reset) packet on the connection and spelled out the reason in its message. The server-side log closes the request with 499. So don't set ShutdownTimeout shorter than your longest legitimate request. If a report export takes 40 seconds, the 30-second default cuts it as well.

The Orchestrator's Patience Is a Setting Too

I found out that ShutdownTimeout alone isn't enough while preparing this experiment. On the first run I hadn't passed --stop-timeout, and the request still dropped with 000. When I looked at the container I saw this:

docker inspect -f 'stoptimeout={{.Config.StopTimeout}} signal={{.Config.StopSignal}}' depwork-api-a
docker inspect -f '{{.State.ExitCode}}' depwork-api-a

stoptimeout=1 signal=
137

137 means 128 + 9, so the process was killed with SIGKILL (the kill signal). According to the Docker documentation the default wait for Linux containers is 10 seconds. Docker Desktop on this machine, however, had written 1 second into the containers. However cleanly the app shuts down, it doesn't matter if whatever runs it pulls the plug after one second. The rule is simple: the orchestrator's wait (--stop-timeout in Docker, TimeoutStopSec in systemd, terminationGracePeriodSeconds which we will see in the upcoming Kubernetes series) must be at least the drain time plus ShutdownTimeout. Write that value explicitly in your deploy scripts; don't trust the default.

Readiness: Let the App Say When It Takes Traffic

It isn't enough for a stopping instance to finish its work; the load balancer also has to stop sending it new requests. The reverse is true for a starting instance: the process being up doesn't mean it's ready for traffic. In Bookstore, requests that arrive before the catalog cache is loaded pile up on the database. That's why there are two separate health endpoints. We covered the mechanics in the ASP.NET Core Web API series post "Health Checks in ASP.NET Core Web API: Liveness, Readiness and Database Checks"; here we use them from the deploy angle:

builder.Services.AddSingleton<AppState>();
builder.Services.AddHostedService<CatalogWarmup>();
builder.Services.AddHostedService<DrainOnShutdown>();
builder.Services.Configure<HostOptions>(options =>
    options.ShutdownTimeout = TimeSpan.FromSeconds(builder.Configuration.GetValue("Shutdown:TimeoutSeconds", 30)));
builder.Services.AddHealthChecks()
    .AddCheck<ReadinessCheck>("ready", tags: ["ready"]);

// ...

app.MapHealthChecks("/health/live", new() { Predicate = _ => false });
app.MapHealthChecks("/health/ready", new() { Predicate = check => check.Tags.Contains("ready") });

public sealed class ReadinessCheck(AppState state) : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        if (state.Draining)
        {
            return Task.FromResult(HealthCheckResult.Unhealthy("draining"));
        }
        return Task.FromResult(state.CatalogLoaded
            ? HealthCheckResult.Healthy()
            : HealthCheckResult.Unhealthy("catalog cache is warming up"));
    }
}

public sealed class DrainOnShutdown(AppState state, IConfiguration configuration, ILogger<DrainOnShutdown> logger) : IHostedLifecycleService
{
    public async Task StoppingAsync(CancellationToken cancellationToken)
    {
                // SIGTERM arrived: readiness goes down first, Kestrel keeps accepting requests meanwhile
        state.Draining = true;
        int seconds = configuration.GetValue("Shutdown:DrainSeconds", 0);
        logger.LogInformation("SIGTERM received, readiness is now Unhealthy, draining for {Seconds}s", seconds);
        await Task.Delay(TimeSpan.FromSeconds(seconds), cancellationToken);
    }

    public Task StartingAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StartedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    public Task StoppedAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}

DrainOnShutdown is an IHostedLifecycleService. The order in .NET 10's Host.StopAsync is: first every StoppingAsync, then ApplicationStopping, then StopAsync of the hosted services, Kestrel included. So while we wait inside StoppingAsync, Kestrel is still open and accepting requests, but /health/ready now returns 503. The load balancer's health check sees this and takes the instance out of the pool, and the in-flight requests finish in the meantime. The Kubernetes counterpart is the preStop hook; its mechanics belong to the upcoming Kubernetes series.

I started the container with WARMUP_SECONDS=3 and Shutdown__DrainSeconds=5 and polled both endpoints every half second. The output, shortened:

06:26:25 ready=Unhealthy live=Healthy
06:26:26 ready=Unhealthy live=Healthy
06:26:27 ready=Unhealthy live=Healthy
06:26:28 ready=Healthy live=Healthy
06:26:29 ready=Healthy live=Healthy
06:26:30 ready=Unhealthy/503 api=200
06:26:31 ready=Unhealthy/503 api=200
06:26:32 ready=Unhealthy/503 api=200
06:26:33 ready=Unhealthy/503 api=200
06:26:34 ready=Unhealthy/503 api=200
06:26:35 docker stop returned
06:26:35 ready=/000 api=000

For the first three seconds the process is up (live=Healthy) but readiness is Unhealthy: the catalog cache is loading. The moment SIGTERM arrives readiness switches to 503, while the API keeps returning 200 for five more seconds. Only after the drain does the port close. Choose a drain longer than the time your load balancer needs to declare the instance unhealthy. If the health check runs every 5 seconds and requires two failures, a drain shorter than 10 seconds is pointless.

One thing to watch: the drain eats into ShutdownTimeout. The host creates a single token with CancelAfter(ShutdownTimeout) and StoppingAsync receives that token too. If you spend 10 seconds of a 30-second timeout on draining, 20 seconds remain for in-flight requests.

Being Stateless: Whichever Instance the User Lands On

During a rolling or canary deploy, two requests from the same user can go to two different instances. Anything you keep in an instance's memory (session, a cart in IMemoryCache, uploads written to local disk) gets lost or becomes inconsistent at that moment. The sneakiest example is ASP.NET Core Data Protection, because the framework uses it, not you: authentication cookies, antiforgery tokens and anything you encrypt with IDataProtector depend on the keys in the key ring. I added two endpoints to Bookstore that create and read a cart token, started two containers, took a token from A and sent it to B:

token=CfDJ8GsWHk8MPAVCmtIEpPw0vezQGo...
A: cart:42
B: 500

warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNPT90M6QESD", Request id "0HNPT90M6QESD:00000001": An unhandled exception was thrown by the application.
      System.Security.Cryptography.CryptographicException: The key {4f1e166b-3c0c-4205-9ad2-04a4fc34bdec} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning

Each container created its own key ring under /root/.aspnet/DataProtection-Keys, and the framework warned about it at startup. B can't decrypt what A encrypted. In real life this means users randomly logged out during a rolling deploy, or antiforgery errors on form posts. It also continues after the deploy, because every new container starts with a new key ring. The fix is to keep the keys somewhere all instances share and to pin the application name:

IDataProtectionBuilder dataProtection = builder.Services.AddDataProtection();
string? keysPath = builder.Configuration["DataProtection:KeysPath"];
if (keysPath is not null)
{
    dataProtection
        .PersistKeysToFileSystem(new DirectoryInfo(keysPath))
        .SetApplicationName("bookstore");
}

docker volume create depwork-dpkeys
docker run -d --name depwork-api-a -p 18610:8080 -v depwork-dpkeys:/keys \
    -e DataProtection__KeysPath=/keys depwork-bookstore-api:1.0.0
docker run -d --name depwork-api-b -p 18611:8080 -v depwork-dpkeys:/keys \
    -e DataProtection__KeysPath=/keys depwork-bookstore-api:1.0.0

A: cart:42
B: cart:42 (200)

In production you'd use Redis, a database or Azure Blob Storage instead of a shared file system, and encrypt the keys with ProtectKeysWith...; the "No XML (Extensible Markup Language) encryptor configured" warning in the logs is a reminder of that. The details are in the Web Security (OWASP Top 10) series post "OWASP A04: Cryptographic Failures (2) — Password Storage, Data Protection and Secret Management". The same logic applies to sessions and caches: put sessions in a distributed cache, write uploads to object storage rather than instance disk. Don't treat sticky sessions as the solution; at best they're a band-aid. When the instance goes away, the user you stuck to it is lost anyway.

N and N-1 Must Run at the Same Time

Every strategy except recreate runs two versions side by side for a while. That window can be minutes in a rolling deploy, hours in a canary, months for an API with a mobile app. During that window three contracts must stay backward compatible:

• The HTTP API contract: the new version may add fields but can't remove an old one or change its meaning. A client (Angular, mobile, another service) may still be talking to the old version.
• Message schemas: the new consumer must read messages already waiting in the queue, and the old consumer must read messages from the new producer. The tolerant reader from the Microservices series post "Service Contracts and Versioning" is a must here.
• The database schema: old code must run on the new schema, new code on the old one. We'll see this in detail with expand-contract in the sixth post of the series.

A practical test: before merging a pull request, ask "while this is being deployed, what happens to a request or message coming from the old version?" If the answer is "it gets a 500", the change needs to be split in two.

Angular-Specific: Deleted Chunks and ChunkLoadError

The browser is an "old version instance" too. The user's open tab runs the main-*.js from before the deploy and knows the chunk names of the lazy routes from that build. In Bookstore's Angular 22 app, /books is lazy loaded. I published v1 on Nginx, opened it in the browser, then deployed v2 and deleted the old files. The Nginx config is the classic SPA (Single Page Application) fallback:

server {
    listen 80;
    root /usr/share/nginx/html;

    location / {
        try_files $uri $uri/ /index.html;
    }
}

When I clicked "Books" without refreshing the tab, the browser console showed:

[error] Failed to load module script: Expected a JavaScript-or-Wasm module script but the server responded with a MIME type of "text/html". Strict MIME type checking is enforced for module scripts per HTML spec.
[error] ERROR TypeError: Failed to fetch dynamically imported module: http://localhost:18603/chunk-MEEZWYMZ.js

curl -s -o /dev/null -w '%{http_code} %{content_type}\n' localhost:18603/chunk-MEEZWYMZ.js

200 text/html

The old chunk is gone and try_files returns index.html with 200 instead. The browser expects JavaScript, gets HTML (HyperText Markup Language), and fails the MIME (Multipurpose Internet Mail Extensions) type check. For the user, the link just seems to do nothing. There are three layers to fixing this. First, turn off the fallback for hashed files so they return a real 404, and separate the caching rules:

server {
    listen 80;
    root /usr/share/nginx/html;

    location = /index.html {
        add_header Cache-Control "no-cache";
    }

    location ~* \.(js|css)$ {
        try_files $uri =404;
        add_header Cache-Control "public, max-age=31536000, immutable";
    }

    location / {
        try_files $uri $uri/ /index.html;
        add_header Cache-Control "no-cache";
    }
}

Second, don't delete the previous version's hashed files during a deploy. Copy the new build over the old files so only index.html changes, and clean up old chunks a release or two later. Because file names carry a content hash, nothing collides. Third, if a chunk is still missing, let the app recover by itself. The Angular router's navigation error handler helps here:

import { ApplicationConfig, provideBrowserGlobalErrorListeners } from '@angular/core';
import { provideRouter, withNavigationErrorHandler } from '@angular/router';
import { routes } from './app.routes';

const reloadKey = 'chunk-reload-at';

export const appConfig: ApplicationConfig = {
  providers: [
    provideBrowserGlobalErrorListeners(),
    provideRouter(
      routes,
      withNavigationErrorHandler(({ error, url }) => {
        const message = error instanceof Error ? error.message : String(error);
        if (!message.includes('Failed to fetch dynamically imported module')) {
          return;
        }
                // a new version is live: load the target URL as a full page, never twice within 10 s
        const last = Number(sessionStorage.getItem(reloadKey) ?? 0);
        if (Date.now() - last > 10_000) {
          sessionStorage.setItem(reloadKey, String(Date.now()));
          location.assign(url);
        }
      }),
    ),
  ],
};

I opened v3, deployed v4, and this time deliberately deleted the old chunks. On the click the console showed:

[error] Failed to load resource: the server responded with a status of 404 (Not Found)
[error] ERROR TypeError: Failed to fetch dynamically imported module: http://localhost:18603/chunk-MBQB7IDL.js

Right after that the handler loaded /books as a full page and "Catalog v4" appeared on screen. The timestamp in sessionStorage prevents an endless reload loop: if the chunk is really broken, it stops at the second attempt. If you use a service worker (@angular/service-worker), you build the same idea with SwUpdate.versionUpdates: when VERSION_READY arrives you tell the user "a new version is available" and ask them to reload, and on the unrecoverable event you reload the page. Pages using SignalR face the same reality: the connection drops when the server goes away. The JavaScript client's withAutomaticReconnect() retries after 0, 2, 10 and 30 seconds by default, but restoring group memberships after a reconnect is your code's job.

In Production

All of these prerequisites are measurable during a deploy. Watch four numbers on every deploy: the 5xx count in the deploy window (from load balancer logs), the number of 499s or dropped connections (whether graceful shutdown was enough), the time a new instance takes from start to readiness (warm-up), and 404 requests for .js files in the first hour after the deploy (the server-side trace of ChunkLoadErrors). If any of these isn't zero, the problem isn't the strategy but the app's readiness.

The order of durations in the shutdown chain must be the same in every environment: health check interval × failure threshold < drain time < drain + ShutdownTimeout < the orchestrator's wait. Writing this chain down once and comparing it with each environment's settings ends many "a few requests fail during deploys" complaints.

Common Pitfalls

• Assuming "we use Kubernetes, so zero downtime is automatic". Kubernetes puts a new pod into the service based on readiness and sends SIGTERM to the old one, and that's it. Making readiness meaningful, draining on SIGTERM and finishing in-flight requests is the app's job. Because removing the endpoint from the service and SIGTERM travel at the same time, a short drain delay is needed in Kubernetes too.
• Wiring liveness and readiness to the same endpoint. If liveness fails when the database slows down, the orchestrator kills healthy processes and the problem grows.
• Raising ShutdownTimeout and forgetting the orchestrator's timeout. The 137 above was exactly that.
• Leaving Data Protection keys inside the container. Every deploy logs users out, and nobody connects it to the deploy.
• Deleting the target folder and copying the Angular build from scratch. Every open tab breaks on its next lazy route.
• Giving index.html a long cache lifetime. If a CDN (Content Delivery Network) or the browser keeps the old index.html, users don't see the deploy for hours.

A Small Experiment

★☆☆ Set Shutdown__TimeoutSeconds to 10 and stop the container with docker stop -t 30 during a /api/slow?ms=12000 request. What does the client see, and what does Kestrel log? Then repeat with -t 5 and look at the exit code.

★★☆ Put two instances running with Shutdown__DrainSeconds=5 behind Nginx and send ten requests per second while stopping one of them. Open source Nginx doesn't do active health checks, so how does the readiness 503 affect Nginx? If you see no errors, look for the reason in the proxy_next_upstream documentation.

★★★ Add @angular/service-worker to the Angular app, write a service that listens to SwUpdate.versionUpdates, and actually trigger the "new version ready" notification after a deploy. Why does navigating to a lazy route in the old tab work this time?

The app now finishes its work when shutting down, says when it's ready, and behaves the same whichever instance you land on. In the next post we build the first real strategy on top of that: a rolling deployment that updates three instances behind Nginx one at a time, and the maxSurge and maxUnavailable decision in Kubernetes.