Graceful Shutdown: SIGTERM, terminationGracePeriodSeconds, preStop and .NET HostOptions

Version note: The examples in this post were run with kind 0.33 on Kubernetes 1.37, kubectl 1.37 and .NET 10 (ASP.NET Core 10.0.14).

Guess First

Three replicas of the Bookstore API (Application Programming Interface) run with readiness probes and maxUnavailable: 0. While a client inside the cluster sends between 700 and 1300 requests a second, you run kubectl rollout restart. How many requests fail: zero, a few, hundreds? The application handles SIGTERM (the termination signal) correctly and the rolling update doesn't reduce capacity; is there any reason for errors? Second question: in a pod with terminationGracePeriodSeconds: 5, what happens if the pod is deleted while a 15-second request is being processed?

What Really Happens When a Pod Is Deleted?

When the delete request reaches the API server, the pod enters the Terminating state and two independent branches start at the same time.

The first branch is on the node: the kubelet sees the pod has been deleted, runs the container's preStop hook if there is one, and when the hook finishes sends SIGTERM to the container's main process. If the process doesn't exit within terminationGracePeriodSeconds (30 seconds by default), SIGKILL (the kill signal) follows. An important detail: the grace period starts before preStop; time spent in preStop comes out of this budget.

The second branch is on the network: the EndpointSlice controller marks the pod's endpoint terminating: true, ready: false, kube-proxy on every node sees the change through its watch and removes the pod from its rules. A Gateway, Envoy or a cloud load balancer gets the same information at its own pace.

There's no synchronization at all between the two branches. SIGTERM can arrive before the rules on every node have been updated. When a .NET application receives SIGTERM, Kestrel stops accepting new connections; but if kube-proxy on another node still routes connections to this pod, those connections get "connection refused". The answer to the first question lies in this race.

The .NET Side: SIGTERM, ApplicationStopping and Ordering

First, let's look at the application's own shutdown. The Bookstore API writes logs in three places: when IHostApplicationLifetime.ApplicationStopping fires, when an 8-second request (/api/slow?seconds=8) finishes, and when a BackgroundService called OrderOutboxWorker stops. When cancelled, the worker spends 3 seconds to simulate sending pending messages:

public sealed class OrderOutboxWorker(ILogger<OrderOutboxWorker> logger, IConfiguration configuration) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        int drainSeconds = configuration.GetValue("Shutdown:WorkerDrainSeconds", 0);
        try
        {
            while (!stoppingToken.IsCancellationRequested)
            {
                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
        }
        catch (OperationCanceledException)
        {
            logger.LogInformation("OrderOutboxWorker stopping, flushing pending messages ({Seconds}s)", drainSeconds);
            await Task.Delay(TimeSpan.FromSeconds(drainSeconds));
            logger.LogInformation("OrderOutboxWorker stopped cleanly");
        }
    }
}

I deleted the pod while the request was in progress (grace period 30 seconds):

04:54:34.484 info: Program[0] Slow request started (8s)
delete 04:54:36
04:54:36.496 info: Microsoft.Hosting.Lifetime[0] Application is shutting down...
04:54:36.496 info: Bookstore.Api[0] ApplicationStopping: SIGTERM received, draining
04:54:42.487 info: Program[0] Slow request finished
04:54:42.500 info: Bookstore.Api.Workers.OrderOutboxWorker[0] OrderOutboxWorker stopping, flushing pending messages (3s)
04:54:45.501 info: Bookstore.Api.Workers.OrderOutboxWorker[0] OrderOutboxWorker stopped cleanly
04:54:45.504 info: Bookstore.Api[0] ApplicationStopped
{"pod":"api-shutdown","seconds":8}

Three things to read. SIGTERM fired ApplicationStopping in the same millisecond; .NET's generic host listens for SIGTERM on Linux by itself, no extra code needed. The in-flight request wasn't cut off; Kestrel waited 6 seconds and the client got the full response. Hosted services stopped in order: first the web server (finishing in-flight requests), then the worker; only then was the worker's stoppingToken cancelled. 9 seconds in total, within the 30-second budget.

HostOptions.ShutdownTimeout and terminationGracePeriodSeconds

This shutdown has a time limit: HostOptions.ShutdownTimeout. Its default is 30 seconds (it was 5 seconds in .NET 5 and earlier); the host stops waiting for hosted services that don't finish within this time. In the Bookstore API we read it from configuration:

builder.Services.Configure<HostOptions>(o =>
    o.ShutdownTimeout = TimeSpan.FromSeconds(builder.Configuration.GetValue("Shutdown:TimeoutSeconds", 30)));

I repeated the experiment with Shutdown__TimeoutSeconds=4 and a 10-second request:

04:55:01.538 info: Program[0] Slow request started (10s)
delete 04:55:03
04:55:03.552 info: Microsoft.Hosting.Lifetime[0] Application is shutting down...
04:55:03.552 info: Bookstore.Api[0] ApplicationStopping: SIGTERM received, draining
04:55:08.558 info: Bookstore.Api[0] ApplicationStopped
04:55:08.558 info: Bookstore.Api.Workers.OrderOutboxWorker[0] OrderOutboxWorker stopping, flushing pending messages (3s)
wget: error getting response: Connection reset by peer

The host waited 4 seconds, then forced the shutdown: the client's connection dropped with "connection reset", the request never finished (no "Slow request finished" log) and the process exited the moment the worker started sending its messages (no "stopped cleanly"). Even though the pod's grace period was 30 seconds, it shut down in 5. Next I shortened the grace period instead: terminationGracePeriodSeconds: 5, ShutdownTimeout at its 30-second default, a 15-second request. The answer to the second question:

delete 04:55:25
04:55:25 running
04:55:30 exit=137 reason=Error finished=2026-11-08T04:55:30Z
wget: error getting response

SIGKILL came after 5 seconds, exit code 137. How long the application wants to wait doesn't matter; the kubelet has the last word. The rule: terminationGracePeriodSeconds > preStop duration + ShutdownTimeout + a few seconds of slack. And ShutdownTimeout should be longer than the application's longest legitimate work (the longest request, the worker's batch). If the order flips, shutdown silently turns into SIGKILL and not even a log line gets written.

The Endpoint Race and preStop Sleep

Now the first question. With the Bookstore API Deployment running (3 replicas, readiness and liveness probes, maxSurge: 1, maxUnavailable: 0), I sent requests to http://api/api/info from the busybox client nonstop for 75 seconds (each request a new connection) and ran kubectl rollout restart in between. Twice:

run 1: ok=66433 fail=26
fail 04:56:03 wget: can't connect to remote host (10.96.24.23): Connection refused
fail 04:56:03 wget: can't connect to remote host (10.96.24.23): Connection refused
fail 04:56:05 wget: download timed out
run 2: ok=51034 fail=23

Why doesn't the readiness probe solve this race? Because readiness dropping depends on the same network branch: the pod is marked ready: false the moment it's deleted; the problem isn't the marking but how long it takes for that information to reach the rules on every node. Some of the errors are "connection refused" (new connections routed to a Kestrel that has stopped), some "download timed out" (connections stuck while the rules were being updated). The EndpointSlice conditions we saw in post 5 matter here: a shutting-down pod can be terminating: true while still serving: true; some Gateway implementations use this to stop sending new requests while letting open connections finish.

Even with everything done "right", every rollout produced 23 to 26 errors. The fix is delaying SIGTERM by a few seconds: let the application keep serving incoming requests while the pod is being removed from the endpoint lists, and start the shutdown once the network side has caught up. The way to do that is a preStop hook. But the Bookstore image is chiseled; there's no sleep in it:

      lifecycle:
        preStop:
          exec:
            command: ["sleep", "5"]

$ kubectl get events --field-selector involvedObject.name=api-shutdown
Normal    Killing             pod/api-shutdown   Stopping container api
Warning   FailedPreStopHook   pod/api-shutdown   PreStopHook failed

The hook failed and the kubelet sent SIGTERM right away; in other words, the classic sleep trick silently doesn't work in a shell-less image. Kubernetes' answer is the sleep action: a preStop where the kubelet itself waits, requiring no binary in the container. It arrived as alpha in 1.29, was enabled by default in 1.30 and became stable in 1.34:

          lifecycle:
            preStop:
              sleep:
                seconds: 5

The same load test, the same rollout, twice:

run 1: ok=95311 fail=0
run 2: ok=97482 fail=0

Zero errors. On the three-node kind cluster, 5 seconds was enough for the network side to catch up. In production you pick this value based on how long every layer the traffic passes through (kube-proxy, the Gateway, a cloud load balancer's health check) takes to drop the pod; 5 to 15 seconds is common. The cost is that every pod shutdown gets that much longer; the total duration of a rolling update grows accordingly. Since Kubernetes 1.33 seconds: 0 is also a valid value, handy for keeping the sleep in the manifest and switching it off per environment.

Long Requests, BackgroundServices and SignalR

Build the budget around your real workload. Bookstore has three kinds of long work. Long HTTP (Hypertext Transfer Protocol) requests (report generation, file uploads): ShutdownTimeout should cover the longest legitimate request, or this work should be moved out of the request into a queue. BackgroundServices: listen to stoppingToken on every loop and, when cancelled, either finish the half-done work or leave it safely (exit without acking the message so another pod picks it up); the idempotent consumer approach from the Background Job and Message Queue series pays off here. SignalR and WebSocket connections: they never end; Kestrel cuts them when ShutdownTimeout runs out. That's unavoidable, so keep automatic reconnect enabled on the client side (SignalR's withAutomaticReconnect) and make sure reconnects spread across the new pods.

If there's special work to do at shutdown (writing a cache to disk, releasing a lock), register on IHostApplicationLifetime.ApplicationStopping or do it inside an IHostedService.StopAsync. You don't need to write your own signal handler with Console.CancelKeyPress or AppDomain.ProcessExit; the generic host does it for you.

Production Notes

Design the three durations together and write them next to each other in the manifest: the preStop sleep (for the network to catch up), ShutdownTimeout (for the application to finish its work) and terminationGracePeriodSeconds (larger than the sum of the two). For Bookstore: 5 + 30 < 45.

Measure, don't guess. The load test above is a one-minute job; repeat it on every new Gateway, new CNI (Container Network Interface) or new managed cluster. Track the 5xx (server error) rate during rollouts on a separate panel in your observability system.

Account for node drains and the autoscaler shutting nodes down too. Pods are terminated the same way not only on deploys but also during node maintenance and when the cluster autoscaler scales in; a PodDisruptionBudget (post 11) limits how many pods can go at once.

Common Pitfalls

Running sleep through exec in a shell-less image. As you saw above, the hook fails, only a Warning event remains and nothing waits. Use the sleep action.

Making the entrypoint a shell wrapper. sh -c "dotnet Bookstore.Api.dll" doesn't forward SIGTERM to the application; the application never sees ApplicationStopping and every shutdown ends in SIGKILL at the end of the grace period (the same problem as the sidecar in post 3).

Keeping the grace period shorter than ShutdownTimeout. SIGKILL arrives while the application is trying to shut down gracefully; no logs, no flush.

Trying to drop readiness by hand at shutdown. Switching the health check to 503 in ApplicationStopping is an old technique; but when SIGTERM arrives the pod is already being removed from endpoints, and the real problem is the delay in rule updates. preStop sleep addresses that directly.

A Small Experiment

★☆☆ Set Shutdown__WorkerDrainSeconds=40 and delete the pod. Can you see the worker's "stopped cleanly" log with the default 30-second grace period? Which two settings do you need to change to see it?

★★☆ Repeat the load test with a preStop sleep of 1, 2 and 5 seconds and put the error counts in a table. Find the smallest value at which errors drop to zero on your own cluster.

★★★ Replace the test client with a .NET console application that sends every request over the same connection (keep-alive). Count errors in rollouts with and without preStop; use the logs to explain how keep-alive connections behave at shutdown (Kestrel sending Connection: close on open connections).

This post completes the series' foundation block: pods start correctly, receive the right traffic, run at the right size and shut down cleanly. In the next post we move on to production topics: horizontal scaling with the HPA (Horizontal Pod Autoscaler), the VPA (Vertical Pod Autoscaler), in-place resize, scaling on queue length with KEDA (Kubernetes Event-driven Autoscaling) and PodDisruptionBudgets.