Kubernetes Health Probes: Startup, Liveness, Readiness and ASP.NET Core Health Checks

Version note: The examples in this post were run with kind 0.33 on Kubernetes 1.37, kubectl 1.37, .NET 10 (ASP.NET Core 10.0.14), Npgsql 10.0 and PostgreSQL 18.

Guess First

The Bookstore API (Application Programming Interface) has two separate Deployments. Both have readiness probes that check PostgreSQL. The first one's liveness probe goes to /health/live, which only checks that the process is up; the second one's goes to /health, which also checks the database. PostgreSQL goes away for two minutes. By the end of those two minutes, how many times have each Deployment's pods restarted? Second question: what happens if you give an application that warms up for 40 seconds at startup a liveness probe with periodSeconds: 5 and failureThreshold: 3?

Three Probes, Three Different Questions

The kubelet can ask three different questions of every container and reacts differently to each answer:

Readiness runs for the pod's whole life; a pod can move back and forth between ready and not ready. As we saw in post 5, a pod whose readiness drops is pulled out of traffic but keeps living. Liveness makes an irreversible decision: killing the process. The probes' timing fields and defaults, according to the cluster's own schema (kubectl explain pod.spec.containers.livenessProbe): periodSeconds 10, timeoutSeconds 1, failureThreshold 3, successThreshold 1 (must be 1 for liveness and startup), initialDelaySeconds 0.

The ASP.NET Core Side: Tagged Health Checks

The mechanics of health checks are in the Health Check post of the ASP.NET Core Web API series; here we only build the separation Kubernetes needs. We give every check a tag and open a separate endpoint per probe that runs only the relevant tags:

builder.Services.AddHealthChecks()
    .AddCheck<LivenessSwitchCheck>("self", tags: ["live"])
    .AddCheck<DatabaseHealthCheck>("database", tags: ["ready"]);

app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = c => c.Tags.Contains("live") });
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = c => c.Tags.Contains("ready") });
app.MapHealthChecks("/health");

LivenessSwitchCheck looks at a flag inside the process; shortly we'll break that flag through a diagnostics endpoint to simulate a real-world deadlock. DatabaseHealthCheck connects to PostgreSQL with a 2-second timeout and runs SELECT 1. /health, with no predicate, runs every check. The three probes in the Deployment:

          startupProbe:
            httpGet:
              path: /health/live
              port: http
            periodSeconds: 2
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /health/live
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /health/ready
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 2

Startup and liveness hit the same endpoint but ask different questions: startup asks "have you started?" often and patiently (every 2 seconds, 30 attempts, a 60-second budget), liveness asks "are you still alive?" rarely and decides quickly. Readiness goes to a separate endpoint because it asks a separate question. The three endpoints from inside the cluster:

/health/live -> Healthy  HTTP/1.1 200 OK
/health/ready -> Healthy  HTTP/1.1 200 OK
/health ->   HTTP/1.1 200 OK Healthy

Slow Startup and the startupProbe

Real applications don't always start in a second: building the EF (Entity Framework) Core model for the first time, warming a large cache, reading from a remote configuration source. In the Bookstore API we simulate this with the Startup__WarmupSeconds setting: the application waits for the given time before opening its port. I ran a 40-second warm-up with only a liveness probe (periodSeconds: 5, failureThreshold: 3):

+0s 0/1 ContainerCreating 0
+4s 1/1 Running 0
+16s 1/1 Running 1 (1s ago)
+33s 1/1 Running 2 (3s ago)
+49s 1/1 Running 3 (4s ago)
+61s 0/1 CrashLoopBackOff 3 (1s ago)
+85s 1/1 Running 4 (25s ago)
+98s 0/1 CrashLoopBackOff 4 (2s ago)
+155s 1/1 Running 5 (59s ago)
$ kubectl get events --sort-by=.lastTimestamp
Normal    Killing       pod/api-slow-7b8669f69b-568zt   Container api failed liveness probe, will be restarted
Warning   Unhealthy     pod/api-slow-7b8669f69b-568zt   Liveness probe failed: Get "http://10.244.1.27:8080/health/live": dial tcp 10.244.1.27:8080: connect: connection refused

The answer to the second question: the application never manages to start. Liveness gives up after 15 seconds (3 × 5) and kills the container, the application starts over before finishing its warm-up, and with CrashLoopBackOff delays in between it ends up in an endless loop. Note that the READY column mostly shows 1/1: this Deployment has no readiness probe, so a container that hasn't even opened its port counts as "ready". A second bug.

The classic fix was giving liveness initialDelaySeconds: 60. The problem: those 60 seconds are waited out at every container start, even if the application starts in 5 seconds; and if startup takes 61 seconds one day, the loop is back. startupProbe separates the two. I added a startup probe with periodSeconds: 2, failureThreshold: 45 (a 90-second budget) to the same Deployment:

+0s 0/1 ContainerCreating 0
+4s 0/1 Running 0
+45s 1/1 Running 0
$ kubectl get events --sort-by=.lastTimestamp
Warning   Unhealthy     pod/api-slow-7cbb485f6-gcdcj    Startup probe failed: Get "http://10.244.1.28:8080/health/live": dial tcp 10.244.1.28:8080: connect: connection refused

The startup probe failed for 40 seconds, but since its budget wasn't exhausted nobody killed the container; on the first successful response liveness took over and the pod was ready at second 45, with zero restarts. Keep the budget comfortably above the longest startup you've observed; liveness meanwhile keeps deciding quickly, independent of startup.

Putting the Database in Liveness: a Restart Cascade

Now the first question. The api Deployment's liveness goes to /health/live, the liveness of its copy api-badlive goes to /health. Both have two replicas. I scaled PostgreSQL to zero and recorded the pods every four seconds (columns: Deployment, READY, STATUS, RESTARTS):

postgres down at 04:42:11
04:42:11 api 1/1 Running 0 | api 1/1 Running 0 | api-badlive 1/1 Running 0 | api-badlive 1/1 Running 0 |
04:42:20 api 1/1 Running 0 | api 0/1 Running 0 | api-badlive 0/1 Running 0 | api-badlive 0/1 Running 0 |
04:42:24 api 0/1 Running 0 | api 0/1 Running 0 | api-badlive 0/1 Running 0 | api-badlive 0/1 Running 0 |
04:42:36 api 0/1 Running 0 | api 0/1 Running 0 | api-badlive 0/1 Running 1 | api-badlive 0/1 Running 1 |
04:43:08 api 0/1 Running 0 | api 0/1 Running 0 | api-badlive 0/1 Running 2 | api-badlive 0/1 Running 2 |
04:43:36 api 0/1 Running 0 | api 0/1 Running 0 | api-badlive 0/1 Running 3 | api-badlive 0/1 Running 3 |
04:44:09 api 0/1 Running 0 | api 0/1 Running 0 | api-badlive 0/1 Running 4 | api-badlive 0/1 Running 4 |
$ kubectl describe pod api-badlive-76f9678986-gptqn
  Normal   Killing    15s (x4 over 105s)   kubelet            spec.containers{api-badlive}: Container api-badlive failed liveness probe, will be restarted

In two minutes the api pods restarted zero times, the api-badlive pods four times each. All four pods were pulled out of traffic within 10 seconds; that's readiness doing its job and it's the same in both Deployments. The difference is in liveness: with no database, every api-badlive container was killed and restarted every 30 seconds. These restarts fix nothing, because the problem isn't in the pod. Worse, they do harm: every restart throws away connection pools, caches and in-flight requests, when the database comes back dozens of pods start cold at the same time and pile onto it, and as the restart count climbs CrashLoopBackOff delays slow down recovery. The Kubernetes documentation also recommends that liveness probes check only the process itself, not its dependencies.

Looking at the three endpoints from inside the pod makes the distinction clear:

/health/live -> Healthy  HTTP/1.1 200 OK
/health/ready ->   HTTP/1.1 503 Service Unavailable wget: server returned error: HTTP/1.1 503 Service Unavailable
/health ->   HTTP/1.1 503 Service Unavailable wget: server returned error: HTTP/1.1 503 Service Unavailable

Then there's the readiness decision: is pulling every replica out of traffic the right call when the database is gone? In Bookstore the book list can't work without the database, but /api/info or a catalogue served from cache could. A pod whose readiness drops produces "connection refused" instead of a 503 (we saw that in post 5); if the application can stay up and return a meaningful 503 or a partial answer, leaving the database out of readiness and handling the error inside the application can give a better user experience. The rule: put into readiness only dependencies the pod can't solve on its own and without which it can't give a meaningful answer to any request.

When Is Liveness Useful?

Liveness's real job is catching situations where the process lives but will never recover by itself: a deadlock, an exhausted thread pool, corrupted internal state. To simulate that I broke the liveness flag in one pod:

04:44:30 break
{"pod":"api-65986689d6-n9fzz","live":false}
04:44:58 1/1 Running 0 3m
04:45:01 1/1 Running 1 (3s ago)
  Normal   Killing    33s                   kubelet            spec.containers{api}: Container api failed liveness probe, will be restarted

28 seconds later the container restarted and came back with a clean process. Detection time is roughly periodSeconds × failureThreshold, i.e. 10 × 3 = 30 seconds. Meanwhile the pod kept showing 1/1, because readiness only looks at the database; in a real deadlock HTTP (Hypertext Transfer Protocol) requests wouldn't get answers either, so readiness would drop too.

Doing the Math: period, timeout and failureThreshold

Probe settings aren't just numbers, they're decisions. The question to ask is different for each:

• Startup budget = periodSeconds × failureThreshold. At least twice the slowest startup you've observed (under load, on a cold node).
• Liveness detection time ≈ periodSeconds × failureThreshold (+ the last attempt's timeoutSeconds). How long are you willing to let a stuck pod occupy capacity? Make it too short and transient slowdowns like a GC (Garbage Collector) pause or a brief CPU (Central Processing Unit) squeeze cause restarts.
• Readiness reaction time ≈ periodSeconds × failureThreshold. You can be more aggressive here, because the cost of a false alarm is only leaving traffic temporarily.
• timeoutSeconds defaults to 1 second. In a .NET pod with a CPU limit that gets throttled (post 9), one second is easily exceeded; even with lightweight health endpoints, 2–3 seconds is more realistic.

gRPC Probes and a Separate Management Port

Besides HTTP there are three more probe types: tcpSocket (is the port open), exec (a command inside the container, like pg_isready for PostgreSQL) and grpc. The gRPC (gRPC Remote Procedure Calls) probe uses the standard gRPC Health Checking Protocol and has been stable since 1.27; in ASP.NET Core the Grpc.AspNetCore.HealthChecks package wires that service to your health checks. In shell-less chiseled images exec probes aren't an option anyway; use HTTP or gRPC.

Serving health endpoints on the same port as the application's traffic raises two questions: are they exposed to the internet through the Gateway, and does the application's authentication middleware block probe requests? A common solution is a separate management port: opening Kestrel on two ports with ASPNETCORE_HTTP_PORTS="8080;8081", binding the health endpoints only to 8081 with RequireHost("*:8081"), and giving only 8080 to the Service and the Gateway. Since probes go straight to the container port, you can probe a port that isn't defined on the Service.

Production Notes

Not defining liveness is also an option. When the process crashes, the kubelet restarts it even without liveness; liveness is only for "alive but useless" states. If the application can't reliably detect such a state, a bad liveness probe is more dangerous than none.

Keep health endpoints lightweight. Every probe is an HTTP request; a readiness probe that queries the database every 5 seconds means 10 queries per second across 50 replicas. Cache expensive checks, or run them periodically in the background and read the result.

Monitor probe failures. Unhealthy events and the RESTARTS counter should go to your metrics system; when a restart cascade begins, you want to see it before your users do. The measurement side is in the upcoming Observability series.

Common Pitfalls

Giving the same endpoint to all three probes. Especially giving liveness a /health that also checks the database is exactly the restart cascade above.

Not defining a readiness probe. As you saw in the slow-startup experiment, without readiness a container that hasn't even opened its port counts as ready, and a rolling update sends it traffic.

Trying to solve slow startup with initialDelaySeconds. You wait for nothing on every start, and one day the estimate isn't enough. Use startupProbe.

Leaving probes behind authentication. If a global authorization policy returns 401 on the health endpoints, the pod never becomes ready. Use AllowAnonymous on health endpoints or move them to a separate port.

A Small Experiment

★☆☆ Switch the api Deployment's readiness to /health/live, scale PostgreSQL to zero and see what a /api/books request returns instead of "connection refused". Which is better for the user?

★★☆ Give liveness timeoutSeconds: 1 and measure whether the pod restarts while you keep its CPU busy with /api/diag/burn?seconds=30 (after giving the pod limits.cpu: 200m).

★★★ Move the health endpoints to port 8081: open two ports with ASPNETCORE_HTTP_PORTS, use MapHealthChecks(...).RequireHost("*:8081"), point the probes at 8081 and verify that /health/ready through the Service now returns 404.

Probes tell Kubernetes the application's state. But for Kubernetes to decide where to place a pod and how many resources to set aside, it needs other information. In the next post we measure resource requests and limits, QoS (Quality of Service) classes, OOMKilled, CPU throttling and how the .NET GC behaves in a container.