Choosing Hosting and a Production Checklist: Nginx, IIS, Caddy, Containers and PaaS

Version note: The examples in this post were run with .NET 10 (ASP.NET Core runtime 10.0.14), Nginx 1.30.6, Caddy 2.11.9 and curl 8.23 on Docker 29; IIS (Internet Information Services), Traefik and cloud services weren't run for this post.

Guess First

By the end of the seventh post Bookstore's Nginx config was ready: TLS (Transport Layer Security), HTTP (Hypertext Transfer Protocol) 2 and 3, security headers, rate limiting, load balancing across three Kestrels and a passive health check. Before go-live I wrote a ten-item check script and ran it against that config.

Guess: how many of the ten items pass? We tried every piece one by one over seven posts; could something still have slipped through?

The Options: Who Takes On What?

The split of roles from the first post of the series becomes a decision tool here. Every hosting option is a different set of answers to the same questions: who terminates TLS, who renews the certificate, who keeps the process running, who spreads the load, who takes unhealthy instances out?

The right-hand column of the table matters most: the right choice is often about the team, not the technology. If nobody can read the Nginx config, even the best Nginx setup is a black box during a night-time outage. Likewise, handing a Linux VM to an operations team that only runs Windows just moves the problem.

Caddy: the Shortest Path with Automatic HTTPS

Caddy is a web server and reverse proxy with secure defaults: give it a domain name and it obtains and renews the certificate and redirects HTTP to HTTPS (HTTP Secure) by itself. The whole Caddyfile I put in front of Bookstore's two Kestrel instances:

bookstore.test {
    tls internal
    reverse_proxy 172.30.85.21:8080 172.30.85.22:8080 {
        lb_policy round_robin
        health_uri /health
        health_interval 5s
    }
}

tls internal tells Caddy to issue the certificate from its own local CA (Certificate Authority); needed for a domain that isn't public on the internet, such as .test. On a real domain, drop that line and Caddy gets the certificate from Let's Encrypt or ZeroSSL. The startup log:

{"level":"info","ts":1794113277.2935407,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"bookstore.test","issuer":"local"}

Its behaviour:

$ curl -s -o /dev/null -D - http://bookstore.test/api/books | grep -E "^HTTP|Location"
HTTP/1.1 308 Permanent Redirect
Location: https://bookstore.test/api/books
$ curl -s -o /dev/null -D - https://bookstore.test/api/books | grep -i -E "^HTTP|server|alt-svc|via"
HTTP/2 200
alt-svc: h3=":443"; ma=2592000
server: Kestrel
via: 1.1 Caddy
$ for i in 1 2 3 4; do curl -s https://bookstore.test/api/whoami | jq -c "{instance,remoteIp,scheme,host}"; done
{"instance":"api1","remoteIp":"172.30.85.50","scheme":"https","host":"bookstore.test"}
{"instance":"api2","remoteIp":"172.30.85.50","scheme":"https","host":"bookstore.test"}
{"instance":"api1","remoteIp":"172.30.85.50","scheme":"https","host":"bookstore.test"}
{"instance":"api2","remoteIp":"172.30.85.50","scheme":"https","host":"bookstore.test"}

Seven lines of config brought an HTTP-to-HTTPS redirect, HTTP/2, HTTP/3 (alt-svc), round robin and correct forwarded headers; the app saw the client IP (Internet Protocol) address and https (on the Kestrel side the KnownIPNetworks setting from the third post is still needed). Two differences stand out. Caddy passes the upstream's Server: Kestrel header through as is; to hide it you have to do it in the config or set AddServerHeader = false in Kestrel. Second, Caddy does active health checks, which open-source Nginx lacks; when I stopped api2:

{"level":"info","ts":1794113300.2571716,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"172.30.85.22:8080","error":"Get \"http://172.30.85.22:8080/health\": dial tcp 172.30.85.22:8080: i/o timeout"}
api1 200
api1 200
api1 200
api1 200

Caddy called /health every five seconds and took the unhealthy instance out itself; real requests never went to it. The readiness endpoint from the "Health Check" post of the ASP.NET Core Web API (Application Programming Interface) series does the job directly here. Caddy's price: its ecosystem is smaller than Nginx's, there are fewer options for advanced scenarios (complex caching rules, script modules such as njs), and most of the team will be seeing it for the first time.

Traefik, YARP, PaaS and Containers

Traefik is a proxy that takes its config from the running platform rather than from files: on Docker it discovers routes from container labels, on Kubernetes from Ingress and Gateway API resources. It's strong where the number and addresses of containers keep changing; for two Kestrels on a single VM it's more complex than needed. I didn't run it for this post; the upcoming Docker and Kubernetes series will cover the container side.

YARP is a reverse proxy Microsoft wrote as a .NET library. The proxy itself becomes an ASP.NET Core app: it can read routing rules from config, you can add middleware in between, and you can do authentication and rate limiting in .NET code. Something in front still terminates TLS (Kestrel itself or a load balancer); YARP mostly plays the API gateway and backend-for-frontend role. Details are in the upcoming API Gateway and YARP series.

On PaaS, for example Azure App Service, the platform does most of the work from posts four to ten of this series: TLS, certificate renewal, load balancing, restarting the process, zero-downtime deploys with deployment slots. In exchange you accept the platform's limits: timeouts, connection limits, file system behaviour and pricing aren't under your control. Topics such as forwarded headers and Data Protection don't disappear on PaaS either; they're just solved the way the platform documents.

Deciding: Four Questions

When choosing for Bookstore I asked these four questions in order:

• What can the team operate? Who will solve the outage at 03:00, looking at which log? If the answer is Windows, IIS; if Linux, Nginx or Caddy; if neither, PaaS.
• How many services, how many instances, how often do you deploy? For one API and one Angular app a VM and Nginx are enough; ten services and several deploys a day justify containers and orchestration.
• Are zero-downtime deploys a must? On a single-instance VM a restart is a short outage; if it's a must, either two instances and drain or a platform with deployment slots.
• Which part of the cost matters? A VM is cheap but operating hours are expensive; on PaaS it can be the other way round. The arithmetic changes as scale grows.

Bookstore's answer: a Linux VM, Nginx, two Kestrel instances under systemd and Angular served from Nginx. This series built exactly that. When the team grows and the number of services increases, the same app can move to containers unchanged: Kestrel listens on 8080, forwarded headers come from configuration, Data Protection keys are in a shared store.

A Production Checklist: the Check Script

Whatever you choose, the same questions need checking before go-live, and testing them with a script is more reliable than ticking them off a list. The script I wrote for Bookstore checks from the outside the face the client sees:

#!/bin/sh
# prodcheck.sh <host> <proxy-ip> <ca-file>: smoke checks before go-live
H=$1; IP=$2; CA=$3
R="--cacert $CA --resolve $H:443:$IP --resolve $H:80:$IP"
ok() { printf "PASS  %s\n" "$1"; }
bad() { printf "FAIL  %s (%s)\n" "$1" "$2"; }
hdr=$(curl -s -o /dev/null -D - $R "https://$H/api/books")
code=$(curl -s -o /dev/null -w "%{http_code}" $R "https://$H/health")
[ "$code" = 200 ] && ok "health endpoint 200" || bad "health endpoint 200" "$code"
loc=$(curl -s -o /dev/null -w "%{http_code} %{redirect_url}" $R "http://$H/api/books")
case "$loc" in 30[18]\ https://*) ok "http redirects to https";; *) bad "http redirects to https" "$loc";; esac
n=$(echo "$hdr" | grep -ci "^strict-transport-security")
[ "$n" = 1 ] && ok "exactly one HSTS header" || bad "exactly one HSTS header" "$n"
echo "$hdr" | grep -qi "^x-content-type-options: nosniff" && ok "nosniff" || bad "nosniff" "missing"
srv=$(echo "$hdr" | grep -i "^server:" | tr -d '\r')
echo "$srv" | grep -q "[0-9]" && bad "server header hides version" "$srv" || ok "server header hides version ($srv)"
echo | openssl s_client -connect "$IP:443" -servername "$H" -tls1_1 -cipher "DEFAULT@SECLEVEL=0" >/dev/null 2>&1 && bad "TLS 1.1 rejected" "accepted" || ok "TLS 1.1 rejected"
who=$(curl -s $R "https://$H/api/whoami")
echo "$who" | grep -q '"scheme":"https"' && ok "app sees https" || bad "app sees https" "$who"
rip=$(echo "$who" | jq -r .remoteIp)
case "$rip" in *"$IP") bad "app sees client IP" "$rip";; *) ok "app sees client IP ($rip)";; esac
code=$(head -c 2000000 /dev/zero | curl -s -o /dev/null -w "%{http_code}" $R -H "Content-Type: application/octet-stream" --data-binary @- "https://$H/api/uploads")
[ "$code" = 413 ] && ok "body limit enforced at proxy (2 MB -> 413)" || bad "body limit enforced at proxy" "$code"
code=$(curl -s -o /dev/null -w "%{http_code}" $R -H "Host: evil.example" "https://$IP/api/books" -k)
[ "$code" = 200 ] && bad "unknown host rejected" "$code" || ok "unknown host rejected ($code)"

Against the seventh post's config, the answer to the guess:

$ ./prodcheck.sh bookstore.test 172.30.85.10 ca.crt
FAIL  health endpoint 200 (404)
PASS  http redirects to https
PASS  exactly one HSTS header
PASS  nosniff
PASS  server header hides version (server: nginx)
PASS  TLS 1.1 rejected
PASS  app sees https
PASS  app sees client IP (172.30.85.50)
PASS  body limit enforced at proxy (2 MB -> 413)
FAIL  unknown host rejected (200)

Two items failed, and both are the kind that only show up in the whole while each piece is right on its own. The health endpoint existed in Kestrel, but Nginx only forwarded /api/; a load balancer or deploy script looking from outside can't reach /health. There was also no default_server on 443, so a request coming by IP address with a random Host fell into the first server in the file, Bookstore; the default server with 444 from the fourth post was only on port 80. Two additions:

# reject TLS handshakes for unknown host names
server {
    listen 443 ssl default_server;
    ssl_reject_handshake on;
}

    # health endpoint for the load balancer and the deploy script
    location = /health {
        proxy_pass http://bookstore_api;
        allow 172.30.85.0/24;
        deny all;
    }

$ ./prodcheck.sh bookstore.test 172.30.85.10 ca.crt
PASS  health endpoint 200
PASS  http redirects to https
PASS  exactly one HSTS header
PASS  nosniff
PASS  server header hides version (server: nginx)
PASS  TLS 1.1 rejected
PASS  app sees https
PASS  app sees client IP (172.30.85.50)
PASS  body limit enforced at proxy (2 MB -> 413)
PASS  unknown host rejected (000)

ssl_reject_handshake on cuts a TLS connection arriving with an unknown SNI (Server Name Indication) without sending a certificate; the HTTPS counterpart of the 444 on port 80. I opened /health to the internal network only: a health endpoint can reveal the app's internal state and doesn't need to be public. The script also had a bug when I first wrote it: it searched the whole response for the client IP check, and since the proxy's address appears in the xOriginalFor field it raised a false alarm. Check scripts are code too; their false positives and negatives need testing.

What the Script Can't Check

A curl from outside can't see everything. Items on Bookstore's go-live list checked by hand or with other tools next to the script:

• Graceful shutdown: KillSignal=SIGTERM (the terminate signal), a sufficient TimeoutStopSec, a systemctl stop attempt with an open request (the eighth post).
• Kestrel listening only on loopback or a Unix socket, so the proxy can't be bypassed (the second post).
• The trusted list in forwarded headers containing only real proxies; ASPNETCORE_FORWARDEDHEADERS_ENABLED not turned on (the third post).
• Data Protection keys sitting encrypted in a persistent store shared between instances (the third and tenth posts).
• Aligned limits: Nginx client_max_body_size, Kestrel MaxRequestBodySize, header buffers, timeouts (the fifth post).
• Logs going to a central place and a request being traceable between proxy and app with a correlation ID.
• Monitoring and alerts: 5xx rate, upstream time, restart count, certificate expiry date. Details in the upcoming Observability series.
• Backup and rollback: the previous release ready on disk, the rollback command tried, a database backup taken and its restore tested at least once.
• A schedule for runtime and proxy updates: .NET servicing releases and security patches of the Nginx stable branch.

In Production

Don't run the check script once and leave it; make it the last step of the deploy pipeline. Every config change, including a location someone adds as "a small fix", should go through the same ten items. A failing item in the pipeline should roll the deploy back.

Document the choice. Why Nginx, why a VM, why not PaaS: a two-paragraph rationale for this decision in an ADR (Architecture Decision Record) answers the "why did we do it this way" question two years later. The Architecture Decisions series explains how to write one.

Common Pitfalls

Choosing by technology. The fastest proxy isn't better than one the team can operate.

Not noticing that the health endpoint isn't visible through the proxy. It exists in the app and has been tested, but if the load balancer can't reach it, an unhealthy instance keeps getting traffic.

Not defining a default server for 443. The team that wrote the 444 on port 80 forgets HTTPS; bots scanning by IP and misdirected domains reach the app.

Filling the checklist once and forgetting it. The config changes, the list stays the same; automate the check.

Thinking PaaS solves everything. Forwarded headers, Data Protection, graceful shutdown and limits apply on the platform too; only the place they're configured changes.

A Small Experiment

★☆☆ Run prodcheck.sh against the Caddy setup. Which items pass and which fail? How would you fix the failing ones in the Caddyfile?

★★☆ Add two items to the script: FAIL if fewer than 30 days remain before the certificate expires, and at least one of 30 simultaneous requests getting a 429. What happens if the second item runs against production, and how would you make it safe?

★★★ Fill in your own decision table for Bookstore: your team's current skills, number of services, deploy frequency and budget. Write the result as an ADR and fill in a "under what conditions this decision changes" section for the two options you didn't choose.

In the last post of the series we look back in time: what changed in hosting from .NET 6 to .NET 10? The move from 80 to 8080 in containers, ASPNETCORE_HTTP_PORTS, the non-root user, Ubuntu-based images, KnownIPNetworks, traps in old tutorials and what's coming with .NET 11.