A Server Troubleshooting Playbook: From "The Site Is Down" to the Root Cause

Version note: The commands in this post were run on a test VM with 2 CPUs and 1 GB of RAM running Ubuntu Server 24.04.5 LTS (kernel 6.8, systemd 255, Nginx 1.24, OpenSSL 3.0.13) and on an Ubuntu 24.04 client on the same network; Bookstore.Api was built with .NET SDK 10.0.402 and run on the ASP.NET Core 10.0.15 runtime, and the faulty release in the case was built with .NET SDK 11.0.100.

Guess First

In the afternoon a new release of Bookstore, the online bookstore used as the shared example across this site's series, is deployed. The deploy script prints this and stops:

Job for bookstore-api.service failed because the control process exited with error code.
See "systemctl status bookstore-api.service" and "journalctl -xeu bookstore-api.service" for details.

A few seconds later Gül from customer support writes: "The site is down." You check from the client:

ada@my-pc:~$ curl -s -i http://10.20.30.10/api/books | head -n 4
HTTP/1.1 502 Bad Gateway
Server: nginx/1.24.0 (Ubuntu)
Date: Sat, 14 Nov 2026 17:41:05 GMT
Content-Type: text/html

You log in to the server and, by reflex, list the broken services:

$ systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.

The application is not running, yet according to systemd no service is broken. Why? The answer is in the first case.

This post brings the previous eleven posts of the series together: a playbook for what to look at, and in what order, when the complaint is "the site is down" or "the site is slow", and three Bookstore cases I actually produced in the lab. The method from the first post of the Debugging and Problem Solving series still applies: observation, hypothesis, a single experiment. The difference is that the observation tools are server commands.

The First 60 Seconds

When a complaint comes in, a fixed checklist that looks at the overall state of the server before you form a hypothesis speeds things up. I adapted this list from Brendan Gregg's "Linux Performance Analysis in 60,000 Milliseconds", published on the Netflix tech blog; that article focuses on performance, and I added service and log checks. I ran it twenty seconds after the 502 above:

$ uptime
 17:41:26 up 4 min,  1 user,  load average: 0.83, 1.07, 0.56
$ sudo dmesg -T | tail -n 3
[Sat Nov 14 17:37:36 2026] loop0: detected capacity change from 0 to 8
[Sat Nov 14 17:37:40 2026] audit: type=1400 audit(1794677859.916:120): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=919 comm="apparmor_parser"
[Sat Nov 14 17:37:42 2026] systemd-journald[313]: /var/log/journal/7a5216d613204956b9683a4a5dcfa072/user-1000.journal: Journal file uses a different sequence number ID, rotating.
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           960Mi       289Mi       536Mi       1.0Mi       277Mi       670Mi
Swap:             0B          0B          0B
$ df -h / /var/log
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        19G  2.6G   16G  15% /
/dev/vda1        19G  2.6G   16G  15% /
$ systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION

0 loaded units listed.
$ journalctl -p err -b --since -15min
Nov 14 17:41:02 bookstore-dev systemd[1]: Failed to start bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel).
Nov 14 17:41:13 bookstore-dev systemd[1]: Failed to start bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel).
Nov 14 17:41:24 bookstore-dev systemd[1]: Failed to start bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel).
$ sudo ss -tlnp
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("nginx",pid=784,fd=7),("nginx",pid=782,fd=7),("nginx",
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=855,fd=3),("systemd",pid=1,fd=91))
LISTEN 0      4096      127.0.0.54:53        0.0.0.0:*    users:(("systemd-resolve",pid=495,fd=17))
LISTEN 0      4096   127.0.0.53%lo:53        0.0.0.0:*    users:(("systemd-resolve",pid=495,fd=15))
LISTEN 0      4096            [::]:22           [::]:*    users:(("sshd",pid=855,fd=4),("systemd",pid=1,fd=92))
$ top -bn1 | head -n 5
top - 17:41:27 up 4 min,  1 user,  load average: 0.83, 1.07, 0.56
Tasks: 124 total,   1 running, 123 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  7.1 sy,  0.0 ni, 85.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    960.1 total,    534.0 free,    291.9 used,    277.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    668.2 avail Mem

Each line rules out a question. uptime: did the machine just boot, is the load unusual (tenth post)? Here, neither. dmesg: is the kernel saying anything, OOM killer, disk errors, the network card? The last lines are from boot, nothing new. free and df: has memory or disk run out? No. systemctl --failed: empty. journalctl -p err: here is the first clue, "Failed to start bookstore-api.service" repeating every 11 seconds. ss -tlnp: Nginx is on 80, but nobody listens on 5000; in the eighth post this list had a 127.0.0.1:5000 line. top: the CPU is idle.

In sixty seconds we learned that there is no resource problem, that the application is not up and that systemd keeps trying to start it. The value of the checklist is less in what it finds than in what it rules out: we no longer look at disk, memory or the network.

Case 1: A 502 After a Deploy

The Nginx side of the 502 and the state of the service:

$ sudo tail -n 1 /var/log/nginx/bookstore.error.log
2026/11/14 17:41:05 [error] 782#782: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.30.5, server: _, request: "GET /api/books HTTP/1.1", upstream: "http://127.0.0.1:5000/api/books", host: "10.20.30.10"
$ systemctl status bookstore-api --no-pager -n 0
● bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel)
     Loaded: loaded (/etc/systemd/system/bookstore-api.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Sat 2026-11-14 17:41:34 UTC; 4s ago
    Process: 2448 ExecStart=/usr/bin/dotnet /opt/bookstore/current/Bookstore.Api.dll (code=exited, status=150)
   Main PID: 2448 (code=exited, status=150)
        CPU: 337ms
$ systemctl show -p NRestarts bookstore-api
NRestarts=3

The answer to the guess is in the Active line: activating (auto-restart). The service crashes, Restart=always restarts it 10 seconds later, and between two attempts its state is "waiting to restart", not "failed". systemctl --failed only shows units that were given up on; a service in a crash loop does not appear there. We saw why in the fifth post: systemd's default limit is 5 starts in 10 seconds, our unit waits 10 seconds between attempts, and the limit is never reached.

On the Nginx side, three different 502 messages point to three different places. 111: Connection refused: nobody listens at the upstream address, either the process is dead or it is on another port (the wrong-port case in the eighth post). upstream timed out: the process listens but does not answer, it is stuck or very slow. upstream prematurely closed connection: the process accepted the request and closed the connection without answering, usually because it crashed during the request. Ours is the first one, consistent with the ss output. Next: why is the process dead?

$ journalctl -u bookstore-api -b -n 18 --no-pager -o cat
...
You must install or update .NET to run this application.
App: /opt/bookstore/releases/2026-11-14-7/Bookstore.Api.dll
Architecture: x64
Framework: 'Microsoft.NETCore.App', version '11.0.0' (x64)
.NET location: /usr/lib/dotnet/
The following frameworks were found:
  10.0.15 at [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
Learn more:
https://aka.ms/dotnet/app-launch-failed
To install missing framework, download:
https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=11.0.0&arch=x64&rid=ubuntu.24.04-x64&os=ubuntu.24.04
bookstore-api.service: Main process exited, code=exited, status=150/n/a
bookstore-api.service: Failed with result 'exit-code'.
Failed to start bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel).
$ dotnet --list-runtimes
Microsoft.AspNetCore.App 10.0.15 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 10.0.15 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]
$ readlink /opt/bookstore/current
/opt/bookstore/releases/2026-11-14-7

The root cause is clear: the new release targets net11.0. .NET 11 shipped on 10 November, Emre tried the upgrade on a branch, the branch was merged and CI deployed the new build; the server only has the .NET 10 runtime we installed in the ninth post. Exit code 150 is the last byte of the .NET host's "framework not found" error code (0x80008096); execution never reached the application code, so there is no application log, only the host's message.

In production, the order is to end the outage first and fix it second. The release folders and the current symlink (second post) exist for exactly this moment; going back is a symlink change:

$ ls /opt/bookstore/releases/ | tail -n 2
2026-11-14-6
2026-11-14-7
$ sudo -u deploy ln -sfn /opt/bookstore/releases/2026-11-14-6 /opt/bookstore/current
$ sudo -u deploy sudo /usr/bin/systemctl restart bookstore-api
$ systemctl is-active bookstore-api
active
ada@my-pc:~$ curl -s -o /dev/null -w "%{http_code}\n" http://10.20.30.10/api/books
200

The site is back. The permanent fix is a separate decision: either install aspnetcore-runtime-11.0 on the server (the ways from the ninth post) or keep the project on .NET 10. Make that decision with the team the next day, not on the server at midnight. Do not delete the 2026-11-14-7 folder either: it is evidence for finding out which build caused the problem.

Making the Crash Loop Visible

In this case systemd's endless retries hid the problem: a "failed" service would have been seen by monitoring. I added a start limit to the [Unit] section of the unit (in the repo as deploy/linux/systemd/bookstore-api.service):

[Unit]
Description=Bookstore API (ASP.NET Core on Kestrel)
After=network.target
StartLimitIntervalSec=300
StartLimitBurst=5

5 starts in 5 minutes. With RestartSec=10 this is a window of about a minute: on a transient problem, say the database down for a few seconds, the application still recovers by itself, but a release that crashes on every start becomes failed after a minute. I deployed the same faulty release again and waited 75 seconds:

$ systemctl --failed
  UNIT                  LOAD   ACTIVE SUB    DESCRIPTION
● bookstore-api.service loaded failed failed Bookstore API (ASP.NET Core on Kestrel)

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

1 loaded units listed.
$ journalctl -u bookstore-api -n 3 -o cat
bookstore-api.service: Start request repeated too quickly.
bookstore-api.service: Failed with result 'exit-code'.
Failed to start bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel).
$ sudo -u deploy ln -sfn /opt/bookstore/releases/2026-11-14-6 /opt/bookstore/current
$ sudo -u deploy sudo /usr/bin/systemctl restart bookstore-api; echo "exit=$?"
Job for bookstore-api.service failed because the control process exited with error code.
See "systemctl status bookstore-api.service" and "journalctl -xeu bookstore-api.service" for details.
exit=1

systemctl --failed now shows the problem, and monitoring can easily catch a unit going failed. But a new trap appeared: the restart after the rollback was refused too. The limit also counts starts made by hand; the journal again says "Start request repeated too quickly". The counter is not reset until the interval has passed or systemctl reset-failed runs. Since the deploy user's sudo rights are only restart and status, I added this one command to sudoers (third post):

deploy ALL=(root) NOPASSWD: /usr/bin/systemctl restart bookstore-api, /usr/bin/systemctl status bookstore-api, /usr/bin/systemctl reset-failed bookstore-api

$ sudo visudo -cf /tmp/bookstore-deploy
/tmp/bookstore-deploy: parsed OK
$ sudo -u deploy sudo /usr/bin/systemctl reset-failed bookstore-api
$ sudo -u deploy sudo /usr/bin/systemctl restart bookstore-api
$ systemctl is-active bookstore-api
active
ada@my-pc:~$ curl -s -o /dev/null -w "%{http_code}\n" http://10.20.30.10/api/books
200

The rollback is now ln -sfn, reset-failed, restart. Put it in the deploy script and add one more step: after the restart, check the health endpoint and roll back to the previous release automatically if it does not answer. The deploy script in the systemd post of the Web Servers and Hosting series does this smoke test, and the real-script post of the Bash Scripting series writes it with error handling. You can also attach a notification unit with OnFailure= that runs when the unit fails.

Case 2: "The Site Is Slow"

The second complaint is vaguer: the site opens, but it is slow. Measure first, because "slow" is not a number. With curl -w from the slow request post of the Networking and HTTP series, I sent 60 requests in a row and sorted the times:

ada@my-pc:~$ for i in $(seq 1 60); do curl -s -o /dev/null -w "%{time_total}\n" http://10.20.30.10/api/books; done | sort -n | awk '{t[NR]=$1; s+=$1} END {printf "n=%d avg=%.3fs p50=%.3fs p95=%.3fs max=%.3fs\n", NR, s/NR, t[int(NR*0.5)], t[int(NR*0.95)], t[NR]}'
n=60 avg=0.038s p50=0.029s p95=0.073s max=0.298s

Normally the same measurement gives a p50 of 3 ms and a p95 of 5–17 ms; now the p50 is 29 ms and the slowest request 300 ms. On the server, the first-60-seconds list pointed somewhere else this time:

$ uptime
 17:48:47 up 12 min,  1 user,  load average: 1.71, 1.20, 0.77
$ top -bn1 -w 110 -o %CPU | sed -n '1,3p;7,10p'
top - 17:48:48 up 12 min,  1 user,  load average: 1.71, 1.20, 0.77
Tasks: 117 total,   1 running, 116 sleeping,   0 stopped,   0 zombie
%Cpu(s): 91.7 us,  8.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   3961 booksto+  20   0  281576 109276   2212 S 120.0  11.1   0:22.22 xz
   3463 booksto+  20   0  261.2g  83120  64988 S  40.0   8.5   0:20.63 dotnet
   4064 ada       20   0   15116   5472   3348 R   6.7   0.6   0:00.10 top
$ ps -o pid,user,unit,%cpu,cmd -C xz
    PID USER     UNIT                            %CPU CMD
   3961 booksto+ bookstore-db-backup.service      138 xz -T0 -3 -c /var/lib/bookstore/backup/BookstoreDb.bak
$ systemctl status bookstore-db-backup --no-pager -n 0 | head -n 8
● bookstore-db-backup.service - Nightly BookstoreDb backup
     Loaded: loaded (/etc/systemd/system/bookstore-db-backup.service; static)
     Active: activating (start) since Sat 2026-11-14 17:48:31 UTC; 16s ago
TriggeredBy: ● bookstore-db-backup.timer
   Main PID: 3959 (bookstore-db-ba)
      Tasks: 4 (limit: 1106)
     Memory: 105.7M (peak: 105.7M)
        CPU: 22.678s

The idle part of the CPU is zero and wa is zero: CPU, not I/O. xz uses the most CPU, 120% of two cores. The unit column of ps says which systemd unit the process belongs to: the nightly backup from the fifth post. Someone changed the backup to compress with xz -T0, using every core, so that it would be smaller. In the lab I started the backup by hand; in real life this happens when the timer's schedule shifts or a growing backup runs into the morning, when traffic picks up. Also in the lab, since there is no database, the backup script compresses a 128 MB file instead of a SQL Server dump; the effect on the CPU is the same.

Stopping the backup is not the fix; setting priorities is. Under systemd every service runs in its own cgroup, and its CPU share is set with CPUWeight= (100 by default). I added two lines to the backup unit (in the repo as deploy/linux/systemd/bookstore-db-backup.service):

# background job: give way to bookstore-api when the CPU is busy
Nice=10
CPUWeight=20

$ systemctl show -p Nice -p CPUWeight bookstore-db-backup
CPUWeight=20
Nice=10
ada@my-pc:~$ for i in $(seq 1 60); do curl ... ; done | sort -n | awk ...
n=60 avg=0.008s p50=0.006s p95=0.012s max=0.082s
$ top -bn1 -w 110 -o %CPU | sed -n "7,9p"
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6540 booksto+  30  10  281576 106380   2216 S 156.2  10.8   0:22.22 xz
   6659 ada       20   0   15116   5476   3348 R  12.5   0.6   0:00.13 top
$ cat /sys/fs/cgroup/system.slice/cgroup.subtree_control
cpu memory pids
$ cat /sys/fs/cgroup/system.slice/bookstore-db-backup.service/cpu.weight /sys/fs/cgroup/system.slice/bookstore-api.service/cpu.weight
20
100

The backup is still running and still uses 156% of the CPU, but requests are back to normal: a p50 of 6 ms. CPUWeight is not a cap but a share under contention: when the CPU is idle the backup uses all of it, and when Bookstore.Api needs something it gets priority at a ratio of 100 to 20. On this system the cpu controller was not enabled in system.slice before the change (cgroup.subtree_control had only memory pids); systemd enabled it with the first unit that used CPUWeight. Nice=10 works between processes in the same cgroup; writing both keeps the backup in the background whatever the situation. Also check separately that the backup finishes during low-traffic hours.

There are two more common causes of slowness that I did not produce in this lab. If memory is short, swap usage (the si/so columns of vmstat in the tenth post) and the kernel's reclaim work slow requests down; a service that reaches MemoryHigh slows down too. If Bookstore's SQL Server connection pool runs out, the server commands look normal: CPU idle, disk quiet, but requests time out waiting for a connection from the pool. Linux tools do not show that; the application's metrics do, and looking at them with dotnet-counters is in the hang and slowness post of the Debugging and Problem Solving series.

Case 3: An Expired Certificate

The third complaint comes from the browser: "Your connection is not private." In the lab I gave Nginx an HTTPS certificate signed by my own lab CA that expired on 12 November; the client trusts this CA:

ada@my-pc:~$ curl -sS --cacert ~/bookstore-lab-ca.crt --resolve bookstore.test:443:10.20.30.10 https://bookstore.test/api/books
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
ada@my-pc:~$ openssl s_client -connect 10.20.30.10:443 -servername bookstore.test < /dev/null 2>/dev/null | openssl x509 -noout -subject -enddate
subject=CN = bookstore.test
notAfter=Nov 12 00:00:00 2026 GMT

On the server the services are healthy, the ports are open and the first-60-seconds list comes back clean; the problem is in the TLS handshake, in the client's decision. openssl s_client fetches the certificate the server sends and openssl x509 reads its end date: it expired two days ago. On the server I checked which file Nginx uses and the state of that file:

$ sudo nginx -T 2>/dev/null | grep -E "^\s*ssl_certificate "
    ssl_certificate /etc/ssl/bookstore/bookstore.test.crt;
$ openssl x509 -in /etc/ssl/bookstore/bookstore.test.crt -noout -enddate -checkend 0; echo "exit=$?"
notAfter=Nov 12 00:00:00 2026 GMT
Certificate will expire
exit=1

nginx -T prints the entire config loaded at that moment; -checkend N says whether the certificate expires within N seconds and exits with 1 if it does. Note that with the expired certificate nginx -t gave no warning, and Nginx loaded it without complaint. I renewed the certificate, signing a new 90-day one with my lab CA, and reloaded Nginx:

$ sudo install -m 644 renewed.crt /etc/ssl/bookstore/bookstore.test.crt
$ sudo nginx -t && sudo systemctl reload nginx
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ openssl x509 -in /etc/ssl/bookstore/bookstore.test.crt -noout -enddate -checkend 1209600; echo "exit=$?"
notAfter=Feb 12 17:56:32 2027 GMT
Certificate will not expire
exit=0
ada@my-pc:~$ curl -sS --cacert ~/bookstore-lab-ca.crt --resolve bookstore.test:443:10.20.30.10 -o /dev/null -w "%{http_code}\n" https://bookstore.test/api/books
200

On a real server the certificate comes from Let's Encrypt and certbot's timer renews it; then the first places to look are systemctl list-timers and the renewal service's journal. Renewals usually fail silently: port 80 closed in the firewall, DNS changed, or the new file fetched but Nginx not reloaded. There is no certbot in this lab and I did not run that part; automatic renewal with Let's Encrypt and certificate monitoring are covered in the SSL/TLS Certificates and HTTPS series. -checkend 1209600, 14 days, is the first step of the permanent fix: make it a monitoring check that alerts when less than 14 days are left.

The Timeline and the Permanent Fix

Once the incident is closed, write down what happened in order. journalctl can merge different sources into a single time order; + means "or":

$ journalctl -b --since 17:40:50 -o short-iso --no-pager _COMM=sudo + _SYSTEMD_UNIT=bookstore-api.service + UNIT=bookstore-api.service | grep -E "COMMAND=/usr/bin/(ln|systemctl)|Main process exited|Started|restart counter" | sed -E 's/ bookstore-dev / /' | cut -c1-150
2026-11-14T17:41:01+00:00 sudo[2269]:      ada : PWD=/home/ada ; USER=deploy ; COMMAND=/usr/bin/ln -sfn /opt/bookstore/releases/2026-11-14-7 /opt/book
2026-11-14T17:41:01+00:00 sudo[2272]:   deploy : PWD=/home/ada ; USER=root ; COMMAND=/usr/bin/systemctl restart bookstore-api
2026-11-14T17:41:02+00:00 systemd[1]: bookstore-api.service: Main process exited, code=exited, status=150/n/a
2026-11-14T17:41:12+00:00 systemd[1]: bookstore-api.service: Scheduled restart job, restart counter is at 1.
2026-11-14T17:41:13+00:00 systemd[1]: bookstore-api.service: Main process exited, code=exited, status=150/n/a
2026-11-14T17:41:23+00:00 systemd[1]: bookstore-api.service: Scheduled restart job, restart counter is at 2.
2026-11-14T17:41:24+00:00 systemd[1]: bookstore-api.service: Main process exited, code=exited, status=150/n/a
2026-11-14T17:41:34+00:00 systemd[1]: bookstore-api.service: Scheduled restart job, restart counter is at 3.
2026-11-14T17:41:34+00:00 systemd[1]: bookstore-api.service: Main process exited, code=exited, status=150/n/a
2026-11-14T17:41:44+00:00 systemd[1]: bookstore-api.service: Scheduled restart job, restart counter is at 4.
2026-11-14T17:41:46+00:00 systemd[1]: bookstore-api.service: Main process exited, code=exited, status=150/n/a
2026-11-14T17:41:56+00:00 systemd[1]: bookstore-api.service: Scheduled restart job, restart counter is at 5.
2026-11-14T17:41:57+00:00 systemd[1]: bookstore-api.service: Main process exited, code=exited, status=150/n/a
2026-11-14T17:42:06+00:00 sudo[2766]:      ada : PWD=/home/ada ; USER=deploy ; COMMAND=/usr/bin/ln -sfn /opt/bookstore/releases/2026-11-14-6 /opt/book
2026-11-14T17:42:06+00:00 sudo[2769]:   deploy : PWD=/home/ada ; USER=root ; COMMAND=/usr/bin/systemctl restart bookstore-api
2026-11-14T17:42:09+00:00 systemd[1]: Started bookstore-api.service - Bookstore API (ASP.NET Core on Kestrel).

Deploy at 17:41:01, first crash a second later, Nginx's first 502 at 17:41:05 (error log), rollback at 17:42:06, service up at 17:42:09: an outage of 68 seconds. In this timeline, who did what comes from the sudo entries and what the system did comes from systemd; nobody writes it from memory. The finding-causes-in-logs post of the Debugging and Problem Solving series builds the same timeline with application logs and a correlation ID.

The permanent fix is the change that makes sure this incident does not happen the same way again, not the manual fix on the server. What came out of the three cases is in the repo: a start limit in the unit, the reset-failed right for the deploy user, CPU priority in the backup unit. What is not in the repo is monitoring and process work: a failed-unit alert, a p95 latency alert, a certificate expiry alert, and a CI check that compares the target framework with the runtime on the server. The root cause analysis post of the Debugging and Problem Solving series covers writing these steps up as a blameless postmortem, and the upcoming Observability series covers the alerts themselves.

Production Notes

End the outage first, then understand the cause. The way back, the previous release and the previous config, must be ready and tested; a rollback tried for the first time during an outage can, as in the second case, bring its own trap. Collect the evidence before rolling back: systemctl status and the journal output take seconds and can be hard to reproduce later.

Keep the checklist in writing. Put the first-60-seconds commands from this post in the team's runbook; someone working from memory under stress skips steps. Turning the commands into a script is easy, but a script cannot read the output for you.

One complaint does not mean one cause. In case 1 there was a 502, but the disk could have been full at the same time. That is also why the checklist exists: to say "do not stop at the first thing you find".

Common Pitfalls

Changing several things at once in production by trial and error. If you restart Nginx, update the runtime and change the config, and the site recovers, you cannot tell which one fixed it, and the other two may have introduced a new problem. One hypothesis, one change, one measurement.

Treating services as healthy because systemctl --failed is empty. A service in a crash loop does not show up there; look for activating (auto-restart) in systemctl status and at NRestarts.

Concluding that the application never ran because there is no application log. Host-level errors such as a missing runtime, a wrong path or permissions happen before the application code runs; their messages are in the journal, between systemd's lines.

Guessing without measuring when the complaint is "slow". Get a number first (curl -w, $request_time in the Nginx log), then look at the server, and repeat the same measurement after the fix.

Looking for a certificate problem in server metrics. A TLS error happens on the client; the server looks healthy. Check from the outside with openssl s_client and curl -v.

A Small Experiment

★☆☆ Run the first-60-seconds list on your own server and write a one-sentence note for each command's output: what did you look at to call it "normal"?

★★☆ On a test server put a non-existent ExecStart path in the unit and compare how systemctl status, NRestarts and systemctl --failed behave with and without StartLimitBurst. Which 502 message do you see in the Nginx error log?

★★★ Call Bookstore.Api's report endpoint (/api/reports/sales?seconds=N) with a request that takes longer than Nginx's proxy_read_timeout and confirm that you get a 504 instead of a 502. Compare the error log message with the three messages in this post and build a timeline.

We have seen how to set up, run and troubleshoot a server on Ubuntu 24.04. In the last post of the series we look both ahead and back: what sudo-rs and the Rust coreutils that come with Ubuntu 26.04 mean for your scripts, and reading old servers left over from the days of SysV, ifconfig and iptables.