Disk, Memory and CPU: df, du, free, vmstat, iostat, the OOM Killer and the "Disk Full" Case

Version note: The commands in this post were run on a test VM with 2 CPUs and 1 GB of RAM running Ubuntu Server 24.04.5 LTS (kernel 6.8, systemd 255, procps 4.0.4, sysstat 12.6.1, lsof 4.95); Bookstore.Api was built with .NET SDK 10.0.402 and run on the ASP.NET Core 10.0.15 runtime.

Guess First

An alarm goes off at midnight: the log disk of Bookstore, the online bookstore used as the shared example across this site's series, is 100% full. You log in to the server, find the culprit, a 9.9 MB api.log, and delete it:

$ sudo rm /var/log/bookstore/api.log
$ df -h /var/log/bookstore
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       11M  9.9M     0 100% /var/log/bookstore

The file is gone, but the disk is still full. Why did the space not come back, and how do you get it back? The answer is in the case section.

In this post we look at three resources of the server: disk (df, du, inodes, deleted but open files), memory (free, the page cache, the OOM killer, cgroup limits) and CPU and I/O (uptime, vmstat, pidstat, iostat). The theory of processes, virtual memory and scheduling is in the Operating System Fundamentals series; here you get which command to look at what with when someone says "the server is slow" or "the disk is full".

Disk: df, du and Inodes

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            97M 1004K   96M   2% /run
/dev/vda1        19G  2.5G   16G  14% /
tmpfs           481M  8.0K  481M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/vda16      881M  117M  703M  15% /boot
/dev/vda15      105M  6.2M   99M   6% /boot/efi
tmpfs            97M   12K   96M   1% /run/user/1000
$ sudo du -xh -d1 /var 2>/dev/null | sort -h | tail -n 6
48K    /var/tmp
1.3M    /var/crash
114M    /var/log
253M    /var/lib
342M    /var/cache
710M    /var

df (disk free) gives the total, used and free space per filesystem; -h means readable units. The tmpfs lines are virtual filesystems living in RAM, not disks. The first question is always "which filesystem is full": /, or a separate /var or log disk? Then you dig in with du (disk usage): -x do not cross into other filesystems, -d1 one level deep only, sort -h sorts human-readable sizes correctly. When you find the big directory, repeat the same command inside it. ncdu (sudo apt install ncdu), which does the same interactively, is faster on a server with hundreds of directories.

A disk fills up in two ways: bytes or inodes. Every file, whatever its size, uses one inode, and the number of inodes is fixed when the filesystem is created. An application producing millions of small files can run out of inodes before it runs out of space. I tried it with a small cache filesystem for book cover thumbnails:

$ sudo -u bookstore bash -c 'for i in $(seq 1 400); do : > /var/lib/bookstore/covers-cache/cover-$i.jpg || { echo "stopped at $i"; break; }; done'
bash: line 1: /var/lib/bookstore/covers-cache/cover-246.jpg: No space left on device
stopped at 246
$ sudo df -h /var/lib/bookstore/covers-cache
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      4.0M   32K  3.4M   1% /var/lib/bookstore/covers-cache
$ sudo df -i /var/lib/bookstore/covers-cache
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop0        256   256     0  100% /var/lib/bookstore/covers-cache

Same error message, "No space left on device", but df -h says 1% used. df -i shows the truth: all 256 inodes are used. I created this filesystem with few inodes on purpose; on real servers the cause is usually session files, a mail queue or cache directories that are never cleaned. On a "disk full" alarm look at df -h and df -i together.

The Case: "No space left on device" at Midnight

To reproduce the case quickly, in the lab I mounted /var/log/bookstore on a separate 16 MB filesystem; a separate disk or partition for logs is common in production too, and that disk filling up is the most common form of this problem. The scenario: a few days ago someone hunting a problem made Bookstore.Api write to a file instead of the journal with a drop-in (StandardOutput=append:/var/log/bookstore/api.log, as in the sixth post), then set the log level to Trace in the appsettings.json of the release folder, and forgot both. There is no logrotate rule for this file either. Normal traffic filled the disk in a few minutes. The first symptom does not show up in the log but in another job:

$ sudo -u bookstore bash -c "echo backup-started >> /var/log/bookstore/db-backup.log"
bash: line 1: echo: write error: No space left on device
$ df -h | grep -E "Filesystem|bookstore|vda1"
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        19G  2.6G   16G  14% /
/dev/vda16      881M  117M  703M  15% /boot
/dev/vda15      105M  6.2M   99M   6% /boot/efi
/dev/loop0       11M  9.9M     0 100% /var/log/bookstore
$ sudo ls -lh /var/log/bookstore
total 9.9M
-rw-r--r-- 1 root      root      9.9M Nov 14 17:00 api.log
-rw-rw-r-- 1 bookstore bookstore    0 Nov 14 17:01 db-backup.log
drwx------ 2 root      root       16K Nov 14 16:53 lost+found
$ sudo tail -n 2 /var/log/bookstore/api.log | cut -c1-120
<7>Microsoft.AspNetCore.Server.Kestrel.Connections[2] Connection id "0HNQ2E1M8KG2Q" stopped.
<7>Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7] Connection id "0HNQ2E1M8KG2Q" se
$ grep -A3 LogLevel /opt/bookstore/current/appsettings.json
    "LogLevel": {
      "Default": "Trace",
      "Microsoft.AspNetCore": "Trace"
    }

When the disk filled up, the application did not crash; the API kept returning 200, it just could not write logs. The real damage hits other jobs sharing the disk: the backup file could not be written. The lines prefixed with <7> are Debug and Trace (the syslog levels from the sixth post), and they make up almost the whole file. This is where the step from the guess comes in: I deleted the file and df did not change at all. The reason is what deleting does on Linux: rm removes the file's name. If a process still holds the file open, the data stays on disk and the space is not released until the last open file descriptor is closed.

$ sudo lsof -nP +L1
COMMAND  PID      USER   FD   TYPE DEVICE   SIZE/OFF NLINK NODE NAME
dotnet  8778 bookstore    1w   REG    7,0   10321920     0   12 /var/log/bookstore/api.log (deleted)
dotnet  8778 bookstore    2w   REG    7,0   10321920     0   12 /var/log/bookstore/api.log (deleted)
dotnet  8778 bookstore    6w   REG    7,0   10321920     0   12 /var/log/bookstore/api.log (deleted)
dotnet  8778 bookstore    7w   REG    7,0   10321920     0   12 /var/log/bookstore/api.log (deleted)
dotnet  8778 bookstore    8u   REG    0,1 1006768128     0  126 /memfd:doublemapper (deleted)
dotnet  8778 bookstore  147w   REG    7,0   10321920     0   12 /var/log/bookstore/api.log (deleted)
dotnet  8778 bookstore  148w   REG    7,0   10321920     0   12 /var/log/bookstore/api.log (deleted)
$ sudo ls -l /proc/$(systemctl show -p MainPID --value bookstore-api)/fd/1
l-wx------ 1 bookstore bookstore 64 Nov 14 16:57 /proc/8778/fd/1 -> /var/log/bookstore/api.log (deleted)

lsof +L1 lists files with a link count below 1, that is, files whose name was deleted but that are still open. The dotnet process holds the deleted api.log on several file descriptors: 1 is stdout, 2 is stderr (systemd connected both to the same file), plus the copies .NET opened for the console. The memfd:doublemapper line is not on disk (DEVICE 0,1): it is a mapping the .NET runtime keeps in memory for the JIT; if you see it in this list, ignore it. There are two ways to get the space back: restart the process, or empty the file through the place where it is open:

$ sudo truncate -s 0 /proc/$(systemctl show -p MainPID --value bookstore-api)/fd/1
$ df -h /var/log/bookstore
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0       11M   24K  9.9M   1% /var/log/bookstore

/proc/<pid>/fd/1 is the only remaining path to the deleted file; truncate emptied it and the 9.9 MB came back immediately, without restarting the application. At midnight this can be the first response. The right order is: free up space first (not with rm but with truncate -s 0 /var/log/bookstore/api.log, or as above), then fix the cause. The permanent fix is four steps, and I did all of them: restore appsettings.json to its state in the release (Default: Information, Microsoft.AspNetCore: Warning), delete the drop-in and send the application back to the journal (the journal has its SystemMaxUse limit), put back the logrotate rule with copytruncate from the sixth post for /var/log/bookstore, and do this through a deploy rather than by hand on the server so the next release brings the same settings. Afterwards lsof +L1 only showed the memfd line.

In this case an alarm should have fired before the disk filled up: a warning at 80%, an emergency at 90%. Disk, inodes and journal size are the first lines of monitoring.

Memory: free, the Page Cache and available

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           960Mi       265Mi       719Mi       7.4Mi       104Mi       694Mi
Swap:             0B          0B          0B
$ cat /var/tmp/catalog-export.bin > /dev/null
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           960Mi       288Mi       412Mi       7.4Mi       404Mi       671Mi
Swap:             0B          0B          0B

I read a 300 MB file once. The free column dropped from 719 to 412 and buff/cache rose from 104 to 404. The kernel keeps the file it read in the page cache (an in-memory copy of disk data); the next read comes from RAM instead of disk. This memory is better than sitting idle, and the kernel hands it back immediately when an application asks for it. That is why the column to look at is not free but available: the estimated memory that can be given to new processes without swapping, which dropped only a little, from 694 to 671. Most "the server is out of memory, free is 50 MB" alarms come from this misreading. This VM has no swap (common on cloud images); if there is swap and the si/so columns in vmstat stay above zero, memory really is not enough. If you use WSL 2, free shows the memory Windows gives to WSL; the memory setting in .wslconfig sets the limit.

The OOM Killer: Who Dies When Memory Runs Out?

When memory really runs out, the kernel picks a process and kills it: the OOM killer. Which one it picks depends on the process's memory use and its oom_score_adj setting; the candidates' scores are in /proc/<pid>/oom_score:

$ for p in $(pgrep -x dotnet) $(pgrep -o -x nginx) $(pgrep -o -x sshd) $(pgrep -x systemd-journal); do printf "%-16s pid=%-6s rss=%6s kB oom_score=%s\n" $(cat /proc/$p/comm) $p $(awk "/VmRSS/{print \$2}" /proc/$p/status) $(cat /proc/$p/oom_score); done
dotnet           pid=9143   rss= 59488 kB oom_score=706
nginx            pid=702    rss=  1488 kB oom_score=667
sshd             pid=807    rss=  8252 kB oom_score=0
systemd-journal  pid=314    rss= 14772 kB oom_score=510

The order of the scores matters more than their absolute values: the highest goes first. On this server Bookstore.Api, which uses the most memory, is the riskiest candidate. sshd scores 0 because sshd sets its own oom_score_adj to -1000 at startup: so that you can still log in to the server even when memory runs out. journald is also protected with -250. I ran a process that eats memory:

$ python3 hog.py 50; echo "exit=$?"
bash: line 1:  9938 Killed                  python3 hog.py 50
exit=137
$ journalctl -k -b --no-pager --grep "Out of memory" | tail -n 1 | cut -c1-200
Nov 14 17:04:16 bookstore-dev kernel: Out of memory: Killed process 9938 (python3) total-vm:789152kB, anon-rss:770560kB, file-rss:2304kB, shmem-rss:0kB, UID:1000 pgtables:1576kB oom_score_adj:0
$ systemctl is-active bookstore-api nginx ssh
active
active
active

hog.py is a three-line script that allocates 50 MB blocks in a loop. When it reached 750 MB the kernel killed it with SIGKILL (exit code 137 = 128 + 9). The process cannot write its own error; the trace stays only in the kernel log, and journalctl -k from the sixth post shows it. We were lucky this time, the hog used the most memory. Had a report job been in its place with memory filling up slowly, the kernel could have picked the process with the highest score, that is, Bookstore.Api. If a .NET process restarts for no reason and there is nothing in the application log, this is the first place to look.

The way to prevent this is to give each service its own memory limit. systemd does this with cgroups. I ran the same script inside a scope limited to 100 MB:

$ sudo systemd-run --scope -p MemoryMax=100M -p MemorySwapMax=0 python3 hog.py 10; echo "exit=$?"
Running as unit: run-rdf0a50a36ab84448bc85be99d754cb16.scope; invocation ID: 5fc4a609e7a64e96a9c3503a69ed6766
bash: line 1: 10475 Killed                  sudo systemd-run --scope -p MemoryMax=100M -p MemorySwapMax=0 python3 hog.py 10
exit=137
$ journalctl -k -b --no-pager --grep "Memory cgroup out of memory" | tail -n 1 | cut -c1-200
Nov 14 17:10:48 bookstore-dev kernel: Memory cgroup out of memory: Killed process 10476 (python3) total-vm:123532kB, anon-rss:102016kB, file-rss:6528kB, shmem-rss:0kB, UID:0 pgtables:272kB oom_score_a

This time the message is "Memory cgroup out of memory": while the rest of the system had plenty of memory, the process hit its own limit and only a process inside that group was killed. We added two lines to Bookstore.Api's unit (in the repository at deploy/linux/systemd/bookstore-api.service):

TimeoutStopSec=30
MemoryHigh=384M
MemoryMax=512M

$ systemctl status bookstore-api --no-pager | grep -E "Active|Memory"
     Active: active (running) since Sat 2026-11-14 17:03:54 UTC; 4s ago
     Memory: 30.2M (high: 384.0M max: 512.0M available: 353.7M peak: 30.2M)

MemoryHigh is the soft limit: once exceeded, the kernel starts reclaiming memory from this group aggressively and the process slows down but does not die. MemoryMax is the hard limit: if it is exceeded, the OOM killer works only inside this service, Nginx and SSH are protected, and systemd restarts the application with Restart=always. The values fit this 1 GB lab VM; on your own server watch the application's peak usage under normal load (peak) for a few days and choose a value with some headroom above it.

CPU: Load Average, vmstat and pidstat

On the two-CPU server I started three infinite loops and waited a bit over a minute:

$ uptime
 17:12:28 up 40 min,  1 user,  load average: 2.26, 1.51, 1.28
$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 4  0      0 591000  74728 157936    0    0   871  2179 1263    7 31 12 56  0  0  0
 3  0      0 591000  74728 157936    0    0     0     0 2013  538 100  0  0  0  0  0
 3  0      0 591000  74728 157936    0    0     0     0 2016  529 100  0  0  0  0  0
 3  0      0 591000  74728 157936    0    0     0     0 2006  529 100  0  0  0  0  0
 3  0      0 591000  74728 157936    0    0     0     0 2015  524 100  0  0  0  0  0
$ pidstat -u 1 1 | grep -E "Average|UID" | sort -k8 -rn | head -n 5
Average:     1000     10664   67.26    0.00    0.00   32.74   67.26     -  sh
Average:     1000     10663   67.26    0.00    0.00   32.74   67.26     -  sh
Average:     1000     10665   61.06    0.00    0.00   38.94   61.06     -  sh
Average:     1000     10672    0.00    3.54    0.00    7.08    3.54     -  pidstat
Average:      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command

The load average is the average over the last 1, 5 and 15 minutes and is read together with the number of cores: on 2 cores 2.0 means "fully busy", 2.26 means "some work is waiting in line". The 1-minute value rises slowly; even a minute after the loops started it has not reached 3 yet. On Linux the load average also counts processes in D state waiting for I/O (fourth post), so a high load is not always a CPU problem. vmstat makes the distinction: r is the number of processes waiting in line to run (3, more than the number of cores), us user code (100%), id idle (0%), wa waiting for I/O (0%). The first line is the average since boot, so ignore it. pidstat -u gives which process uses the CPU; the %wait column shows how long the process waited in line for the CPU, and since three processes share two cores, each waited close to a third of the time. On cloud VMs also look at the st (steal) column: CPU time the hypervisor takes away from the VM; if it stays high, the problem is not your application but the VM's size or its neighbors.

When the culprit turns out to be a .NET process, the Linux tools have done their job: which thread and which method burn the CPU is told by dotnet-counters and dotnet-trace. The hang and slowness post of the Debugging and Problem Solving series and the upcoming .NET Profiling and Diagnostics series cover this.

Disk I/O: iostat and pidstat -d

I started small writes that flush to disk after each write (oflag=direct,dsync); a log writer or a database that fsyncs after every line looks like this:

$ vmstat 1 4
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 2  0      0 696036  29428 112220    0    0   961  2078 1229    7 32 13 55  0  0  0
 0  1      0 694272  32104 112288    0    0     0  9380 3778 5360  0  5 74 21  0  0
 0  1      0 688476  36200 112376    0    0     0 14344 5324 8235  1  2 76 22  0  0
 0  1      0 684444  40088 112468    0    0     0 16328 5909 9406  2  2 75 21  0  0
$ iostat -dx 2 2 vda | tail -n 3
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
vda              0.50      2.00     0.00   0.00    1.00     4.00 2880.50  26884.00  3840.50  57.14    0.18     9.33    1.00  65590.00     0.00   0.00    0.50 65590.00 1920.50    0.12    0.74  96.65
$ pidstat -d 1 1 | grep -E "Average.*(UID|dd)"
Average:      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
Average:     1000     10351      0.00   3146.67      0.00       0  dd

This time the picture is different: r is 0, b 1 (a process in D state waiting for I/O), wa 21%, and three quarters of the CPU idle. iostat -x (in the sysstat package) gives per-device detail; the first report is the average since boot, so I read the second one. The important columns: w/s write requests per second (2880), w_await the average wait of a write (ms), aqu-sz the average queue length and %util the share of time the device was busy (96.65%). %util is meaningful for old single-queue disks; on SSD and NVMe disks working in parallel, seeing 100% does not mean the device has hit its limit, the real measure is rising await. pidstat -d tells which process the writes come from. In a real case, if you see this picture at the same time as a "Bookstore is slow" complaint, a backup or log job sharing the disk is the suspect.

sysstat comes installed on Ubuntu, but history collection is off (ENABLED="false" in /etc/default/sysstat). If you set it to true and enable the service, you can answer "what was the disk doing at 02:30 last night" after the fact with sar; after an incident, live commands do not show that moment.

Production Notes

The known sources of full disks, in order: application logs that are not rotated, backups that are not compressed or deleted, Docker's images, build cache and container logs (/var/lib/docker; check with docker system df, the upcoming Docker series), and the apt cache (/var/cache/apt). Put a limit on each: a logrotate rule, a retention period for backups, the log driver's max-size setting in Docker.

Write resource limits in the unit. Next to MemoryMax, CPUQuota= limits a service's CPU share and LimitNOFILE= the number of open files. A service without limits can sink the whole server along with itself; a limited one only restarts itself.

Measure, then decide. Load average, available memory, disk and inode usage and await belong in monitoring; the commands in this post are for answering "why" after the alarm. On a server with no history you cannot answer "was it always like this".

Common Pitfalls

Deleting an open log file with rm and assuming space was freed. As we saw in the guess, the space does not come back until the process closes it. When freeing space on a full disk use truncate -s 0; if you did use rm, find who holds it with lsof +L1.

Raising a memory alarm based on the free column. The page cache looks "used" but is free; look at available. For the same reason, emptying the cache with echo 3 > /proc/sys/vm/drop_caches is not a fix in production, it only slows down the next reads; in this post I used it once only to show the page cache.

Reading the load average without the number of cores. A load of 4 is a problem on a 2-core server and idle on a 16-core one. Compare with nproc and check with vmstat whether it is CPU or I/O.

Mistaking the OOM killer for an application bug. The process dies without writing a log, systemd restarts it, and all that remains is a "Main process exited, code=killed, status=9/KILL" line in journalctl -u. When you see it, look at journalctl -k --grep "Out of memory".

Looking only at df -h. Inodes run out too; if you see "No space left on device" while space is 1% used, check df -i.

A Small Experiment

★☆☆ On your own server find the biggest directories with sudo du -xh -d1 / 2>/dev/null | sort -h | tail, then repeat the same command inside the biggest one. Also check inode usage with df -i.

★★☆ In a shell keep a file open with exec 3>>/var/tmp/big.log and write 100 MB into it with dd if=/dev/zero bs=1M count=100 >&3. From another terminal delete the file with rm, confirm that df did not change and that lsof +L1 shows it, then get the space back with truncate.

★★★ Temporarily lower MemoryMax in Bookstore.Api's unit to 80M, push it to the limit by sending parallel requests to the report endpoint, and note what you see in journalctl -u bookstore-api and journalctl -k. Let systemctl show -p NRestarts tell you how many times it restarted.

We have seen how to diagnose disk, memory and CPU problems. In the next post we harden the server against the outside: SSH settings, ufw, Fail2ban, automatic updates and least privilege.