Debugging Containers: logs, exec, inspect, stats, events, docker debug and Exit Code 137
Debugging a container starts with three questions: does it never start, does it start and die, or does it run but cannot be reached? With real output we work through exec format error and a CRLF script, a restart loop, looking into a shell-less chiseled image with a helper container sharing its PID and network namespaces and with docker debug, and a Bookstore.Api that dies with 137 every 30 seconds because of its health check process.
Version note: The examples in this post were run with Docker Engine 29.9.3, Docker Desktop 4.64 (docker debug), .NET 10 (ASP.NET Core Runtime 10.0.15), busybox:1.37 and alpine:3.
Guess First
You started Bookstore.Api with --cpus 2 --memory 32m. After 12 seconds docker stats shows 22.92MiB / 32MiB 71.63%. Memory is under the limit and the health check says healthy. About 35 seconds later the container is Exited (137), and it never received a request. Why? Second question: what does docker exec -it dockerseries-api sh do in this container with a chiseled image?
A Systematic Flow: Three Questions
"The container is not working" is a common name for three different problems. Each has a different first place to look, and asking the right question is much faster than trying commands at random:
| Symptom | First place to look | Typical causes |
|---|---|---|
It never starts (Created or immediately Exited) | docker run's own error message, exit codes 125–127 and 255 | Wrong entrypoint, missing file, broken script, wrong architecture |
| It starts, then dies | docker logs, docker inspect --format '{{json .State}}' | Application error, missing config, OOM, health check |
| It runs but cannot be reached | PORTS in docker ps, docker port, the network | Port not published, app listening on 127.0.0.1, wrong network |
We saw exit codes in post 3 and the causes in the third row in post 12. In this post we break the containers of Bookstore, the online bookstore used as the shared example across this site's series, on purpose and look at how each failure appears. The containers in this post (except docker debug in the later section) ran on Docker Engine on Linux.
It Never Starts: Entrypoint, Script and Architecture
The simplest failure is a typo in the entrypoint. The .NET host gives a self-explanatory message in this case:
docker run --rm --entrypoint dotnet dockerseries-bookstore-api:1.2.0 Bookstore.Ap.dll
echo "exit code: $?"The command could not be loaded, possibly because:
* You intended to execute a .NET application:
The application 'Bookstore.Ap.dll' does not exist or is not a managed .dll or .exe.
* You intended to execute a .NET SDK command:
No .NET SDKs were found.
...
exit code: 155In images that start with a script, the messages are less clear. We made a small script for Bookstore's maintenance jobs the entrypoint, in two broken versions (labs/14-exec-format): one is missing #!/bin/sh on the first line, the other was saved on Windows and has CRLF line endings (\r\n):
docker run --name dockerseries-job dockerseries-job:noshebang; echo "exit code: $?"
docker run --name dockerseries-job dockerseries-job:crlf; echo "exit code: $?"exec /job.sh: exec format error
exit code: 255
exec /job.sh: no such file or directory
exit code: 255Both are misleading. exec format error says the kernel does not know how to run the file: a script without a shebang line is, to the kernel, a binary it does not recognize. The second is even more confusing: /job.sh is right there, but the kernel looks for the interpreter in the shebang as /bin/sh\r and cannot find it because of the invisible \r at the end. Everyone who develops on Windows and puts scripts into Linux images sees this error one day. The fix is to keep scripts in the repository with LF: a *.sh text eol=lf line in .gitattributes stops Git from changing line endings on checkout. We added a .gitattributes with this rule to Bookstore's repository, with a -text exception for job-crlf.sh, which we keep broken on purpose.
The famous cause of exec format error is the wrong architecture. In post 9 we saw that an image produced without a RID on an ARM developer machine is linux/arm64; on an x64 Linux server this image dies with exec /usr/bin/dotnet: exec format error, the same message format as above, and Docker prints a warning line saying the image's platform does not match the host's. The diagnosis is one command; for an image built on an ARM machine the result is:
docker image inspect dockerseries-bookstore-api:1.2.0 --format '{{.Os}}/{{.Architecture}}'linux/arm64Watch out for one trap: Docker Desktop can run images of other architectures under emulation with QEMU, so the same image runs on the developer's Windows machine, slowly, and the error only shows up on the server where emulation is not installed. The permanent fix is the one from post 9: produce the production image in CI with the platform stated explicitly.
It Starts and Dies: Exit Code and State
We ran bookstore-web without the required API_BASE_URL variable, set to restart three times on failure:
docker run -d --name dockerseries-web --restart on-failure:3 dockerseries-bookstore-web:1.2.0
docker ps -a --filter name=dockerseries-web --format "{{.Names}} {{.Status}}"
docker inspect dockerseries-web --format 'ExitCode={{.State.ExitCode}} RestartCount={{.RestartCount}} OOMKilled={{.State.OOMKilled}}'
docker logs --tail 1 dockerseries-webdockerseries-web Exited (2) 7 seconds ago
ExitCode=2 RestartCount=3 OOMKilled=false
/docker-entrypoint.d/40-runtime-config.sh: line 6: API_BASE_URL: API_BASE_URL must be setThe three fields tell the story together: exit code 2 (the script's own error, not a signal), three retries and not OOM. docker logs keeps the output of all runs of the container together; for a container in a restart loop, the last lines belong to the last attempt. The restart policy did not fix the error; it only produced the same error four times, which is the warning from post 3 in practice.
Looking into a Shell-less Container: a Helper Container
The answer to the second question:
docker exec dockerseries-api shOCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATHA chiseled image has no shell, and since post 13 the container is read-only with its capabilities dropped. None of this costs us anything, because a container is a process and Linux namespaces can be shared. We attached another container that has tools in it (busybox) to the API's PID and network namespaces:
docker run --rm --pid container:dockerseries-api --network container:dockerseries-api busybox:1.37 \
sh -c 'ps; wget -qO- localhost:8080/health; echo; netstat -tln; ls /proc/1/root/app'PID USER TIME COMMAND
1 1654 0:00 dotnet Bookstore.Api.dll
44 root 0:00 sh -c ps; wget -qO- localhost:8080/health; echo; netstat -tln; ls /proc/1/root/app
50 root 0:00 ps
Healthy
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.11:42491 0.0.0.0:* LISTEN
tcp 0 0 :::8080 :::* LISTEN
ls: /proc/1/root/app: Permission deniedThe helper container sees the API's process as PID 1, localhost:8080 is the API itself (the same network namespace), and netstat shows which address the app listens on: :::8080, all interfaces. This line is the shortest way to diagnose the "binding to 127.0.0.1" problem from post 12. (The port on 127.0.0.11 is Docker's embedded DNS.)
The error on the last line matters: /proc/1/root is a door into the file system the process sees, and the kernel opens it only to someone with ptrace permission over the target process. The helper container is root, but its uid differs from the API's and it has no SYS_PTRACE capability. When we gave that capability to this container only:
docker run --rm --pid container:dockerseries-api --network container:dockerseries-api --cap-add SYS_PTRACE busybox:1.37 \
sh -c 'ls /proc/1/root/app | head -3; ls /proc/1/root/tmp; tr "\0" "\n" < /proc/1/environ | grep ASPNETCORE'Bookstore.Api
Bookstore.Api.deps.json
Bookstore.Api.dll
clr-debug-pipe-1-4423922-in
clr-debug-pipe-1-4423922-out
dotnet-diagnostic-1-4423922-socket
ASPNETCORE_HTTP_PORTS=8080We see the app's files, the diagnostic socket we mentioned in post 13 and the process's environment variables. The last command is one more reminder of post 13's topic: a secret you put in the environment is open to anyone who can reach the process. The API container did not change at all; only the helper container, which lives for a few seconds, got the extra permission. The same method lets you attach a container with dotnet-dump and dotnet-counters to the API; for the tools to reach the diagnostic socket, /tmp must be shared between the two containers. The Debugging series post "Analyzing Crash and Memory Dumps: dotnet-dump, SOS Commands and Taking Dumps in Containers" goes through this step by step. The Kubernetes counterpart is an ephemeral container with kubectl debug, in the Kubernetes series post "Debugging in Kubernetes: CrashLoopBackOff, ImagePullBackOff, Pending and kubectl debug".
docker debug
Docker Desktop turns the same idea into a single command. docker debug opens a shell full of tools next to the container without touching it; since Docker Desktop 4.49 it is free for all users. Since it is a Docker Desktop feature, it is not available in Docker Engine on a Linux server; the helper container method above remains there. We ran the same API image on Docker Desktop and tried it:
docker debug dockerseries-api -c "ps aux; ls /app | head -3; id; which curl"USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
app 1 0.5 0.9 3175076 77944 ? Ssl 12:46 0:00 dotnet Bookst
root 62 0.0 0.0 5312 1188 pts/0 Ss+ 12:47 0:00 /nix/var/nix/
root 68 0.0 0.0 6460 3228 pts/0 R+ 12:47 0:00 ps aux
Bookstore.Api
Bookstore.Api.deps.json
Bookstore.Api.dll
uid=0(root) gid=0(root) groups=0(root)
/nix/var/nix/profiles/default/bin/curlThe shell opened as root, it sees the app's files directly, and tools such as curl are ready. The tools are not copied into the container's file system; they live elsewhere and leave no trace once the container stops. Without -c, an interactive shell opens. You can also give an image name instead of a running container; then you work on a copy of the image.
A Case: 137 Every 30 Seconds
The experiment for the first question. The limits are values someone who wants to fit more containers on the same server might choose by looking at docker stats:
docker run -d --name dockerseries-api --network dockerseries-net --cpus 2 --memory 32m dockerseries-bookstore-api:1.2.0
docker stats --no-stream --format "{{.Name}} {{.MemUsage}} {{.MemPerc}} {{.PIDs}}" dockerseries-api
docker logs dockerseries-api 2>&1 | grep -A1 "GC memory" | tail -1dockerseries-api 22.92MiB / 32MiB 71.63% 18
Environment: Production, GC memory limit: 24 MB, processors: 2Half a minute later:
docker ps -a --filter name=dockerseries-api --format "{{.Names}} {{.Status}}"
docker inspect dockerseries-api --format '{{json .State}}' \
| jq '{Status, ExitCode, OOMKilled, StartedAt, FinishedAt, Health: .Health.Status, LastProbe: .Health.Log[-1]}'dockerseries-api Exited (137) 17 seconds ago
{
"Status": "exited",
"ExitCode": 137,
"OOMKilled": true,
"StartedAt": "2026-11-14T12:53:48.713885709Z",
"FinishedAt": "2026-11-14T12:54:24.011305045Z",
"Health": "unhealthy",
"LastProbe": {
"Start": "2026-11-14T12:54:23.905362878Z",
"End": "2026-11-14T12:54:24.010611712Z",
"ExitCode": 137,
"Output": ""
}
}137 and OOMKilled: true: the kernel's OOM killer. But memory use was under the limit and there were no requests. The clue is in two timestamps: the last health check started at 12:54:23.905 and the container died at 12:54:24.011. docker events gives the container's lifecycle second by second; we started listening for events before the container started:
docker events --filter container=dockerseries-api --format '{{.Time}} {{.Action}}'12:53:48 create
12:53:48 start
12:53:53 exec_create: dotnet Bookstore.Api.dll healthcheck
12:53:53 exec_start: dotnet Bookstore.Api.dll healthcheck
12:53:53 exec_die
12:53:53 health_status: healthy
12:54:23 exec_create: dotnet Bookstore.Api.dll healthcheck
12:54:23 exec_start: dotnet Bookstore.Api.dll healthcheck
12:54:24 exec_die
12:54:24 oom
12:54:24 die(I converted the Unix timestamps to clock times.) Here is the reason: the HEALTHCHECK we wrote for the chiseled image in post 8 starts a second .NET process inside the container every 30 seconds. That process also runs in the container's cgroup and asks for a share of the same 32 MB. The 23 MB shown by docker stats is the API alone; a second runtime starting pushed the total over the limit, and the kernel killed the biggest process in the cgroup, the API. The first health check passed while the app had not warmed up yet; the second did not. --cpus plays a part too: when we ran the same experiment with --cpus 1, the container stayed up at 32 MB and died at 28 MB. The runtime's threads and the memory set aside by Server GC, ASP.NET Core's default, grow with the number of processors it sees; the same limit behaves differently with 1 and 2 processors.
The fix is to choose limits by measuring, with headroom. We started with --memory 128m, put load on /books with 20 parallel clients from a container on the same network, and read docker stats every five seconds:
dockerseries-api 35.45MiB / 128MiB 27.69% 18
dockerseries-api 32.19MiB / 128MiB 25.15% 18
dockerseries-api 31.73MiB / 128MiB 24.79% 18
dockerseries-api 31.73MiB / 128MiB 24.79% 18
dockerseries-api 32.45MiB / 128MiB 25.35% 18
dockerseries-api 36.15MiB / 128MiB 28.24% 18
dockerseries-api Up About a minute (healthy)
OOMKilled=false RestartCount=0At most 36 MB under load; even with the second process opened by the health check, 128 MB leaves comfortable headroom. In a real application you need to make the same measurement with database connections open, under realistic load and over a long time. Note also that the same container would not have this problem in Kubernetes: Kubernetes does not run the HEALTHCHECK from the Dockerfile; it runs probes with HTTP requests from outside the container. The second classic memory limit mistake is the opposite: setting a GC heap limit (DOTNET_GCHeapHardLimit) above the container limit. The heap thinks it can grow up to that limit, and the container again dies with 137; GC settings and diagnosing OOMKilled are in the Kubernetes series post "Resource Requests and Limits: QoS, OOMKilled, CPU Throttling and the .NET GC in Containers".
In Production
Watch disk usage before it fills up. docker system df shows the space taken by images, containers, volumes and the build cache, and how much of it can be reclaimed:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 17 0 3.94GB 3.078GB (78%)
Containers 0 0 0B 0B
Local Volumes 3 0 1.931GB 1.931GB (100%)
Build Cache 139 0 3.13GB 2.293GBThe volumes showing as "100% reclaimable" on this series' lab engine is because no container was using them at that moment; BookstoreDb is in them. The RECLAIMABLE column means "not in use right now", not "safe to delete". When you clean up with commands such as docker image prune and docker builder prune, remember the volume warning from post 11; on build servers an age-based filter such as docker builder prune --filter until=168h is safer for the build cache.
Logs fill the disk too. Docker's default json-file log driver does not rotate log files; a container that runs for days and logs a lot can fill the server's disk on its own. --log-opt max-size=10m --log-opt max-file=3, or the same settings in daemon.json, set a limit. In production you need to ship logs out of the container to a central place anyway; that is the topic of the Observability series.
Watch docker events even when nothing is wrong. Sending oom, die and health_status: unhealthy events to a monitoring system beats finding out in the morning that a container restarted a hundred times overnight.
Common Pitfalls
Fixing things by hand with exec in a production container. Getting into an image that has a shell and fixing a config file solves the problem for the moment, but the change stays in the container's writable layer: it disappears with the next deploy, a restart or another node, and nobody knows why it works "sometimes". Do the diagnosis in the container and the fix in the image, the config or the Dockerfile. The --read-only from post 13 makes this habit impossible anyway.
Restarting the container with --privileged for debugging. Changing the running container can destroy the problem itself (memory state, open connections) and leaves behind a privilege someone forgets to take back. A helper container or docker debug lets you look without touching the main container.
Choosing a limit by looking only at docker stats. The case above is exactly this: stats show the main process's current use; the health check, processes opened with exec, the GC's growth under load and native memory are not visible there. The limit should leave headroom above the highest value you measured.
Reading 137 on its own as "out of memory". As we saw in post 3, docker kill and a docker stop timeout also give 137. The OOMKilled field and the oom event in docker events make the distinction.
A Small Experiment
★☆☆ Examine the script of the dockerseries-job:crlf image with a hex viewer (od -c labs/14-exec-format/job-crlf.sh | head -2) and find the \r. Convert the file to LF and rebuild the image.
★★☆ Repeat the 32 MB case, this time with the Dockerfile's HEALTHCHECK disabled (docker run --no-healthcheck). Does the container stay up? Then put load on /books; at what point and with which exit code does it die?
★★★ Start the API with -v dockerseries-diag:/tmp, install dotnet-counters in a helper container based on mcr.microsoft.com/dotnet/sdk:10.0, mount the same volume at /tmp, attach with --pid container:dockerseries-api and run dotnet-counters monitor -p 1. What does the GC heap size show under load?
With this post we have finished setting up Bookstore's containers, connecting them, protecting them and reading them when they break. In the last post of the series we look back: the legacy builder's build output, old .NET images running on port 80, the move from Debian to Ubuntu, the changes Engine 29 brought, and moving a Dockerfile from 2021 to 2026 line by line.