Compose in Production on a Single Server: Image Tags, Restart, Log Rotation, Resource Limits and the Limits of Compose

Version note: The examples in this post were run with Docker Compose 5.7.1, Docker Engine 29.9.3, Docker Registry 3, SQL Server 2025 (RTM-CU11) and .NET 10; the server was set up on a separate engine standing in for Docker Engine on a Linux machine.

Guess First

Bookstore runs with Compose on a single Linux server. You write the new release's tag into .env and run docker compose pull && docker compose up -d. How long does a user who refreshes the page meanwhile see errors: not at all, less than a second, or more than 10 seconds? Second question: you restart the server for an update. When Docker stops the SQL Server container, does the database shut down cleanly?

The Server Does Not Build: Registry and Immutable Tags

So far the images were produced with build: on the machine running docker compose. On a production server that has three problems: the server needs the source code and the SDKs, every server builds the "same" release itself and may actually run different images, and when something goes wrong it is hard to tell from the image which code is running. The "Build Once, Deploy Many" principle from the CI/CD Fundamentals series applies here too: the image is built once on CI, put into a registry with an immutable tag, and every environment pulls the same image.

In the base file we added an image: name next to build:; when both are written, Compose tags the build result with that name:

  api:
    build:
      context: .
      dockerfile: src/Bookstore.Api/Dockerfile
      args:
        APP_VERSION: ${TAG:-0.0.0-dev}
    # build + image: the built image gets this name, so CI can push it (docker compose push api web)
    image: ${REGISTRY:-localhost:5000}/bookstore/api:${TAG:-dev}
  ...
  web:
    build: ./web
    image: ${REGISTRY:-localhost:5000}/bookstore/web:${TAG:-dev}

The release step on CI is two commands. In the example the registry is a local Registry 3 container; in reality it would be GHCR, Docker Hub or Azure Container Registry. The GitHub Actions series post "Deploying with GitHub Actions 2: Your Own Linux Server via SSH, Docker Images and GHCR" sets up the same flow, connecting to the server over SSH, as a workflow.

REGISTRY=localhost:5000 TAG=1.3.0 docker compose build api web
REGISTRY=localhost:5000 TAG=1.3.0 docker compose push api web

 Image localhost:5000/bookstore/api:1.3.0 Built
 Image localhost:5000/bookstore/web:1.3.0 Built
 localhost:5000/bookstore/api:1.3.0 Pushed
 localhost:5000/bookstore/web:1.3.0 Pushed 

The service names after push are required: a push without names also tries to send db's mcr.microsoft.com/mssql/server image. TAG also goes to the Dockerfile as the APP_VERSION build argument and is written into the image's org.opencontainers.image.version label; that is where we will read which release runs on the server. The production file, on the other hand, removes the build entirely:

# Production on a single server: COMPOSE_FILE=compose.yaml:compose.prod.yaml in the server's .env.
# The server never builds: images come from the registry with an immutable tag (TAG=1.3.0, never latest).
x-logging: &default-logging
  driver: json-file
  options:
    # Rotate: at most 3 files of 10 MB per container, otherwise the log grows until the disk is full
    max-size: "10m"
    max-file: "3"

services:
  db:
    # Time for a clean shutdown (checkpoint) of a large database before Docker sends SIGKILL; the default is 10 s
    stop_grace_period: 60s
    environment:
      MSSQL_MEMORY_LIMIT_MB: "2048"
    deploy:
      resources:
        limits:
          memory: 3g
    logging: *default-logging

  migrate:
    logging: *default-logging

  api:
    build: !reset null
    image: ${REGISTRY:?set REGISTRY}/bookstore/api:${TAG:?set TAG}
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512m
    logging: *default-logging

  web:
    build: !reset null
    image: ${REGISTRY:?set REGISTRY}/bookstore/web:${TAG:?set TAG}
    environment:
      APP_ENVIRONMENT: production
    # !override: replace the base list instead of appending to it (8080 must not stay open)
    ports: !override
      - "80:8080"
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: 128m
    logging: *default-logging

build: !reset null turns the service into one with only an image:: even without source code on the server, Compose does not try to build anything and pulls the missing image from the registry. ${TAG:?set TAG} is the required-variable pattern from post 3: if you forget the tag, Compose stops without changing anything. The reason not to use latest in production is the difference between tag and digest from post 4 of the Docker series: latest points to one image today and another one tomorrow, and there is no previous value you could "go back" to.

The Folder on the Server and Secret Permissions

Not the whole repo goes to the server, only what is needed to run: three Compose files (compose.override.yaml is deliberately missing), config/, the db/ scripts, secrets/ and an .env:

COMPOSE_PROJECT_NAME=composeseries-prod
COMPOSE_FILE=compose.yaml:compose.prod.yaml
REGISTRY=localhost:5000
TAG=1.3.0

In post 3 we left the secret files at 644 for development convenience. On the server we opened each file only to the group of the container user that reads it: the SQL Server image runs with uid and gid 10001, Bookstore.Api with uid and gid 1654.

chmod 700 secrets
chown root:10001 secrets/sa-password.txt secrets/db-app-password.txt
chown root:1654 secrets/api-connection-string.txt
chmod 640 secrets/*.txt
ls -ln secrets

-rw-r-----    1 0        1654           117 <date> api-connection-string.txt
-rw-r-----    1 0        10001           27 <date> db-app-password.txt
-rw-r-----    1 0        10001           28 <date> sa-password.txt

The groups 10001 and 1654 do not exist on the host, but permission checks work by number. The folder being 700 does not affect the containers: Docker sets up the bind mount as root, and the container only sees the file mounted for it. Other users on the server can neither enter the folder nor read the files. We started the stack with this layout; all three services read their secrets.

pull, up -d --wait and Rollback

The deploy step is a small script (labs/compose/06-deploy.sh); CI runs it on the server over SSH:

#!/usr/bin/env bash
# Runs on the server, in the folder with compose.yaml, compose.db.yaml, compose.prod.yaml, config/, db/, secrets/ and .env.
# usage: ./06-deploy.sh 1.3.1   (rollback: the same command with the previous tag)
set -euo pipefail
tag="${1:?usage: $0 <tag>}"
sed -i "s/^TAG=.*/TAG=${tag}/" .env
docker compose pull
docker compose up -d --wait
docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Status}}"

pull comes first: the old release keeps running while the images download, and if the download fails nothing changes. Then up -d --wait: it recreates the changed services and waits until all of them are healthy; if not, the script ends with a non-zero exit code and the pipeline turns red. After deploying 1.3.1 we went back to 1.3.0:

./06-deploy.sh 1.3.0
docker inspect composeseries-prod-api-1 --format '{{index .Config.Labels "org.opencontainers.image.version"}}'

 Image localhost:5000/bookstore/api:1.3.0 Pulled
 Image localhost:5000/bookstore/web:1.3.0 Pulled
 Container composeseries-prod-api-1 Recreate
 Container composeseries-prod-web-1 Recreate
 ...
 Container composeseries-prod-api-1 Healthy
 Container composeseries-prod-web-1 Healthy
SERVICE   IMAGE                                        STATUS
api       localhost:5000/bookstore/api:1.3.0           Up 7 seconds (healthy)
db        mcr.microsoft.com/mssql/server:2025-latest   Up 21 seconds (healthy)
web       localhost:5000/bookstore/web:1.3.0           Up 1 second (healthy)
1.3.0

The reason rollback is this ordinary is the immutable tag: 1.3.0 is still in the registry, still with the same content. There is one condition: the database schema must be backward compatible. Had 1.3.1 dropped a column, going back to the 1.3.0 image would bring the application back but not the schema.

How Long Is the Outage?

The experiment for the first question: with 1.3.0 running we moved to 1.3.1 and called /api/books about every 0.1 seconds meanwhile.

     62 000
     87 200
      1 502

63 of 150 requests failed; the first error at second 1.2, the last at second 12.7. An outage of 11.5 seconds. The output of up -d shows why:

 Container composeseries-prod-api-1 Recreate
 Container composeseries-prod-api-1 Recreated
 Container composeseries-prod-web-1 Recreate
 Container composeseries-prod-web-1 Recreated
 Container composeseries-prod-db-1 Waiting
 Container composeseries-prod-db-1 Healthy
 Container composeseries-prod-migrate-1 Starting
 ...
 Container composeseries-prod-migrate-1 Exited
 Container composeseries-prod-api-1 Starting
 Container composeseries-prod-api-1 Started
 Container composeseries-prod-api-1 Waiting
 Container composeseries-prod-api-1 Healthy
 Container composeseries-prod-web-1 Starting
 Container composeseries-prod-web-1 Started 

Compose stops and recreates both containers, then walks the dependency chain from the start: migrate runs again, the API is waited on until healthy, and only then does web start. During all of that the web container does not exist and connections are refused (000). For comparison we recreated only the API without touching its dependencies (docker compose up -d --no-deps --force-recreate api): only one of 150 requests got a 502, because web stayed up and the API was listening again in under a second.

With a single replica, Compose has no zero downtime; what you can do is shorten the outage. Updating services one by one and in the right order (migration first, then the API, web last) and recreating web only when web's image changed brings the time down from seconds to a single request; the price is that the deploy script carries Compose's dependency chain itself. Real zero downtime needs two copies and a proxy in front that switches the traffic: the Deployment Strategies series post "Blue-Green Deployment: Two Environments with Nginx, App Service Slots and Instant Switchback" builds that with nginx.

Restart, Log Rotation and Resource Limits

restart: unless-stopped was already in the base file: if a container crashes or the Docker service restarts, the container comes back, unless you stopped it with stop. For logs, Docker's default json-file driver does not rotate; a container's log file can grow until the disk is full. To give all services the same setting we used an x- extension field and a YAML anchor:

docker inspect composeseries-prod-api-1 --format "{{json .HostConfig.LogConfig}} memory={{.HostConfig.Memory}} nanocpus={{.HostConfig.NanoCpus}}"
docker compose logs api | grep "GC memory"
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}" composeseries-prod-api-1 composeseries-prod-web-1

{"Config":{"max-file":"3","max-size":"10m"},"Type":"json-file"} memory=536870912 nanocpus=1000000000
api-1  | 14:21:14 info: Bookstore.Api[0] Environment: Production, GC memory limit: 384 MB, processors: 1
NAME                       MEM USAGE / LIMIT
composeseries-prod-api-1   36.09MiB / 512MiB
composeseries-prod-web-1   8.977MiB / 128MiB

deploy.resources.limits was once only honored by Swarm; today's Compose applies it on a single engine too, and the container got 512 MB and 1 CPU. .NET's GC saw the limit and set the heap to 384 MB (75% of the limit), and Environment.ProcessorCount became 1; we saw the same behavior in post 8 of the Docker series and in the Kubernetes series post "Resource Requests and Limits: QoS, OOMKilled, CPU Throttling and the .NET GC in Containers". For SQL Server we also set MSSQL_MEMORY_LIMIT_MB: its default is 80% of physical memory, and we want to pin the memory SQL Server uses explicitly below the container limit.

When the Server Restarts, and How SQL Server Shuts Down

We restarted the Docker service; before that we had stopped web by hand with docker compose stop web:

SERVICE   STATUS
api       Up 41 seconds (healthy)
db        Up 41 seconds (healthy)
migrate   Exited (0) About a minute ago
web       Exited (0) 44 seconds ago

The API and the database came back, the web we stopped by hand did not: that is the "unless" in unless-stopped. If you reboot the server without restarting a service you stopped for maintenance, the system comes up half-way; if you want it back in every case, use always. One more detail: the restart policy is applied by Docker, not Compose. When the Docker service starts, it starts containers without looking at the depends_on order, migrate does not run at all and the API can be ready before the database. The SqlClient retry from post 2 pays off here too.

The second question surprised us. In the development environment the database's exit code after docker compose stop was 137. We tried it with the 60-second stop_grace_period from the production file:

docker compose exec db ps -eo pid,ppid,comm
s=$(date +%s); docker compose stop db; echo "stop took $(( $(date +%s)-s )) s"
docker inspect composeseries-prod-db-1 --format "exit={{.State.ExitCode}}"

    PID    PPID COMMAND
      1       0 bash
     10       1 sqlservr
     12      10 sqlservr
 Container composeseries-prod-db-1 Stopped
stop took 60 s
exit=137

60 seconds of waiting and still SIGKILL. The process list shows why: the image's entrypoint is a bash script called launch_sqlservr.sh; it is PID 1, starts sqlservr in the background and waits. The SIGTERM sent by docker stop goes to PID 1, that is, to bash; bash does not pass it on to sqlservr. This is the SQL Server image's version of "if the application is started behind a shell, the signal never arrives" from post 3 of the Docker series. Every stop and every server reboot ended like a power cut for the database. In compose.db.yaml we made the entrypoint sqlservr itself:

  db:
    image: mcr.microsoft.com/mssql/server:2025-latest
    # The image's entrypoint is a bash script that does not pass SIGTERM on to sqlservr: docker stop waited
    # for the timeout and killed it (exit 137). As PID 1, sqlservr shuts down cleanly in under a second.
    entrypoint: ["/opt/mssql/bin/sqlservr"]

 Container composeseries-prod-db-1 Stopped
stop took 0 s
exit=0
db-1  | 2026-11-14 14:27:24.35 spid51s     Service Broker manager has shut down.

The shutdown took under a second, exit code 0 and an orderly shutdown in the log. With this change we skip two things the launch script does: the permission check at startup and the /mssql-server-setup-scripts.d support; we use neither. We also tried a first start from an empty volume, and the database and migrate worked without problems.

Backups: BACKUP DATABASE and Compose Jobs

Backing up the volume itself (the tar method from post 11 of the Docker series) does not give a consistent copy of a running SQL Server. The right way is the database's own BACKUP DATABASE command. We wrote it with the jobs that arrived partially in Compose 5.6: a job is a container that runs to completion and is not started by up. For now only manually triggered jobs are supported; scheduled (schedule) jobs are not there yet.

jobs:
  # Manual job (Compose 5.6+): docker compose run backup
  backup:
    image: mcr.microsoft.com/mssql/server:2025-latest
    entrypoint:
      - /bin/bash
      - -c
      - >-
        SQLCMDPASSWORD="$$(cat /run/secrets/sa-password)"
        exec /opt/mssql-tools18/bin/sqlcmd -C -S db -U sa -b -Q
        "BACKUP DATABASE BookstoreDb TO DISK = N'/var/opt/mssql/backup/BookstoreDb-$$(date -u +%Y%m%d-%H%M%S).bak' WITH COMPRESSION, CHECKSUM"
    secrets:
      - sa-password
    depends_on:
      db:
        condition: service_healthy
    triggers:
      manual: true

docker compose run --rm backup
mkdir -p backup && docker compose cp db:/var/opt/mssql/backup/. backup/

Processed 416 pages for database 'BookstoreDb', file 'BookstoreDb' on file 1.
Processed 2 pages for database 'BookstoreDb', file 'BookstoreDb_log' on file 1.
BACKUP DATABASE successfully processed 418 pages in 0.076 seconds (42.917 MB/sec).
 composeseries-prod-db-1 Copied composeseries-prod-db-1:/var/opt/mssql/backup/. to backup/

On every up, Compose reminds you that jobs are not started: jobs are not started by up; trigger them with `docker compose run`: backup. The backup file is first created on SQL Server's own file system, that is, in the same volume as the database; if the volume is lost, so is the backup. docker compose cp brings it to the host, and from there moving it off the server (object storage, another machine) is a must. For scheduling, the host's cron or a systemd timer can call docker compose run --rm backup for now. A backup is worth as much as it can be restored: verifying the file with RESTORE VERIFYONLY and regularly restoring the backup into another environment to try it matters as much as taking it.

In Production

A single server is a job Compose does well: a few services, one database, low and predictable traffic. The signs that suggest moving to Kubernetes: shortening the outage to a single request is not enough and you need rolling updates; a single server going down is not acceptable, so you need at least two nodes; the traffic needs horizontal scaling; several teams want to deploy their own services independently on the same platform. If none of these apply, Kubernetes' operational cost takes more than it gives; the first post of the Kubernetes series, "What Is Kubernetes and When Don't You Need It?", discusses this decision in detail. While staying with Compose, watch the disk (logs, old images, backups), security updates of the base images (even SQL Server's 2025-latest tag should be pinned to a CU release in production) and updating Docker Engine itself.

Common Pitfalls

• Using latest or build: in production: you cannot know what is running and there is no release left to go back to.
• Leaving .env and the secret files readable by everyone on the server: open the files only to the group of the relevant container user.
• Typing docker compose down -v out of habit on a server: the database volume and the backups in it go together.
• Believing up -d has no downtime: because of the dependency chain web was recreated too and did not answer for 11.5 seconds.
• Not configuring the log driver: json-file does not rotate by default.
• Assuming a database image with a shell-script entrypoint receives SIGTERM: a longer stop_grace_period is not enough; verify that the signal reaches the process with exit=0.
• Keeping the backup only in the database's volume: the same failure takes both.

A Small Experiment

★☆☆ Run 06-deploy.sh with a tag that does not exist (9.9.9). At which step does it stop, is the running release affected? The script had already changed .env; what does the next docker compose up -d typed by hand do? Change the script so that it handles this case.

★★☆ Rewrite the deploy script to shorten the outage: first docker compose run --rm migrate, then up -d --no-deps --wait api, and web last, only if its image changed. Measure the new outage with the loop that calls /api/books.

★★★ Write a restore job: a restore job that loads the latest .bak file with RESTORE DATABASE BookstoreDb ... WITH REPLACE. What happens if the API is running during it? Build an order with Compose commands that stops the API before the job and starts it afterwards, and measure the time.

In this post we used Compose 5.3's init containers, 5.6's jobs and 5.7's secret warnings; in an older project, though, you will still see a docker-compose.yml starting with version: "3.8" and docker-compose up. In the last post of the series we trace Compose's evolution from v1 to v5, the Compose Specification, and put together a checklist for converting an old file to today's form.