Multiple Environments: Override Files, include, extends and Merge Rules

Version note: The examples in this post were run with Docker Compose 5.7.1, Docker Engine 29.9.3, SQL Server 2025 (RTM-CU11) and .NET 10.

Guess First

In compose.yaml the web service is published with "8080:8080". For a production-like environment you write a second file, give web ports: ["80:8080"] and run docker compose -f compose.yaml -f compose.prod.yaml up -d. Which ports are open on the server: only 80, 80 and 8080, or does Compose fail? Second question: there is a compose.override.yaml in the project folder. When you pass files explicitly with -f compose.yaml, is it loaded too?

Why One File Is Not Enough

At the end of four posts, compose.yaml held production-like definitions, the Angular dev server, watch rules, development ports and optional tools side by side. Running the same file on CI meant opening ports 1433 and 4200 on the runner and building the dev image; taking it to production meant leaving the SQL Server port open on the host. Copying a separate file for every environment is no solution either: three copies drift apart within a few weeks and nobody knows which one is right. Compose's way is layers: the shared part in one file, each environment writes only its difference.

Bookstore's files at the end of this post:

Layers with compose.override.yaml and -f

When no -f is given at all, Compose reads compose.yaml in the project folder and, if present next to it, compose.override.yaml, and merges the second one on top of the first. That is why we moved the development additions into this file: a developer still types only docker compose up or docker compose watch.

# Development (loaded automatically by docker compose up/watch when no -f is given)
services:
  db:
    ports:
      # SSMS / sqlcmd on the host
      - "127.0.0.1:1433:1433"

  api:
    develop:
      watch:
        # A change in the API's code: build the image again and recreate the container
        - path: src/Bookstore.Api
          action: rebuild

  # Angular dev server: docker compose --profile dev watch
  web-dev:
    ...
    develop:
      watch:
        # Source files are copied into the running container; ng serve rebuilds and reloads the browser
        - path: web/src
          target: /app/src
          action: sync
          initial_sync: true
        ...

  # Optional services: docker compose --profile cache / --profile tools
  redis:
    ...
  mailpit:
    ...

The initial_sync promised in the previous post is here: when watch starts, the files in the existing web-dev container are brought in line with the ones on the host. An override file does not have to be a complete definition; for db we wrote only ports, for api only develop, the rest comes from compose.yaml. The answer to the second question:

docker compose config db | grep -E "published|host_ip"
docker compose -f compose.yaml config db | grep -c published

        host_ip: 127.0.0.1
        published: "1433"
0

As soon as -f is given, automatic loading is off: Compose merges only the files you gave, in the order you gave them. On CI and in production that is exactly what we want, so we pass compose.ci.yaml and compose.prod.yaml with -f. Instead of typing two -fs in every command, the file list can also be given with the COMPOSE_FILE environment variable (separated by : on Linux, ; on Windows); writing COMPOSE_FILE=compose.yaml:compose.prod.yaml into the .env on a server means "always the production files on this machine".

Merge Rules: What Merges, What Gets Replaced?

The answer to the first question is in the merge rules. According to the Compose Specification, maps (for example environment, healthcheck, labels) merge key by key, and for the same key the later file wins. Lists (sequences) are appended by default. ports, volumes, secrets and configs are lists, but they merge by a "unique key": target for volumes, and for ports the combination of IP, target port, published port and protocol. "8080:8080" and "80:8080" have different published ports, so they count as different entries:

# The first version of compose.prod.yaml
services:
  web:
    ports:
      - "80:8080"

docker compose -f compose.yaml -f prod-naive.yaml config web

    ports:
      - mode: ingress
        target: 8080
        published: "8080"
        protocol: tcp
      - mode: ingress
        target: 8080
        published: "80"
        protocol: tcp

The answer is "80 and 8080". Compose does not fail, because the two ports are different; on the production server 8080 silently stays open. Only someone looking at the config output or at ss -tlnp on the server notices it. The fix is two YAML tags that came with the Compose 2.24 releases: !override replaces a value entirely, bypassing the merge rules, !reset removes the value from the base file.

# Production-like: docker compose -f compose.yaml -f compose.prod.yaml up -d (post 6 goes further)
services:
  web:
    environment:
      APP_ENVIRONMENT: production
    # !override: replace the base list instead of appending to it (8080 must not stay open)
    ports: !override
      - "80:8080"

docker compose -f compose.yaml -f compose.prod.yaml config web | grep -E "published|target: 8080"

        target: 8080
        published: "80"

One more exception: command, entrypoint and healthcheck.test are not appended even when written as lists; the value in the later file replaces the earlier one. When you want to add an argument to a command, you have to write the whole command again.

Modular Compose with include

We moved everything about the database (db, migrate, sqlcmd, its volume and two secrets) into compose.db.yaml and included it from the main file:

# Bookstore: the parts every environment shares. Development additions are in compose.override.yaml,
# CI in compose.ci.yaml, a production-like setup in compose.prod.yaml.
name: composeseries

include:
  - compose.db.yaml

services:
  api:
    ...
    depends_on:
      db:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully

include (since Compose 2.20) looks like -f but does a different job. -f merges files as layers of the same model, and all relative paths are resolved against the folder of the first file. An included file, on the other hand, is loaded like a Compose application of its own: its relative paths (./db, ./secrets) are resolved against its own folder, it can have its own .env, then its resources are copied into the main model and api can depend on them as if they were in its own file. The difference shows when you move the file to another folder: the included file keeps working with its own paths. So include is for "take a part from another team or another repo as it is", -f for "the per-environment difference of the same application".

For a name conflict, the Compose Specification says a warning should be shown and the definitions should not be merged. We tried it on Compose 5.7.1: adding a db with image: mcr.microsoft.com/mssql/server:2022-latest to the main file showed no warning and the two definitions were merged, the image became 2022, the health check and secrets came from the included file. Do not rely on this behavior; if you want to change an included service, do it explicitly in an override file.

Service Inheritance with extends

migrate and sqlcmd did the same thing: run the client in the SQL Server image, reading the password from a file. Written separately, a flag fixed in one was forgotten in the other. We derived migrate from sqlcmd:

  # Ad-hoc queries, never started by up: docker compose run --rm sqlcmd -Q "SELECT COUNT(*) FROM dbo.Books"
  sqlcmd:
    image: mcr.microsoft.com/mssql/server:2025-latest
    profiles: [cli]
    entrypoint:
      - /bin/bash
      - -c
      - >-
        SQLCMDPASSWORD="$$(cat /run/secrets/sa-password)"
        BOOKSTORE_DB_PASSWORD="$$(cat /run/secrets/db-app-password 2>/dev/null)"
        exec /opt/mssql-tools18/bin/sqlcmd -C -S db -U sa -b "$$@"
      - sqlcmd
    environment:
      SQLCMDDBNAME: BookstoreDb
    secrets:
      - sa-password
    depends_on:
      db:
        condition: service_healthy

  # One-off: creates BookstoreDb and the app login, then exits with 0. Same client as sqlcmd.
  migrate:
    extends:
      service: sqlcmd
    profiles: !reset []
    command: ["-i", "/scripts/01-bookstore.sql", "-i", "/scripts/02-app-login.sql"]
    environment:
      SQLCMDDBNAME: master
    secrets:
      - db-app-password
    volumes:
      - ./db:/scripts:ro

extends' merge rules are similar to those of file layers: environment is a map, so the SQLCMDDBNAME key was overridden (migrate must connect to master, not to the not-yet-existing BookstoreDb), the secrets list was merged and migrate got both secrets, and profiles was cleared with !reset; otherwise migrate would also stay in the cli profile and never run on up. SQLCMDDBNAME is the environment variable sqlcmd reads instead of -d; since the arguments given to run go to "$$@" in the entrypoint, passing the database as a flag would be a detail to repeat on every call.

docker compose run --rm sqlcmd -W -Q "SELECT DB_NAME() AS Db, COUNT(*) AS Books FROM dbo.Books"

Db Books
-- -----
BookstoreDb 6

(1 rows affected)

extends, or YAML anchors and extension fields starting with x-? Anchors only work within the same file and are a feature of YAML itself; Compose never sees them, it only reads the expanded result. extends can also derive from a service in another file (file:) and applies Compose's merge rules. Anchors for a few repeated lines, extends for a definition shared at the service level.

CI: A Clean Database, --wait and the Exit Code

For integration tests we want an empty database on every run, no ports opened on the host and a step that reports the result through its exit code:

# CI: docker compose -f compose.yaml -f compose.ci.yaml up -d --wait, then run --rm smoke
services:
  db:
    # A clean database on every run: data lives in memory and is gone after down
    volumes: !override
      - type: tmpfs
        target: /var/opt/mssql
        tmpfs:
          size: 2g
          # SQL Server runs as uid 10001; a fresh tmpfs belongs to root
          mode: 01777

  web:
    # Nothing is published on the CI runner; the smoke test runs on the project network
    ports: !reset []

  smoke:
    image: alpine:3
    profiles: [test]
    command:
      - sh
      - -c
      - >-
        n=$$(wget -qO- http://web:8080/api/books | grep -o '"isbn"' | wc -l);
        echo "books: $$n"; test "$$n" -eq 6

We had not written the tmpfs mode on the first attempt, and SQL Server did not start: The system directory [/.system] could not be created ... Permission denied. In post 11 of the Docker series we solved the same error with docker run and tmpfs-mode=1777; the Compose counterpart is mode: 01777 (the leading zero means an octal number in YAML). We also added a health check to web (wget -q --spider http://127.0.0.1:8080/), because --wait can only tell that services with a health check are ready. The pipeline step is a single script (labs/compose/05-ci.sh):

#!/usr/bin/env bash
# CI step: start the stack with compose.ci.yaml, wait until healthy, run the smoke test; the exit code fails the job
set -euo pipefail
export COMPOSE_PROJECT_NAME=composeseries-ci
export COMPOSE_FILE=compose.yaml:compose.ci.yaml
# Always clean up, also when a step fails: the database lives on tmpfs, nothing is kept
trap 'docker compose down -v' EXIT
docker compose up -d --wait
docker compose run --rm smoke

labs/compose/05-ci.sh; echo "exit code: $?"

 Container composeseries-ci-migrate-1 Exited
 Container composeseries-ci-db-1 Healthy
 Container composeseries-ci-api-1 Healthy
 Container composeseries-ci-web-1 Healthy
books: 6
 Container composeseries-ci-web-1 Removed
 Container composeseries-ci-api-1 Removed
 Container composeseries-ci-migrate-1 Removed
 Container composeseries-ci-db-1 Removed
 Network composeseries-ci_default Removed
exit code: 0

With the images in the cache, the whole step took 32 seconds. --wait did not count migrate's Exited (0) as an error. Had the smoke test failed, run would have returned the container's exit code, set -e would have stopped the script and trap would still have cleaned up; when we tried sh -c "exit 3", run returned 3. We set COMPOSE_PROJECT_NAME separately: running the CI step on a machine where the development stack is up would recreate the development containers without a separate project name. There is a one-command alternative too: docker compose --profile test up --abort-on-container-exit --exit-code-from smoke. When we tried it, smoke waited for nothing and started before web: wget: bad address 'web:8080', books: 0, exit 1, and Compose stopped the whole stack. If you choose this way, the test service needs depends_on: web: condition: service_healthy; separating up --wait and run moves that dependency into the script.

In one attempt where we ran two SQL Servers (development and CI) on the same machine, the CI database crashed for lack of memory and --wait failed after 70 seconds with container composeseries-ci-db-1 is unhealthy. Plan the CI runner's memory and the tmpfs size together: data on tmpfs takes RAM too.

Compose or Testcontainers? An environment set up with Compose is independent of the test code: a smoke test, Playwright and manual checks can connect to the same stack at the same time, but the tests assume the environment is ready and isolation only comes from the project name. With Testcontainers the test code starts the container, every test class can get its own database and the tests need nothing but dotnet test; the price is that the environment exists only inside the tests. We prefer Testcontainers for integration tests and Compose for end-to-end scenarios where several services work together; the details are in the Integration and E2E Test series.

In Production

Write down the list of files that go to production and do not leave it to typing by hand: putting COMPOSE_FILE into the .env on the server, or fixing it in the deploy script, keeps someone from typing docker compose up -d one day and loading compose.override.yaml (Mailpit, an open port 1433) into production. Not copying the override file to the production server at all is even safer. Generating the final form of each environment's files with docker compose config and reviewing it in the pull request catches the surprises of the merge rules before production; config --hash "*" gives one hash per service and quickly shows the difference between two environments.

Common Pitfalls

• Expecting an override to replace the port list: different ports merge and the base port stays open; use !override to replace, !reset to remove.
• Thinking compose.override.yaml is loaded too after you pass -f: it is not, unless given explicitly.
• Trying to add arguments to command in an override: even as a list it does not merge, it is replaced completely.
• Copying a complete file per environment: the copies drift; write only the difference and keep the shared part in one place.
• Writing relative paths in an included file relative to the main file: the included file's paths are resolved against its own folder.
• Running the CI and development stacks with the same project name: one recreates the other's containers or deletes its data with down -v.
• Mounting tmpfs for a non-root database without mode: the container does not start.

A Small Experiment

★☆☆ Remove !override from compose.prod.yaml, run docker compose -f compose.yaml -f compose.prod.yaml up -d web and look at the PORTS column of docker compose ps. Then put !override back and run up -d again; did Compose recreate the container?

★★☆ Add APP_ENVIRONMENT: ci for web to compose.ci.yaml and extend the smoke test to also check the environment value in config.json. How did the environment map merge, did API_BASE_URL disappear?

★★★ Move compose.db.yaml into a separate folder (infra/) and turn the include into the long syntax with path and env_file. How are the paths of the secret files resolved? Try doing the same with -f infra/compose.db.yaml and compare the docker compose config output of the two approaches.

We now have a production-like file, but it still builds images on the server and says nothing about restarts, logs and resource limits. In the next post we run Bookstore with Compose on a single server: immutable-tagged images from a registry, the pull and up -d flow, log rotation, resource limits, backups and where Compose falls short.