Testing Scripts and Making Them Portable: ShellCheck, shfmt, bats-core, CI and From Bash 3.2 on macOS to Bash 5.3

Version note: The examples in this post were run on Ubuntu 24.04 LTS with Bash 5.2.21 and Bash 5.3.20 built from source, with Bash 5.3.9 in an Ubuntu 26.04 container and with Bash 3.2.57, the /bin/bash of macOS 27, using ShellCheck 0.11.0, shfmt 3.14.1, bats-core 1.14.0 and actionlint 1.7.12. All of these were the latest stable versions at the time of writing, November 2026.

Guess First

One of the deploy script's bats-core tests: it runs the deploy with an artifact whose checksum does not match, and checks that it ends with 1, prints the error message and leaves current unchanged.

@test "a checksum mismatch stops before anything is extracted" {
    publish_artifact 1.4.7
    echo "0000  bookstore-api-1.4.7-linux-x64.tar.gz" >"$STORE/bookstore-api-1.4.7-linux-x64.tar.gz.sha256"
    run "$SCRIPTS/bookstore-deploy.sh" -e staging -v 1.4.7
    [ "$status" -eq 1 ]
    [[ $output == *"checksum mismatch"* ]]
    [ "$(current)" = 1.4.6 ]
    [ ! -e "$BOOKSTORE_ROOT/releases/1.4.7" ]
}

Can this test pass in an environment where the deploy script crashes in its first lines and never gets to the checksum? The answer is in the macOS section, from a real run.

By the end of the previous post the deploy script had grown to about 300 lines and the library to 120. Trying code of this size by hand after every change does not scale, and what is tried by hand works on the machine it was tried on. In this last post of the series we protect the ops scripts of Bookstore, the online bookstore used as the shared example across this site's series, with three tools: ShellCheck, shfmt and bats-core. Then we run the same tests on Bash 3.2 on macOS, on Bash 5.3 and on Ubuntu 26.04's coreutils written in Rust, to see where portability really breaks.

ShellCheck: in the Editor and on Every Commit

We have used ShellCheck since the second post. Warnings while you type in the editor, for example with the ShellCheck extension in VS Code, are half of the job; the other half is warnings breaking the build in CI. The .shellcheckrc from the fourth post lives in the repo root, and the editor and CI read the same settings. The tests now go on the list too; ShellCheck recognises the #!/usr/bin/env bats shebang:

$ shellcheck scripts/*.sh scripts/lib/*.sh tests/*.bats tests/*.bash && echo "shellcheck: clean"
shellcheck: clean

If you have to silence a warning, do it right above the line and with a reason, not for the whole file: # shellcheck disable=SC2016 # expands when the stub runs. As SC2329 in the sixth post showed, ShellCheck can be wrong. Bash 5.3's new syntax gives another example: ${ command; } does not start a subshell, yet ShellCheck 0.11.0 says the assignment inside it may be lost (line 7 of new53.sh below):

$ shellcheck -f gcc new53.sh
new53.sh:5:7: note: Modification of count is local (to subshell caused by $(..) expansion). [SC2030]
new53.sh:6:36: note: count was modified in a subshell. That change might be lost. [SC2031]
new53.sh:7:17: note: count was modified in a subshell. That change might be lost. [SC2031]

The first two notes are right, the third is not. A new version of a language arrives before the lint tools catch up; before you use a new feature, check whether your tools understand it.

Formatting with shfmt, and a Hidden Trap

shfmt, from the mvdan/sh project, is to shell code what dotnet format or Prettier is to other code: it ends the discussion about indentation and spacing. It can take its settings from the command line, and it also reads .editorconfig; we put one in the repo root, so the editor and shfmt follow the same rules:

# shfmt reads its formatting options from here (shfmt -d, shfmt -w)
root = true

[*]
end_of_line = lf
insert_final_newline = true

[*.{sh,bash,bats}]
indent_style = space
indent_size = 4
switch_case_indent = true

shfmt -d shows the differences and returns 1 if there are any, which suits CI; shfmt -w fixes the files. Where it disagreed most with the style we kept by hand throughout the series was the spaces inside arithmetic parentheses:

$ shfmt -d scripts/bookstore-health.sh
diff scripts/bookstore-health.sh.orig scripts/bookstore-health.sh
--- scripts/bookstore-health.sh.orig
+++ scripts/bookstore-health.sh
@@ -22,7 +22,7 @@
 }

 main() {
-    if (( $# > 1 )); then
+    if (($# > 1)); then
         usage
         exit 2
     fi
@@ -40,7 +40,7 @@
     # A known environment name maps to its URL; anything else is used as a URL.
     local url="${HEALTH_URLS[$target]:-$target}"
     local curl_opts=(-fsS --max-time "$timeout" -o /dev/null)
-    if (( retries > 0 )); then
+    if ((retries > 0)); then
         # curl waits 1, 2, 4... seconds between retries.
         curl_opts+=(--retry "$retries" --retry-connrefused)
     fi

That is a matter of taste; the important difference was somewhere else. In the first post we made the last line of every script main "$@"; exit: because bash reads the whole line, it exits as soon as main returns, even if the file is changed in place while the script runs. shfmt splits that line in two:

$ shfmt -d oneline.sh
diff oneline.sh.orig oneline.sh
--- oneline.sh.orig
+++ oneline.sh
@@ -5,4 +5,5 @@
     echo "step 2: copy files"
     echo "step 3: start service"
 }
-main "$@"; exit
+main "$@"
+exit

We tend to assume a formatter does not change behaviour; here it does. I repeated the experiment from the first post with shfmt's output: while the script was waiting in sleep, I overwrote it with cp using a version that has one extra comment line at the top. For comparison, the second script has its last line wrapped in braces, also passed through shfmt:

$ cp live-split-v1.sh live.sh; ./live.sh & sleep 1; cp live-split-v2.sh live.sh; wait
[1] 14903
step 1: stop service
step 2: copy files
step 3: start service
./live.sh: line 9: o: command not found
./live.sh: line 10: syntax error near unexpected token `}'
./live.sh: line 10: `}'
$ cp live-braces-v1.sh live.sh; ./live.sh & sleep 1; cp live-braces-v2.sh live.sh; wait
[1] 14910
step 1: stop service
step 2: copy files
step 3: start service

In the version split over two lines, after main finished, bash went on reading the new content of the file from the byte where it had stopped. In the version with braces, bash has to read the { ... } block up to its closing brace before running it; even though shfmt spreads the block over several lines, the protection stays. That is why the last lines of all scripts now look like this:

{
    main "$@"
    exit
}

Testing with bats-core

bats-core is a test framework written in Bash. Every @test block is a function; if one of the commands in it fails, the test is red. run runs a command and puts the exit code into $status, the output into $output and the lines into ${lines[@]}. We installed it with a checksum as well: bats-core's GitHub release has no separate file to download, so we cloned the v1.14.0 tag and checked that the commit hash (eb7f42f8…) is the commit that GitHub's verified signed tag points to.

The first test file tested the library and ran into a surprise right away. As soon as the test sourced common.sh, all tests broke with meaningless errors:

not ok 3 run prints a shell-quoted command in a dry run
# (in test file tests/common.bats, line 27)
#   `[ "$status" -eq 0 ]' failed with status 2
# [dry-run] run rm -rf /opt/bookstore/releases/1.4.0\ old
# /home/user/bookstore-ops/tests/common.bats: line 27: [: : integer expression expected

The library's run function had replaced bats' own run; in bash, function names are global and the last definition wins. We solved it with a small helper that calls library functions in a child bash instead of in the test process. As a bonus, the exit in die ends only that child process, not the test:

# Calls a library function in a child bash. Sourcing the library into the test itself
# would replace bats' own run() with the library's run().
lib() {
    bash -c 'set -Eeuo pipefail; source "$1"; shift; "$@"' lib "$SCRIPTS/lib/common.sh" "$@"
}

@test "retry succeeds on the third attempt" {
    make_stub sleep 'exit 0'
    make_stub flaky "n=\$(( \$(cat $BATS_TEST_TMPDIR/count) + 1 )); echo \$n > $BATS_TEST_TMPDIR/count; (( n >= 3 ))"
    echo 0 >"$BATS_TEST_TMPDIR/count"
    run lib retry 5 1 flaky
    [ "$status" -eq 0 ]
    [ "$(cat "$BATS_TEST_TMPDIR/count")" -eq 3 ]
}

make_stub creates a small executable script in the test's temporary folder (BATS_TEST_TMPDIR), and that folder is at the front of PATH. This is what the plan called the PATH trick: when retry calls sleep, the stub that returns immediately runs instead of the real sleep, and the test does not wait for seconds. flaky is a command that fails on the first two calls and succeeds on the third.

For the deploy tests, test_helper.bash builds a fake server for every test: releases, config files and stubs under BATS_TEST_TMPDIR. The artifact store needs no server, because curl also downloads file:// addresses. The health endpoint is answered by a stub that stands in for curl and hands every other call to the real curl:

    # The health endpoint answers 503 while the current release has an UNHEALTHY file;
    # every other curl call (the artifact download) goes to the real curl.
    local real_curl
    real_curl=$(command -v curl)
    make_stub curl "for arg in \"\$@\"; do
    if [[ \$arg == http://127.0.0.1:5000/health ]]; then
        [[ ! -e $BOOKSTORE_ROOT/current/UNHEALTHY ]] && exit 0
        echo 'curl: (22) The requested URL returned error: 503' >&2
        exit 22
    fi
done
exec $real_curl \"\$@\""
    make_stub sudo 'exec "$@"'
    # shellcheck disable=SC2016  # expands when the stub runs
    make_stub systemctl 'echo "systemctl $*" >> "$BATS_TEST_TMPDIR/systemctl.calls"'
    export PATH="$STUBS:$PATH"

With that, every scenario we tried by hand in the seventh post became a test: a successful deploy, running the same deploy again, a checksum error, a rollback after the health check, and --rollback:

@test "a failed health check rolls back to the previous release" {
    publish_artifact 1.4.7 unhealthy
    run "$SCRIPTS/bookstore-deploy.sh" -e staging -v 1.4.7
    [ "$status" -eq 1 ]
    [[ $output == *"rolled back to 1.4.6"* ]] || false
    [ "$(current)" = 1.4.6 ]
}

$ bats tests
1..19
ok 1 log writes a UTC timestamp and the message to stderr only
ok 2 die exits with the given code
ok 3 run prints a shell-quoted command in a dry run
ok 4 retry succeeds on the third attempt
ok 5 retry gives up and returns the last exit code
ok 6 current_release prints the release that current points to
ok 7 current_release fails when there is no current release
ok 8 load_env_file exports KEY=VALUE lines and skips comments
ok 9 load_env_file refuses a secrets file that others can read
ok 10 load_env_file reports a bad line by number, not by content
ok 11 --help prints the usage and exits 0
ok 12 an unknown environment is a usage error
ok 13 a version that is not X.Y.Z is a usage error
ok 14 a dry run changes nothing
ok 15 a deploy switches current and writes RELEASE
ok 16 deploying the current release again does nothing
ok 17 a checksum mismatch stops before anything is extracted
ok 18 a failed health check rolls back to the previous release
ok 19 --rollback returns to the release before current

To see that the tests are worth something, I reverted the most important fix of the seventh post: I changed the mv -T in switch_current into a plain mv.

$ sed -i 's|run mv -T "$ROOT/current.new" "$ROOT/current"|run mv "$ROOT/current.new" "$ROOT/current"|' scripts/bookstore-deploy.sh
$ bats tests/deploy.bats
1..9
ok 1 --help prints the usage and exits 0
ok 2 an unknown environment is a usage error
ok 3 a version that is not X.Y.Z is a usage error
ok 4 a dry run changes nothing
not ok 5 a deploy switches current and writes RELEASE
# (in test file tests/deploy.bats, line 42)
#   `[ "$(current)" = 1.4.7 ]' failed
not ok 6 deploying the current release again does nothing
# (in test file tests/deploy.bats, line 51)
#   `[ "$status" -eq 0 ]' failed
# efbundle: ok
# OK http://127.0.0.1:5000/health
ok 7 a checksum mismatch stops before anything is extracted
not ok 8 a failed health check rolls back to the previous release
# (in test file tests/deploy.bats, line 68)
#   `[ "$status" -eq 1 ]' failed
not ok 9 --rollback returns to the release before current
# (in test file tests/deploy.bats, line 76)
#   `[ "$(current)" = 1.4.5 ]' failed

Without -T, mv moved current.new into the folder that current points to, and current never changed. The script reported no error and the health check passed too, since the old release was still running; only a test that checks the result catches this.

Lint and Tests in GitHub Actions

The CI job we promised throughout the series. The tool versions and checksums are pinned at the top of the workflow; we install the tools ourselves so that the result does not change when the runner image's versions do:

name: ci

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

env:
  SHELLCHECK_VERSION: v0.11.0
  SHELLCHECK_SHA256: 8c3be12b05d5c177a04c29e3c78ce89ac86f1595681cab149b65b97c4e227198
  SHFMT_VERSION: v3.14.1
  SHFMT_SHA256: 76e77641faa025814b77f153b29796b8e6fa2fca03e0c76a691608b86c7ea7bf
  BATS_VERSION: v1.14.0
  BATS_COMMIT: eb7f42f8d608ac693d7a4b67474f6714ea68cfc5

jobs:
  lint:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - name: Install ShellCheck and shfmt
        run: |
          cd "$RUNNER_TEMP"
          curl -fsSLo shellcheck.tar.xz \
            "https://github.com/koalaman/shellcheck/releases/download/$SHELLCHECK_VERSION/shellcheck-$SHELLCHECK_VERSION.linux.x86_64.tar.xz"
          echo "$SHELLCHECK_SHA256  shellcheck.tar.xz" | sha256sum -c
          tar -xJf shellcheck.tar.xz
          sudo install -m 755 "shellcheck-$SHELLCHECK_VERSION/shellcheck" /usr/local/bin/shellcheck
          curl -fsSLo shfmt "https://github.com/mvdan/sh/releases/download/$SHFMT_VERSION/shfmt_${SHFMT_VERSION}_linux_amd64"
          echo "$SHFMT_SHA256  shfmt" | sha256sum -c
          sudo install -m 755 shfmt /usr/local/bin/shfmt
      - name: ShellCheck
        run: shellcheck scripts/*.sh scripts/lib/*.sh tests/*.bats tests/*.bash
      - name: shfmt
        run: shfmt -d scripts tests

  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - name: Install bats-core
        run: |
          git clone --quiet --depth 1 --branch "$BATS_VERSION" \
            https://github.com/bats-core/bats-core.git "$RUNNER_TEMP/bats-core"
          test "$(git -C "$RUNNER_TEMP/bats-core" rev-parse HEAD)" = "$BATS_COMMIT"
          sudo "$RUNNER_TEMP/bats-core/install.sh" /usr/local
      - name: Run the tests
        run: bats tests

The checksums are the SHA-256 values that GitHub's release API gives for each file; I downloaded the x86_64 files and checked them with sha256sum -c in the lab as well. Pinning actions/checkout to a full commit SHA is the rule from the security post of the GitHub Actions series. I did not run the workflow on GitHub: actionlint 1.7.12 passed without errors, and I ran the commands of the lint and test jobs in the lab with the same versions; the shellcheck: clean and the 19 ok lines above come from that run. One detail: when bats output does not go to a terminal, it writes TAP, the line-based Test Anything Protocol format that tools can read, by default; those are the ok and not ok lines you see in CI logs.

Bash 3.2 and the BSD Tools on macOS

The answer to Guess First. I ran the tests on this Mac with only the system tools on PATH. On Bash 3.2 the deploy script crashes on the shopt -s inherit_errexit line right after its header, because that option arrived in 4.4. Still, three tests passed in the first run (detail lines removed):

$ bats tests
1..19
ok 1 log writes a UTC timestamp and the message to stderr only
ok 2 die exits with the given code
ok 3 run prints a shell-quoted command in a dry run
ok 4 retry succeeds on the third attempt
ok 5 retry gives up and returns the last exit code
ok 6 current_release prints the release that current points to
ok 7 current_release fails when there is no current release
not ok 8 load_env_file exports KEY=VALUE lines and skips comments
not ok 9 load_env_file refuses a secrets file that others can read
ok 10 load_env_file reports a bad line by number, not by content
not ok 11 --help prints the usage and exits 0
not ok 12 an unknown environment is a usage error
not ok 13 a version that is not X.Y.Z is a usage error
not ok 14 a dry run changes nothing
not ok 15 a deploy switches current and writes RELEASE
not ok 16 deploying the current release again does nothing
ok 17 a checksum mismatch stops before anything is extracted
ok 18 a failed health check rolls back to the previous release
not ok 19 --rollback returns to the release before current

17 and 18 passed for the wrong reason. The script exited with 1, the expected code was 1, and [[ $output == *"checksum mismatch"* ]] did not stop the test even though it was false. The "gotchas" page of the bats-core documentation explains this: before Bash 4.1, set -e does not stop at a failing [[ ]] or (( )) unless it is the last command of the function. Its advice is to append || false; we added it to all our [[ ]] checks, and 17, 18 and 10 honestly turned red. This is not only a macOS problem but a good general lesson: a test must not stop at the expected exit code, it must check the result too.

The rest of the breakage comes from individual tools. What macOS's Bash 3.2 lacks, and how the GNU and BSD tools differ:

$ echo "$BASH_VERSION"
3.2.57(1)-release
$ bash -c 'mapfile -t lines < deploy.env; echo "${#lines[@]}"'
bash: mapfile: command not found
0
$ bash -c 'env_name=STAGING; echo "${env_name,,}"'
bash: ${env_name,,}: bad substitution
$ bash -c 'set -u; args=(); echo "args: ${args[@]}"'; echo "exit=$?"
bash: args[@]: unbound variable
exit=127
$ bash -c 'shopt -s inherit_errexit'; echo "exit=$?"
bash: line 0: shopt: inherit_errexit: invalid shell option name
exit=1
$ stat -c %a deploy.env
stat: illegal option -- c
usage: stat [-FLnq] [-f format | -l | -r | -s | -x] [-t timefmt] [file ...]
$ stat -f %Lp deploy.env
644
$ sed -i 's/staging/production/' deploy.env
sed: 1: "deploy.env
": extra characters at the end of d command
$ sed -i '' 's/staging/production/' deploy.env && cat deploy.env
production
$ date -u -d '1 day ago' +%F
date: illegal option -- d
usage: date [-jnRu] [-I[date|hours|minutes|seconds|ns]] [-f input_fmt]
            [ -z output_zone ] [-r filename|seconds] [-v[+|-]val[y|m|w|d|H|M|S]]
            [[[[mm]dd]HH]MM[[cc]yy][.SS] | new_date] [+output_fmt]
$ mkdir r1 && ln -s r1 current.new && mv -T current.new current
mv: illegal option -- T
usage: mv [-f | -i | -n] [-hv] source target
       mv [-f | -i | -n] [-v] source ... directory
$ flock -n 9
bash: flock: command not found

The way declare -A silently turns into an indexed array, which we saw in the second post, belongs on this list too. Expanding an empty array under set -u is an error on 3.2 but not on the lab's 5.2; a difference that only shows up on macOS. BSD sed -i expects a backup suffix and takes 's/.../' as the suffix and the file name as the command. On this Mac readlink -f, sort -V and sha256sum worked; stat -c, find -printf, mv -T, date -d, flock and timeout are missing.

The decision that follows: Bookstore's ops scripts are written for a Linux server and do not have to run on macOS. Instead of leaving that as a silent assumption, every script now checks it explicitly at the top:

if ((BASH_VERSINFO[0] < 5)); then
    echo "${0##*/}: needs Bash 5 or newer, this is $BASH_VERSION" >&2
    exit 1
fi
set -Eeuo pipefail
shopt -s inherit_errexit

$ ./scripts/bookstore-deploy.sh --help; echo "exit=$?"
bookstore-deploy.sh: needs Bash 5 or newer, this is 3.2.57(1)-release
exit=1

The check comes before the set line, because shopt -s inherit_errexit fails by itself on 3.2. For a developer on a Mac the way forward is Homebrew's bash, 5.3.20 at the time of writing according to the formulae.brew.sh API, or running the tests in a Linux container. Homebrew's bash goes on PATH and #!/usr/bin/env bash finds it; that was one reason the first post chose env over #!/bin/bash. But a newer bash does not turn BSD stat into GNU stat; for these scripts the realistic test environment is Ubuntu in CI or a container locally. The same goes for readers on Windows: Git Bash is not a Linux server, so run the tests in Ubuntu under WSL or in CI.

What Came with Each Bash Version: From 4.0 to 5.3

From the NEWS file in Bash's source package, the additions most useful for scripting, with years taken from the tarball dates on ftp.gnu.org:

I tried the two 5.3 additions that matter most for scripts side by side with 5.2. ${ command; } captures a command's output like $(...), but without a subshell; assignments inside it survive. GLOBSORT sets how glob results are sorted; as promised in the third post, numeric only helps with names made entirely of digits:

#!/usr/bin/env bash
cd /tmp/globsort || exit 1
echo "bash $BASH_VERSION"
count=0
out=$(count=$((count + 1)); echo "release 1.4.7")
echo "\$(...):      out=$out count=$count"
out=${ count=$((count + 1)); echo "release 1.4.7"; }
echo "\${ ...; }:   out=$out count=$count"
echo "default:     $(echo releases/*)"
GLOBSORT=numeric
echo "numeric:     $(echo releases/*)"
GLOBSORT=-mtime
echo "-mtime:      $(echo releases/*)"
cd /tmp/globsort/builds || exit 1
GLOBSORT=
echo "builds, name:    $(echo *)"
GLOBSORT=numeric
echo "builds, numeric: $(echo *)"

$ ./new53.sh
bash 5.2.21(1)-release
$(...):      out=release 1.4.7 count=0
./new53.sh: line 7: ${ count=$((count + 1)); echo "release 1.4.7"; }: bad substitution
${ ...; }:   out=release 1.4.7 count=0
default:     releases/1.4.10 releases/1.4.2 releases/1.4.9
numeric:     releases/1.4.10 releases/1.4.2 releases/1.4.9
-mtime:      releases/1.4.10 releases/1.4.2 releases/1.4.9
$ PATH="/opt/bash-5.3/bin:$PATH" ./new53.sh
bash 5.3.20(1)-release
$(...):      out=release 1.4.7 count=0
${ ...; }:   out=release 1.4.7 count=1
default:     releases/1.4.10 releases/1.4.2 releases/1.4.9
numeric:     releases/1.4.10 releases/1.4.2 releases/1.4.9
-mtime:      releases/1.4.10 releases/1.4.9 releases/1.4.2
builds, name:    10 2 9
builds, numeric: 2 9 10

5.2's behaviour is instructive too: it did not reject the new syntax while parsing; it printed "bad substitution" when it reached the line, and the script, which has no set -e, went on with the previous value of out. On 5.3, -mtime sorted the releases from newest to oldest; for version numbers you still need sort -V. source -p /tmp/globsort mylib.sh also looks for the library in the given folder on 5.3, while 5.2 answers with "source: -p: invalid option".

Using these features ties a script to the oldest bash it will run on. At the time of writing, the package repositories looked like this; the macOS and Ubuntu rows come from my own runs, Homebrew from formulae.brew.sh and the rest from Repology:

As long as Bookstore's server runs Ubuntu 24.04, the target is 5.2; using anything specific to 5.3 breaks the script on a server running 24.04. Our BASH_VERSINFO check asks for 5, because the tests run on 5.2 and 5.3; using 5.3's features makes sense once the servers move to 26.04.

Ubuntu 26.04: Bash 5.3 and Rust Coreutils

Ubuntu 26.04 LTS comes with Bash 5.3 and, by default, with uutils, the coreutils rewritten in Rust; the GNU versions are still installed under names such as gnuls and gnuln. I ran the tests in a 26.04 container too:

$ grep PRETTY_NAME /etc/os-release; echo "$BASH_VERSION"; ls --version | head -n 1
PRETTY_NAME="Ubuntu 26.04.1 LTS"
5.3.9(1)-release
ls (uutils coreutils) 0.10.0
$ bats tests
1..19
ok 1 log writes a UTC timestamp and the message to stderr only
ok 2 die exits with the given code
ok 3 run prints a shell-quoted command in a dry run
ok 4 retry succeeds on the third attempt
ok 5 retry gives up and returns the last exit code
ok 6 current_release prints the release that current points to
ok 7 current_release fails when there is no current release
ok 8 load_env_file exports KEY=VALUE lines and skips comments
ok 9 load_env_file refuses a secrets file that others can read
ok 10 load_env_file reports a bad line by number, not by content
ok 11 --help prints the usage and exits 0
ok 12 an unknown environment is a usage error
ok 13 a version that is not X.Y.Z is a usage error
ok 14 a dry run changes nothing
ok 15 a deploy switches current and writes RELEASE
ok 16 deploying the current release again does nothing
ok 17 a checksum mismatch stops before anything is extracted
ok 18 a failed health check rolls back to the previous release
ok 19 --rollback returns to the release before current

All green; stat -c, sort -V, mv -T, sha256sum and mktemp gave the same results with uutils. But the difference we saw with strace in the seventh post does not show here: 26.04's ln -sfn deletes first and creates afterwards. That is the kind of difference tests cannot catch; at the end of the test current points to the right place, and the gap of a few milliseconds in between is invisible. Our script is not affected because it uses mv -T. A green test proves that the result you test for is the same, not that the behaviour is. The other effects of moving to Ubuntu 26.04, such as sudo-rs and nftables, are in the post of the Linux and the Shell series about going from 24.04 to 26.04.

When to Fall Back to POSIX sh

This series was written for Bash, but in some places there is no Bash: in Alpine-based container images the default shell is BusyBox's ash, and on Debian and Ubuntu /bin/sh is dash. If a Dockerfile's ENTRYPOINT script or a package's install script runs with #!/bin/sh, falling back to POSIX sh is the right call. An entrypoint written out of Bash habit:

#!/bin/sh
# Container entrypoint: waits for the database, then starts the API.
retries=0
until [[ -f /run/db-ready ]]; do
    retries=$((retries + 1))
    if (( retries > 30 )); then
        echo "database not ready" >&2
        exit 1
    fi
    sleep 1
done
exec dotnet Bookstore.Api.dll "$@"

$ timeout 3 sh entrypoint.sh; echo "exit=$?"
entrypoint.sh: 4: [[: not found
entrypoint.sh: 6: retries: not found
entrypoint.sh: 4: [[: not found
entrypoint.sh: 6: retries: not found
entrypoint.sh: 4: [[: not found
entrypoint.sh: 6: retries: not found
exit=124
$ ls
30
entrypoint.sh

dash took [[ for a command name and could not find it; until read that as "not ready yet". Worse is (( retries > 30 )): dash read it as two nested subshells and a redirection, looked for a command called retries, and created an empty file called 30 on every round. The script would never have stopped after 30. ShellCheck looks at the shebang and catches this:

$ shellcheck -f gcc entrypoint.sh
entrypoint.sh:4:7: warning: In POSIX sh, [[ ]] is undefined. [SC3010]
entrypoint.sh:6:8: warning: In POSIX sh, standalone ((..)) is undefined. [SC3006]

The POSIX equivalents are [ -f /run/db-ready ] and [ "$retries" -gt 30 ]. Our rule is simple: if the code will run where bash is not guaranteed (a container entrypoint, a package script, other teams' machines), use POSIX sh and #!/bin/sh; our own ops scripts on our own servers use Bash, because arrays, [[ ]], pipefail and traps make them safer. Staying in between, writing #!/bin/sh and using bashisms, is the worst choice; as we saw in the first post, it works on one system and silently breaks on another.

In Production

Make lint and tests a required check on pull requests. A job that runs in CI and that nobody looks at is no better than no job at all. Mark the lint and test jobs as required checks in branch protection, so that no script with a ShellCheck warning gets into main.

Know how real your tests are. Ours use stubs for systemctl, sudo and the health endpoint; they test the script's logic, not its integration with the server. What complements them is running a real deploy and a real rollback on staging from time to time. Like the restore test in the seventh post, it is the only way to try a mechanism before you actually need it.

Pin tool versions and upgrade them deliberately. A new ShellCheck version brings new warnings, like SC2329 in 0.11; pinning the version in the workflow keeps an unrelated pull request from turning red one morning. Upgrade in a separate pull request and clean up the new warnings there.

Write down the script's target platform. A line such as "Ubuntu 24.04+, Bash 5+, GNU coreutils, util-linux" in the header comment or the README saves the next person from trying the script on macOS or Alpine and searching for the reason it broke. The BASH_VERSINFO check is the executable version of that line.

Common Pitfalls

Testing only the exit code. In the macOS run above, three tests passed although the script never ran, because the expected exit code was 1 as well. Every test should check both the exit code and the result (output, files, symlinks), and in bats every [[ ]] check should end with || false.

Assuming a formatter is independent of behaviour. shfmt splitting main "$@"; exit in two changed not just the layout but how the script behaves when it is changed in place. When you apply a formatter to the whole code base for the first time, read the diff line by line and run the tests.

Giving library functions generic names. Our run clashed with bats' run; the same can happen with names like log, die or cleanup when you source another library. The Google Shell Style Guide suggests a prefix such as bookstore::run for library functions; in a growing library, doing it from the start is cheaper than changing every script later.

Running the tests only on your own machine. A script written and tested on a Mac has not been tested when it first runs on a Linux server. The environment the tests run in should be the one the script will run in: an Ubuntu 24.04 runner in CI and a container locally.

A Small Experiment

★☆☆ Write a test file for bookstore-health.sh: HEALTH_TIMEOUT=abc should end with 2, an unknown environment name should be used as a URL, and a health endpoint returning 503 should end with 1. Fake curl with make_stub.

★★☆ Add a matrix to CI so that the tests also run with Bash 5.3. Hint: use the ubuntu:26.04 container in the job, or add a step that builds 5.3 from source; compare what each costs in time and caching.

★★★ Write tests for bookstore-backup.sh: replace sqlcmd with a stub that writes the query it receives to a file, then check that the query contains BACKUP DATABASE [BookstoreDb] TO DISK, that the script refuses to run any query when given DB_NAME='x]; DROP DATABASE y; --', and that files aged with touch -d are deleted.

This post completes the Bash Scripting series. We started with the shebang and exit codes and went through quoting, loops, functions, data flow and error handling to a real deploy and backup script, and finally to testing those scripts and wiring them into CI. Bookstore's ops folder is now code that passes every change through ShellCheck, shfmt and 19 tests and states its target platform openly. How to use scripts inside a CI pipeline is covered in the GitHub Actions series; managing the server itself is covered in the Linux and the Shell series.