Error Handling: How set -euo pipefail Really Behaves, trap, Cleanup, Retry and Locking

Version note: The examples in this post were run on Ubuntu 24.04 LTS with Bash 5.2.21 and Bash 5.3.20 built from source, using ShellCheck 0.11.0 and flock from util-linux 2.39.

Guess First

The script starts with set -euo pipefail, which we take to mean "stop if any command fails". What happens when the artifact is missing?

#!/usr/bin/env bash
set -euo pipefail
check_artifact() {
    ls /var/tmp/bookstore/bookstore-api-9.9.9-linux-x64.tar.gz
    echo "artifact found"
}
if check_artifact; then
    echo "deploying"
fi

Does the script stop at the ls error, or does it print deploying? What is the exit code? The answer is in the next section.

While preparing the previous post, in an attempt where I had forgotten to add start_log to the library, the deploy script printed start_log: command not found and carried on as if nothing had happened. That is Bash's default: when a command fails, it moves on to the next one. In this post we change that default and look at the limits of the change: the cases set -e does not catch, set -u, pipefail and inherit_errexit, an ERR trap that shows where the error happened, cleanup with an EXIT trap, retries and timeout, one deploy at a time with flock, and a deploy that does no harm when it is run again. The examples again come from the deploy script of Bookstore, the online bookstore used as the shared example across this site's series.

What set -e Really Does

The answer to Guess First:

$ ./predict.sh; echo "exit=$?"
ls: cannot access '/var/tmp/bookstore/bookstore-api-9.9.9-linux-x64.tar.gz': No such file or directory
artifact found
deploying
exit=0

set -e, the errexit option, stops the shell when a command returns non-zero, but not when the command's result is being used as a condition. The condition of if, while and until, every command in an && or || list except the last one, and a command negated with ! are all exempt. The surprising part is that the exemption reaches into a function called in a condition. Because check_artifact was called in an if, the failure of the ls inside it was ignored too; the function returned the result of its last command, echo, which is 0. The other cases, each in its own bash -c:

$ bash -c 'set -e; false && echo "not printed"; echo "after the && list"'; echo "exit=$?"
after the && list
exit=0
$ bash -c 'set -e; f() { false; echo "f continues after false"; }; f || echo "f failed"'; echo "exit=$?"
f continues after false
exit=0
$ bash -c 'set -e; f() { local v=$(false); echo "v=[$v], still running"; }; f'; echo "exit=$?"
v=[], still running
exit=0
$ bash -c 'set -e; v=$(false; echo "inside, after false"); echo "v=[$v]"'; echo "exit=$?"
v=[inside, after false]
exit=0
$ bash -c 'set -e; count=0; (( count++ )); echo "count=$count"'; echo "exit=$?"
exit=1
$ bash -c 'set -e; count=0; count=$((count + 1)); echo "count=$count"'; echo "exit=$?"
count=1
exit=0
$ bash -c 'set -e; false | true; echo "pipeline without pipefail"'; echo "exit=$?"
pipeline without pipefail
exit=0

In order: when the first command of the && list fails, the list stops but the script goes on. Inside a function called as f || ..., set -e behaves as if it were switched off entirely. local v=$(false) is the masking we saw in the fourth post: the line's exit code is that of local, which is 0. By default the inside of a command substitution does not inherit set -e; the echo after false ran as well. (( count++ )) surprises from the other side: post-increment returns the old value, an expression whose value is 0 makes (( )) return 1, and the script dies on the first increment. Write count=$((count + 1)) for counters. The last line is from the previous post: without pipefail the failure on the left of a pipeline is invisible.

That is why set -e is a safety net, not a control mechanism. Check the result of an important step explicitly (tar ... || die "..."), and know that a value-returning function called inside a condition is not protected. Item 105 of the BashFAQ on Greg's Wiki keeps a long list of these exceptions, and some people recommend not relying on set -e at all. This series uses it, because carrying on after an unnoticed error is more expensive than stopping in an unexpected place, but it uses it knowing where the gaps are.

set -u, pipefail and inherit_errexit

set -u, the nounset option, treats the expansion of an undefined variable as an error. It catches typos and forgotten arguments:

$ bash -c 'set -u; echo "root=$BOOKSTORE_ROOT"'; echo "exit=$?"
bash: line 1: BOOKSTORE_ROOT: unbound variable
exit=127
$ bash -c 'set -u; echo "root=${BOOKSTORE_ROOT:-/opt/bookstore}"'; echo "exit=$?"
root=/opt/bookstore
exit=0
$ ./unset-arg.sh; echo "exit=$?"
./unset-arg.sh: line 4: $1: unbound variable
exit=1

Write ${var:-default} or ${1:-} for every optional value; the series' scripts have done so since the second post. We saw pipefail in the previous post. Since bash 4.4 there is an option that makes command substitutions inherit set -e:

$ bash -c 'set -e; shopt -s inherit_errexit; v=$(false; echo "inside, after false"); echo "v=[$v]"'; echo "exit=$?"
exit=1

All scripts in the series now start with the same two lines right below the header comment:

set -Eeuo pipefail
shopt -s inherit_errexit

-E, the errtrace option, makes the ERR trap apply inside functions as well; more on that in the next section. Adding these lines changes behaviour, so do not add them without testing. For us they caused a regression:

$ BOOKSTORE_ROOT=/tmp/empty ./scripts/bookstore-ctl.sh releases; echo "exit=$?"
exit=1

No message, just 1. The cause is current=$(readlink -f "$ROOT/current") in cmd_releases: when the root folder is missing, readlink -f fails, and the script that used to carry on with an empty value now stops silently. The fix is to handle the expected case explicitly:

    [[ -d $ROOT/releases ]] || die "no releases under $ROOT"
    current=$(readlink -f "$ROOT/current") || current=""

$ BOOKSTORE_ROOT=/tmp/empty ./scripts/bookstore-ctl.sh releases; echo "exit=$?"
2026-11-15T04:23:22Z ERROR: no releases under /tmp/empty
exit=1

Seeing Where It Failed with an ERR Trap

set -e stops the script but does not say where. The ERR trap runs for every failed command at which set -e would kick in. The thing to watch is that, by default, it does not apply inside functions:

#!/usr/bin/env bash
set -eu
trap 'echo "ERR trap: line $LINENO: $BASH_COMMAND (exit $?)" >&2' ERR
step() {
    false
}
step
echo "not reached"

$ ./errtrace.sh; echo "exit=$?"
exit=1
$ ./errtrace-E.sh; echo "exit=$?"
ERR trap: line 5: false (exit 1)
exit=1

The first script ended silently with 1; the second, whose only difference is set -Eeu, printed the line and the command that failed. $LINENO and $BASH_COMMAND describe the failing command while the trap runs; that is why the trap string must be in single quotes, because with double quotes the trap takes the values from the moment it was defined. We added a handler to the library that logs this information together with the call stack:

# ERR trap handler: logs the failed command, its exit code and the call stack.
# Usage: trap 'on_error "$?" "$LINENO" "$BASH_COMMAND"' ERR   (with set -E)
on_error() {
    local status=$1 line=$2 command=$3 i
    log "ERROR: '$command' exited with $status"
    # FUNCNAME[0] is on_error itself; the last entry is the script's top level.
    for (( i = 1; i < ${#FUNCNAME[@]} - 1; i++ )); do
        log "  at ${FUNCNAME[i]} (${BASH_SOURCE[i]##*/}:$line)"
        line=${BASH_LINENO[i]}
    done
}

FUNCNAME, BASH_SOURCE and BASH_LINENO are three arrays read with the same index: the names of the functions currently running, the files they were defined in, and the line each one was called from. The deploy script's main sets the traps first. Let us try to deploy a version whose artifact was cut short and is therefore corrupted:

$ PATH="$HOME/lab-stubs:$PATH" ./scripts/bookstore-deploy.sh -e staging -v 1.4.5; echo "exit=$?"
2026-11-15T04:22:52Z deploying 1.4.5 to staging

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
2026-11-15T04:22:52Z ERROR: '"$@"' exited with 2
2026-11-15T04:22:52Z   at run (common.sh:40)
2026-11-15T04:22:52Z   at deploy (bookstore-deploy.sh:159)
2026-11-15T04:22:52Z   at main (bookstore-deploy.sh:190)
2026-11-15T04:22:52Z removing unfinished release folder /opt/bookstore/releases/.incoming-1.4.5.s9M4M0
exit=2

$BASH_COMMAND gives the command's text from the source code; the command is the "$@" inside run, so that is what the message shows. The stack tells you which run call it was: line 159 of the deploy script, tar. The exit code is tar's too. The cleanup on the last line comes from the EXIT trap. The lab container has no systemd, so two small stubs again stand in for sudo and systemctl, and the health endpoint is a small test service that gives the same answer as Bookstore.Api.

Cleanup with an EXIT Trap

The EXIT trap runs however the script ends: when it reaches the last line, with exit, because of set -e, or through an exit called by a signal trap. It is the right place to delete temporary files and folders:

#!/usr/bin/env bash
tmp=$(mktemp -d)
cleanup() {
    rm -rf -- "$tmp"
    [[ -n ${KEEP_LOG:-} ]] && echo "keeping log"
}
trap cleanup EXIT
echo "working in $tmp"
exit 3

$ ./exit-trap.sh; echo "exit=$?"
working in /tmp/tmp.KQI4EG0qA1
exit=3

Although the trap's last command ([[ ... ]] && echo) returned 1, the script ended with 3; bash keeps the script's exit code unless the trap calls exit itself. If you need the exit code inside the trap, capture it on the first line: local status=$?.

Signals need their own traps. When the trap for INT, which Ctrl-C sends, or for TERM, which kill and a stopping systemd service send, calls exit, the EXIT trap runs as well; 130 and 143 follow the convention of 128 plus the signal number:

main() {
    trap cleanup EXIT
    trap 'exit 130' INT
    trap 'exit 143' TERM
    trap 'on_error "$?" "$LINENO" "$BASH_COMMAND"' ERR
    ...

The deploy itself changed so that it does not leave half-done work behind. Before, the artifact was extracted straight into releases/1.4.6; an interrupted deploy left a half-filled release folder with the right name. Now it is extracted into a hidden folder first, and the folder gets its name only when it is complete:

# ShellCheck 0.11 does not see the call from the trap when main is followed by "; exit".
# shellcheck disable=SC2329
cleanup() {
    if [[ -n $INCOMING && -d $INCOMING ]]; then
        log "removing unfinished release folder $INCOMING"
        rm -rf -- "$INCOMING"
    fi
}

    # Extract into a hidden folder first; releases/VERSION appears only when it is complete.
    if (( DRY_RUN )); then
        INCOMING="$ROOT/releases/.incoming-$VERSION.XXXXXX"
    else
        INCOMING=$(mktemp -d "$ROOT/releases/.incoming-$VERSION.XXXXXX")
    fi
    run tar -xzf "$artifact" -C "$INCOMING"
    if (( DRY_RUN )); then
        printf '[dry-run] write %s\n' "$INCOMING/RELEASE"
    else
        release_info > "$INCOMING/RELEASE"
    fi
    run mv -T "$INCOMING" "$release_dir"
    INCOMING=""

Because its name starts with a dot, the releases/*/ glob does not see this folder, so the prune and report scripts leave it alone. ShellCheck 0.11.0 thinking that a function called only from a trap is unused when the script ends with the main "$@"; exit pattern, warning SC2329, is a false alarm I ran into in the lab; removing ; exit makes the warning go away, but we keep the pattern and silence it with a one-line directive.

To try Ctrl-C, I wrote a small script that starts the deploy in its own process group and sends TERM to the whole group half a second later; in a terminal, Ctrl-C also sends INT to every process in the foreground:

#!/usr/bin/env bash
# Starts a deploy in its own process group and, after half a second, sends TERM
# to the whole group, the way Ctrl-C (INT) in a terminal reaches every process.
setsid ./scripts/bookstore-deploy.sh "$@" &
pid=$!
sleep 0.5
kill -TERM -- -"$pid"
wait "$pid"
echo "deploy exit=$?"

The artifact of 1.4.6 is 400 MB; extracting it takes about two seconds, so the signal lands right in the middle. The first attempt found a bug:

$ PATH="$HOME/lab-stubs:$PATH" ./interrupt-deploy.sh -e staging -v 1.4.6
2026-11-15T04:22:55Z deploying 1.4.6 to staging
deploy exit=141
$ ls -A /opt/bookstore/releases
.incoming-1.4.6.pRF7Eh
1.4.0
1.4.1
1.4.2
1.4.3
1.4.4

The expected exit code was 143; we got 141, which is SIGPIPE, and the hidden folder was not deleted. TERM went to the whole process group, and the tee started by start_log in the previous post died too. Then the log in cleanup tried to write to a pipe nobody was reading any more, and the script died of SIGPIPE before it could remove the folder. The fix is to make the tee that carries the log ignore INT and TERM; an ignored signal stays ignored after exec:

    # tee ignores INT and TERM: a Ctrl-C to the whole process group must not stop
    # the log while the script's traps are still writing to it.
    exec 2> >(trap '' INT TERM; exec tee -a "$file" >&2)

$ PATH="$HOME/lab-stubs:$PATH" ./interrupt-deploy.sh -e staging -v 1.4.6
2026-11-15T04:22:56Z deploying 1.4.6 to staging
2026-11-15T04:22:56Z removing leftover /opt/bookstore/releases/.incoming-1.4.6.pRF7Eh
Terminated
2026-11-15T04:22:57Z removing unfinished release folder /opt/bookstore/releases/.incoming-1.4.6.0ziJko
deploy exit=143
$ ls -A /opt/bookstore/releases
1.4.0
1.4.1
1.4.2
1.4.3
1.4.4

Terminated is the line bash prints for the tar that was killed by TERM. This time the exit code is 143, the half-extracted folder is gone and current was never touched. We will come to the "leftover" cleanup on the second line below. One more detail: had you sent TERM only to bash's PID, the Bash manual says the trap waits until the foreground command (here tar) finishes; cleanup is delayed but still runs.

Retries and timeout

Right after a service restart the health endpoint may return 503 for a few seconds. Instead of failing the deploy on a single failed health check, it is better to try a few times, doubling the wait each time:

# Runs a command until it succeeds, at most ATTEMPTS times; the delay doubles after each failure.
# Usage: retry ATTEMPTS DELAY_SECONDS COMMAND [ARGS...]; returns the last exit code.
retry() {
    local attempts=$1 delay=$2 attempt=1 status
    shift 2
    while true; do
        "$@" && return 0
        status=$?
        if (( attempt >= attempts )); then
            log "giving up after $attempt attempts (exit $status): $*"
            return "$status"
        fi
        log "attempt $attempt failed (exit $status), retrying in ${delay}s: $*"
        sleep "$delay"
        attempt=$((attempt + 1))
        delay=$((delay * 2))
    done
}

The "$@" && return 0 line is an && list on purpose: the command's failure does not trigger set -e, and it can be read with $? right afterwards. The deploy now calls the health check with run retry 5 1 "$SCRIPT_DIR/bookstore-health.sh" local. Let us make the test service return 503 for two and a half seconds and deploy:

$ touch /tmp/bookstore-unhealthy; setsid -f sh -c 'sleep 2.5; rm -f /tmp/bookstore-unhealthy'
$ PATH="$HOME/lab-stubs:$PATH" ./scripts/bookstore-deploy.sh -e staging -v 1.4.4; echo "exit=$?"
2026-11-15T04:22:52Z deploying 1.4.4 to staging
(lab stub) systemctl restart bookstore-api
curl: (22) The requested URL returned error: 503
FAIL http://127.0.0.1:5000/health
2026-11-15T04:22:52Z attempt 1 failed (exit 1), retrying in 1s: /home/user/bookstore-ops/scripts/bookstore-health.sh local
curl: (22) The requested URL returned error: 503
FAIL http://127.0.0.1:5000/health
2026-11-15T04:22:53Z attempt 2 failed (exit 1), retrying in 2s: /home/user/bookstore-ops/scripts/bookstore-health.sh local
OK http://127.0.0.1:5000/health
2026-11-15T04:22:55Z deployed 1.4.4
exit=0

Retrying only makes sense for transient failures: the network, a service starting up, a short-lived lock. A wrong argument or a missing file gives the same result in all five attempts and only makes the deploy longer. With five attempts and a one-second start, the worst case waits 1 + 2 + 4 + 8 = 15 seconds; choose that upper bound deliberately.

The companion of retry is timeout: it stops a command that does not finish within a given time and returns 124. To put an upper bound on each attempt you can combine the two: retry 5 1 timeout 10 command.

$ timeout 2 sleep 5; echo "exit=$?"
exit=124

One Deploy at a Time with flock

If two people start a deploy at the same moment, or a cron job runs again before the previous run has finished, two scripts write to the same folders and the same symlink. flock, from util-linux, takes an advisory lock that the kernel keeps on a file, which means it only stops processes that check the lock; the lock is released by itself when the file descriptor closes, so however the script ends:

# Takes an exclusive lock that is held until the script exits.
# Fails at once if another process holds it. Usage: acquire_lock FILE
acquire_lock() {
    exec 9> "$1"
    flock -n 9 || die "another run holds the lock $1"
}

exec 9> keeps the file open on file descriptor 9, and flock -n 9 tries to take the lock without waiting. The deploy script takes the lock right after it opens the log, except in a dry run. Let us start a second deploy while the first one is still running in the background:

$ PATH="$HOME/lab-stubs:$PATH" setsid -f ./scripts/bookstore-deploy.sh -e staging -v 1.4.6 > /tmp/first-deploy.log 2>&1; sleep 0.5
$ PATH="$HOME/lab-stubs:$PATH" ./scripts/bookstore-deploy.sh -e staging -v 1.4.6; echo "exit=$?"
2026-11-15T04:22:57Z ERROR: another run holds the lock /opt/bookstore/.deploy.lock
exit=1
$ sleep 4; cat /tmp/first-deploy.log
2026-11-15T04:22:57Z deploying 1.4.6 to staging
(lab stub) systemctl restart bookstore-api
OK http://127.0.0.1:5000/health
2026-11-15T04:23:00Z deployed 1.4.6

The content of the lock file does not matter, and it never needs to be deleted; that is its advantage over the "write a PID file, exit if it exists" pattern. With that pattern, if the script dies from kill -9 the file stays behind and every following run is refused forever. For backups running from cron you can use the same lock from the command line: flock -n /run/lock/bookstore-backup.lock ./backup.sh. For work that should wait, -w 60 instead of -n waits for at most 60 seconds.

Idempotent Design: Running It Twice Must Do No Harm

An idempotent script, run a second time with the same arguments, either reaches the same result or says there is nothing to do. For a deploy that means deploying the same version again is not an error.

    # Running the same deploy twice is not an error: the second run has nothing to do.
    if [[ $current == "$VERSION" ]]; then
        log "release $VERSION is already current, nothing to do"
        return 0
    fi

$ PATH="$HOME/lab-stubs:$PATH" ./scripts/bookstore-deploy.sh -e staging -v 1.4.4; echo "exit=$?"
2026-11-15T04:22:55Z release 1.4.4 is already current, nothing to do
exit=0

This makes the deploy step of a pipeline that is re-run in CI safe. The same idea applies to small steps: mkdir -p does not fail if the folder exists, ln -sfn points the symlink at the same place every time, rm -f is quiet about a missing file. The one case in which traps cannot run at all is SIGKILL: kill -9 or the OOM killer; that is why the deploy also removes .incoming-* folders left over from earlier runs every time it starts. The "removing leftover" line in the TERM experiment above comes from this loop:

    # Leftovers of a deploy that was killed with SIGKILL, where no trap can run.
    local stale
    for stale in "$ROOT"/releases/.incoming-*; do
        [[ -d $stale ]] || continue
        log "removing leftover $stale"
        run rm -rf -- "$stale"
    done

The lock matters here too: this loop could delete an .incoming folder that another deploy is using at that moment; it is safe because flock guarantees that only one deploy runs at a time.

In Production

When the health check fails, the deploy currently ends with die, but the current symlink keeps pointing at the new release. The script reported an error, while the server stayed on an unhealthy version. The fix is to return to the previous release automatically; that is the topic of the seventh post.

A trap is set once, and a trap set later for the same signal replaces the earlier one. If a library function sets its own EXIT trap, the script's cleanup silently disappears. Set traps only in the script's main, never in library functions. In the same way, a library should not change set -e or shopt settings; those are the calling script's decision.

Exit codes are part of the interface. The deploy can now return 0 (success), 1 (deploy failed), 2 (usage error), 130/143 (interrupted) and the code of the command that set -e stopped at (2 in the tar example above). That last case can report a usage error and a corrupted artifact with the same code; if CI needs to tell them apart, fix the code with exit 1 in the ERR trap.

In the lab we used /opt/bookstore/.deploy.lock. On a real server, do not put the lock file in a place everyone can write to (/tmp); another user could open a file with the same name and hold the lock. Use a folder under /run/lock that only the deploy user can write to.

Common Pitfalls

Silencing errors with || true. When a line stops because of set -e, the easiest way out is to append || true; but that says the command can never fail:

#!/usr/bin/env bash
set -euo pipefail
rm /opt/bookstore/releases/1.3.0/cache.db 2>/dev/null || true
cp /var/tmp/bookstore/missing.tar.gz /opt/bookstore/releases/ || true
echo "deploy finished"

$ ./or-true.sh; echo "exit=$?"
cp: cannot stat '/var/tmp/bookstore/missing.tar.gz': No such file or directory
deploy finished
exit=0

The first line is acceptable: a missing cache file really does not matter, although rm -f says so more clearly. The second is the deploy's main step, and its failure was lost. Wherever you write || true, leave a comment that answers the question "why is it fine for this command to fail?".

Reading the exit code too late in a trap. When the trap starts, $? holds the script's exit code, but it changes after the first command:

#!/usr/bin/env bash
cleanup() {
    echo "cleaning up"
    exit $?
}
trap cleanup EXIT
false

$ ./trap-late-status.sh; echo "exit=$?"
cleaning up
exit=0
$ ./trap-exit-code.sh; echo "exit=$?"
cleanup sees status 1
exit=1

The failed script ended with 0, and CI counted it as a success. The second script reads local status=$? on cleanup's first line and ends with exit "$status". Safest of all is not to call exit in cleanup at all; as we saw above, bash keeps the exit code by itself.

Forgetting exit in an INT trap. A trap replaces the signal's default behaviour; without exit in the trap, Ctrl-C does not stop the script:

$ ./trap-int-no-exit.sh; echo "exit=$?"
cleaning up
still running after INT
exit=0
$ ./trap-int-exit.sh; echo "exit=$?"
cleaning up
exit=130

The first script caught INT, printed "cleaning up" and carried on. Put cleanup in the EXIT trap, and in signal traps only call exit with the right code.

Counting with (( count++ )). As we saw above, if the counter starts at 0, the first increment ends the script under set -e. The same happens when a condition such as (( errors )) is written on its own outside an if; assign arithmetic with $(( )) and use (( )) only in if and while.

A Small Experiment

★☆☆ What happens when bookstore-prune-releases.sh runs under set -Eeuo pipefail and KEEP equals the number of releases, or the root folder does not exist? Make sure that every case gives a message and a meaningful exit code.

★★☆ Add an upper limit to retry: RETRY_MAX_DELAY (default 8) keeps the delay from growing beyond that value. While the health check returns 503, measure the total time of six attempts with time and compare it with your calculation.

★★★ Handle a deploy that fails after the health check: if the health check fails, point current back at the release from before the deploy, restart the service and exit with 1. What should happen if something fails during the rollback too? Think about how the script should write that to the log.

The third exercise is the starting point of the next post. In the seventh post we bring the pieces of the series together in a real script: a deploy that downloads the artifact and verifies its checksum, runs the migration bundle, switches current atomically and returns to the previous release if the health check fails, plus a scheduled, rotating database backup script for SQL Server.