Mert Özen Further With Every Line
DevOps Bash Scripting — #05

Data Flow in Scripts: Pipes, Redirection, Here-docs, Process Substitution and xargs

Mert Özen Nov 15, 2026 18 min 17 views
Data Flow in Scripts: Pipes, Redirection, Here-docs, Process Substitution and xargs

In a script, stdout should carry only data and stderr the messages; a pipeline's exit code is its last command's by default and becomes visible with PIPESTATUS and pipefail. We add a log file to Bookstore's deploy script with exec and tee and a RELEASE file with a here-doc, recover the variable a pipe-fed loop loses with < <(...), run work in parallel with xargs -0 -P and produce a JSON report with jq.

Version note: The examples in this post were run on Ubuntu 24.04 LTS with Bash 5.2.21 and Bash 5.3.20 built from source, using ShellCheck 0.11.0 and jq 1.7.1.

Guess First

The server of Bookstore, the online bookstore used as the shared example across this site's series, has three releases, and since the third post bookstore-ctl.sh releases prints each of them on its own line. This script counts them:

#!/usr/bin/env bash
count=0
./scripts/bookstore-ctl.sh releases | while IFS= read -r line; do
    count=$((count + 1))
done
echo "releases: $count"

Does the script print releases: 3 or releases: 0? There is no error message and the exit code is 0. The answer is in the section "A Loop Fed by a Pipe".

In the previous post we gave the deploy script a clean interface. As a script grows, three separate flows pass through it: data that another program will read, messages that a person will read, and exit codes. In this post we separate the three: keeping stdout for data, writing messages to stderr and to a log file, catching the exit codes that pipelines lose with PIPESTATUS and pipefail, generating files with here-docs, running work in parallel with xargs, and reading and producing JSON with jq. The interactive basics of pipes and redirection are in the log reading post of the Linux and the Shell series (Reading Logs 2); here we look at the behaviour that matters inside scripts.

stdout Is for Data, stderr Is for Messages

Every process has two outputs: stdout (file descriptor 1) and stderr (file descriptor 2). $(...), a pipe and > file take only stdout; stderr keeps going to the terminal. So the rule is simple: everything another command will process goes to stdout, everything meant for a person goes to stderr. The library's log has done this from the start. To see what happens when the rule is broken, take a function that extracts an artifact and returns the folder's path:

#!/usr/bin/env bash
# Extracts an artifact into a new folder and prints the folder.
extract() {
    local dir
    dir=$(mktemp -d)
    tar -xzvf "$1" -C "$dir"
    echo "$dir"
}
release_dir=$(extract /var/tmp/bookstore/bookstore-api-1.4.3-linux-x64.tar.gz)
echo "release_dir=[$release_dir]"
ls "$release_dir"
$ ./extract.sh
release_dir=[./
./VERSION
./Bookstore.Api.dll
/tmp/tmp.SDz8Px7QQW]
ls: cannot access './'$'\n''./VERSION'$'\n''./Bookstore.Api.dll'$'\n''/tmp/tmp.SDz8Px7QQW': No such file or directory

tar -v writes the list of extracted files to stdout; $(...) captured that list too and release_dir became a four-line string. Every command ran without an error; what broke is the value the caller received. The fix is to send output that is not data to stderr:

    tar -xzvf "$1" -C "$dir" >&2
$ ./extract-fixed.sh
./
./VERSION
./Bookstore.Api.dll
release_dir=[/tmp/tmp.qU9nCRzWGP]
Bookstore.Api.dll
VERSION

The file list still shows up in the terminal, but it is no longer part of the value. The same problem appears with any chatty command called inside a function: the progress bar of curl, the messages of git, the output of dotnet build. Even when you do not want to see a command's output, do not reach for 2>/dev/null right away; throwing away the error message deletes the line you will most want when something goes wrong.

Order matters too, because redirections are applied from left to right. 2>&1 > file first connects stderr to the current stdout (the terminal) and then points stdout at the file, so the error message does not go to the file:

$ ls /opt/bookstore /missing 2>&1 > /tmp/out.txt; echo "--- file:"; cat /tmp/out.txt
ls: cannot access '/missing': No such file or directory
--- file:
/opt/bookstore:
current
releases
$ ls /opt/bookstore /missing > /tmp/out.txt 2>&1; echo "--- file:"; cat /tmp/out.txt
--- file:
ls: cannot access '/missing': No such file or directory
/opt/bookstore:
current
releases

Writing the Script's Output to a Log File Too: exec and tee

If you give exec only a redirection and no command, the redirection applies to the rest of the script from that line on. >(...) is process substitution: bash starts the command inside the parentheses and puts a file name connected to it in its place (echo >(true) prints /dev/fd/63 in the lab). Together they make writing all output of a cron job to both the terminal and a log file a one-liner:

#!/usr/bin/env bash
# From here on, stdout and stderr go to the terminal and to the log file.
exec > >(tee -a /tmp/nightly.log) 2>&1
echo "nightly job started"
ls /var/backups/bookstore
echo "nightly job finished"
$ ./nightly.sh
nightly job started
ls: cannot access '/var/backups/bookstore': No such file or directory
nightly job finished
$ cat /tmp/nightly.log
nightly job started
ls: cannot access '/var/backups/bookstore': No such file or directory
nightly job finished

In the deploy script we want something slightly different: only messages should go to the log, and stdout should stay clean for data. We added a start_log function to the library:

# Copies everything written to stderr from now on into FILE as well.
# stdout is left alone, so it stays clean for data. Usage: start_log FILE
start_log() {
    local file=$1
    mkdir -p "$(dirname "$file")" || return 1
    exec 2> >(tee -a "$file" >&2)
}

Although exec runs inside a function, its effect applies to the whole script, because a function does not start a subshell. tee -a appends its stdin to the file and also writes it to its own stdout; the trailing >&2 connects that stdout to the old stderr, which is the terminal. The deploy script opens the log only for a real run; a dry run changes nothing, so it leaves no trace in the log:

DEPLOY_LOG="${DEPLOY_LOG:-/var/log/bookstore/deploy.log}"

main() {
    parse_args "$@"
    validate_args
    if (( ! DRY_RUN )); then
        start_log "$DEPLOY_LOG"
    fi
    deploy
}
$ PATH="$HOME/lab-stubs:$PATH" ./scripts/bookstore-deploy.sh -e staging -v 1.4.3
2026-11-15T04:10:28Z deploying 1.4.3 to staging
(lab stub) systemctl restart bookstore-api
OK http://127.0.0.1:5000/health
2026-11-15T04:10:28Z deployed 1.4.3
$ PATH="$HOME/lab-stubs:$PATH" ./scripts/bookstore-deploy.sh -e staging -v 1.4.3; echo "exit=$?"
2026-11-15T04:10:28Z ERROR: release 1.4.3 already exists: /opt/bookstore/releases/1.4.3
exit=1
$ cat /var/log/bookstore/deploy.log
2026-11-15T04:10:28Z deploying 1.4.3 to staging
2026-11-15T04:10:28Z deployed 1.4.3
2026-11-15T04:10:28Z ERROR: release 1.4.3 already exists: /opt/bookstore/releases/1.4.3

The lab container has no systemd, so, as in the previous post, two small stubs that print what they do stand in for sudo and systemctl; the health endpoint on 127.0.0.1:5000 is a small test service that gives the same answer as Bookstore.Api. The stub's line and OK http://... go to stdout, so they are not in the log; the log collects the messages of log and die and the error output of commands.

Process substitution has a cost: according to the Bash manual it runs asynchronously. Bash carries on without waiting for tee to finish, and tee may still be writing when the script ends. A tee that starts writing one second late makes this easy to see:

#!/usr/bin/env bash
# The process substitution starts writing only after one second.
exec > >(sleep 1; tee -a /tmp/slow.log)
echo "line written by the script"
$ ./slow-log.sh; cat /tmp/slow.log; echo "--- after 2 seconds:"; sleep 2; cat /tmp/slow.log
cat: /tmp/slow.log: No such file or directory
--- after 2 seconds:
line written by the script
line written by the script

When the script finished, the log file did not exist yet; the line arrived in the terminal and in the file a second later. A step that reads the log file right after a cron job can miss the last lines. When it matters you can close stdout and wait for tee: $! holds the PID of the most recent process substitution, and wait can wait for it:

#!/usr/bin/env bash
exec > >(sleep 1; tee -a /tmp/slow-wait.log)
tee_pid=$!
echo "line written by the script"
# Close stdout so tee sees end of input, then wait for it.
exec >&-
wait "$tee_pid"
$ ./slow-log-wait.sh; cat /tmp/slow-wait.log
line written by the script
line written by the script

A Pipeline's Exit Code: PIPESTATUS and pipefail

By default the exit code of a pipeline is the exit code of its last command. If the command on the left fails but the one on the right succeeds, the pipeline counts as a success:

$ curl -fsS http://127.0.0.1:5000/version | jq -r .status; echo "exit=$? PIPESTATUS=${PIPESTATUS[*]}"
curl: (22) The requested URL returned error: 404
exit=0 PIPESTATUS=22 0
$ set -o pipefail
$ curl -fsS http://127.0.0.1:5000/version | jq -r .status; echo "exit=$? PIPESTATUS=${PIPESTATUS[*]}"
curl: (22) The requested URL returned error: 404
exit=22 PIPESTATUS=22 0

The test service has no /version endpoint. curl -f reported the 404 with 22, jq handled the empty input without complaint and returned 0, and the pipeline's result was 0. The PIPESTATUS array holds the exit code of every command in the pipeline in order; the next command overwrites it, so read it immediately. set -o pipefail changes the pipeline's result to the code of the rightmost command that returned non-zero; if all of them succeed, the result is 0.

pipefail has a trap of its own. A command that reads as much as it needs and exits early (head, grep -q) closes the pipe; when the command on the left keeps writing, it is killed by the SIGPIPE signal and returns 141 (128 + 13):

#!/usr/bin/env bash
set -o pipefail
first=$(seq 1 100000 | head -n 1)
echo "first=$first exit=$?"
$ ./pipefail-head.sh
first=1 exit=141

We got the value we wanted, but with pipefail on the pipeline counts as failed; in the sixth post, once we add set -e, this line will stop the script. If the left command's output is small it fits in the pipe's buffer and the problem stays hidden; it shows up when the data grows. The fix is to take the early-exiting command off the end of the pipe (read the file directly, as in head -n 1 file, or capture the data in a variable first), or to accept 141 deliberately on that line.

A Loop Fed by a Pipe: Subshells, lastpipe and < <(...)

The answer to Guess First:

$ ./count.sh
releases: 0

Bash runs every command of a pipeline in its own subshell, the while loop included. The loop incremented count three times, but in its own copy; when the subshell ended the copy went with it, and the script's count stayed at 0. Bash says nothing about this, but ShellCheck does:

$ shellcheck -f gcc count.sh
count.sh:3:58: warning: line appears unused. Verify use (or export if used externally). [SC2034]
count.sh:4:5: note: Modification of count is local (to subshell caused by pipeline). [SC2030]
count.sh:6:17: note: count was modified in a subshell. That change might be lost. [SC2031]

There are two fixes. The first is shopt -s lastpipe: it runs the last command of a pipeline in the current shell. According to the Bash manual it only works while job control is off, which is the default in scripts. The second, and the one I prefer, is to feed the loop from a process substitution instead of a pipe: <(command) lets the loop read the command's output like a file, and the loop stays in the current shell:

#!/usr/bin/env bash
count=0
while IFS= read -r line; do
    count=$((count + 1))
done < <(./scripts/bookstore-ctl.sh releases)
echo "releases: $count"
$ ./count-lastpipe.sh
releases: 3
$ ./count-procsub.sh
releases: 3

The process substitution version works the same in every version, Bash 3.2 on macOS included, and in an interactive shell as well. The prune script has used this pattern since the third post. The | while inside bookstore-ctl.sh releases can stay as it is: that loop only prints, and no variable has to survive it.

Generating Files with Here-docs and Here-strings

A here-doc passes the text that follows the <<EOF line, up to the line that contains only the label, to the command's stdin. If the label is unquoted, the text is expanded as if it were inside double quotes: $variable, $(command) and $(( )) all work. The deploy script now leaves a RELEASE file in each release that records who deployed it, when, and to which environment:

# Prints the content of the RELEASE file that is written into each release.
release_info() {
    cat <<EOF_INFO
version=$VERSION
environment=$ENV_NAME
deployed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)
deployed_by=${SUDO_USER:-$(id -un)}
EOF_INFO
}
    run tar -xzf "$artifact" -C "$release_dir"
    if (( DRY_RUN )); then
        printf '[dry-run] write %s\n' "$release_dir/RELEASE"
    else
        release_info > "$release_dir/RELEASE"
    fi
$ cat /opt/bookstore/current/RELEASE
version=1.4.3
environment=staging
deployed_at=2026-11-15T04:10:28Z
deployed_by=user

release_info does not write the file itself; it prints the content to stdout, and writing it to a file is the caller's job. That way you can print the same function to the screen in a test or a dry run. When the script runs under sudo, SUDO_USER gives the user who started the command; otherwise id -un does.

Expansion becomes a trap when the generated file contains variables of another program rather than bash. nginx's $host is a good example:

#!/usr/bin/env bash
port=5000
cat <<EOF_CONF
location /api/ {
    proxy_pass http://127.0.0.1:$port/;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
EOF_CONF
$ ./nginx-snippet.sh
location /api/ {
    proxy_pass http://127.0.0.1:5000/;
    proxy_set_header Host ;
    proxy_set_header X-Forwarded-For ;
}
$ shellcheck nginx-snippet.sh

In nginx-snippet.sh line 6:
    proxy_set_header Host $host;
                          ^---^ SC2154 (warning): host is referenced but not assigned.


In nginx-snippet.sh line 7:
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                                     ^------------------------^ SC2154 (warning): proxy_add_x_forwarded_for is referenced but not assigned.

For more information:
  https://www.shellcheck.net/wiki/SC2154 -- host is referenced but not assign...

$host and $proxy_add_x_forwarded_for are undefined for bash; they expanded to empty strings and an invalid nginx config came out. If you want no bash expansion at all in the text, quote the label: <<'EOF_CONF'; then $port stays as it is too. If you want both, put a backslash in front of every $ that bash must leave alone:

    proxy_set_header Host \$host;
    proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
$ ./nginx-snippet-fixed.sh
location /api/ {
    proxy_pass http://127.0.0.1:5000/;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Choose a label that cannot be confused with the content (EOF_USAGE, EOF_CONF): a line in the text that consists of the label alone ends the here-doc right there. The <<- form strips leading tabs but not spaces; it does not help in code indented with 4 spaces like this series, so we write here-doc content from the start of the line.

A here-string (<<<) passes a single string as stdin and appends a newline to it. Together with read it is handy for splitting a version number into its parts:

$ IFS=. read -r major minor patch <<< "1.4.3"; echo "major=$major minor=$minor patch=$patch"
major=1 minor=4 patch=3

The IFS=. here applies only to that read command; the script's IFS does not change.

Running Commands in Bulk and in Parallel with xargs

xargs runs a command with the arguments it reads from stdin. In a script, nearly always write two options: -0 separates arguments with NUL bytes, so names containing spaces or newlines are not split, and -n 1 gives each call one argument. -P N runs N calls at the same time. Let us run the health check of Bookstore's three environments in parallel:

$ printf '%s\0' local staging production | xargs -0 -n 1 -P 3 ./scripts/bookstore-health.sh; echo "xargs exit=$?"
OK http://127.0.0.1:5000/health
curl: (6) Could not resolve host: staging.bookstore.test
curl: (6) Could not resolve host: bookstore.test
FAIL https://bookstore.test/health
FAIL https://staging.bookstore.test/health
xargs exit=123

In the lab only local is up; the other two names do not resolve. The order of the output is not the order of the environments but the order in which they finished; output of parallel jobs can interleave, so if order matters, write each job's output to its own file. Watch the exit code too: if one of the calls exits with a code from 1 to 125, xargs returns 123. The gain from parallelism is easy to measure:

$ time (printf '%s\0' 1 1 1 | xargs -0 -n 1 sleep)

real	0m3.009s
user	0m0.003s
sys	0m0.002s
$ time (printf '%s\0' 1 1 1 | xargs -0 -n 1 -P 3 sleep)

real	0m1.004s
user	0m0.002s
sys	0m0.002s

For work that waits on disk or network (compressing backups, copying files to several servers) -P shortens the run; for work that loads the same database, keep the limit low. find ... -print0 | xargs -0 is the shorter form of the third post's find -print0 | while IFS= read -r -d '' loop when running one command per file is all you need.

Reading and Producing JSON with jq

Reading JSON with grep or sed breaks the day the field order or the whitespace changes. Use jq for JSON in scripts; the tool itself is covered in the Terminal and Command Line series, and here we look at two options that matter in scripts. -e turns a false or null result into an exit code, so jq can be used as a condition:

$ curl -fsS http://127.0.0.1:5000/health | jq -e '.status == "Healthy"' > /dev/null && echo "healthy"
healthy

Do not build JSON by concatenating strings. If the value contains a quote, the structure of the generated JSON changes; the "version" below added another field to itself:

$ printf '{"version":"%s"}\n' '1.4.3" , "admin":"yes' | jq -c .
{"version":"1.4.3","admin":"yes"}
$ jq -nc --arg version '1.4.3" , "admin":"yes' '{version: $version}'
{"version":"1.4.3\" , \"admin\":\"yes"}

--arg hands the value to jq as a string, and jq writes it with the correct escaping. When the value comes from a user, a file or a CI variable, this difference is a security matter: it is the JSON version of the script injection problem from the security post of the GitHub Actions series.

A Summary Report: bookstore-report.sh

A small script that puts the pieces of this post together: it summarises the releases on the server and the number of successful and failed deploys in the deploy log, and with --json it gives the same data in a form another tool can read. It was also added to bookstore-ctl.sh as the report command.

#!/usr/bin/env bash
#
# bookstore-report.sh - summary of the releases on this server and of the deploy log.
#
# Usage:      bookstore-report.sh [--json]
# Variables:  BOOKSTORE_ROOT  installation root (default /opt/bookstore)
#             DEPLOY_LOG      deploy log (default /var/log/bookstore/deploy.log)
# Exit codes: 0 success, 1 error, 2 usage error

SCRIPT_DIR=$(dirname "$(readlink -f "$0")")
# shellcheck source=lib/common.sh
source "$SCRIPT_DIR/lib/common.sh"

ROOT="${BOOKSTORE_ROOT:-/opt/bookstore}"
DEPLOY_LOG="${DEPLOY_LOG:-/var/log/bookstore/deploy.log}"

usage() {
    echo "Usage: ${0##*/} [--json]" >&2
}

# Release names, oldest first.
list_releases() {
    local dir
    shopt -s nullglob
    for dir in "$ROOT"/releases/*/; do
        dir=${dir%/}
        echo "${dir##*/}"
    done | sort -V
}

# One tab-separated row per release: version, deployed_at, deployed_by, current (yes/no).
release_rows() {
    local current name key value deployed_at deployed_by is_current
    current=$(current_release "$ROOT") || current=""
    while IFS= read -r name; do
        deployed_at="-"
        deployed_by="-"
        if [[ -f $ROOT/releases/$name/RELEASE ]]; then
            while IFS='=' read -r key value; do
                case $key in
                    deployed_at) deployed_at=$value ;;
                    deployed_by) deployed_by=$value ;;
                esac
            done < "$ROOT/releases/$name/RELEASE"
        fi
        is_current=no
        if [[ $name == "$current" ]]; then
            is_current=yes
        fi
        printf '%s\t%s\t%s\t%s\n' "$name" "$deployed_at" "$deployed_by" "$is_current"
    done < <(list_releases)
}

# Counts successful and failed deploys in the deploy log.
deploy_counts() {
    local ok=0 failed=0 line
    if [[ ! -r $DEPLOY_LOG ]]; then
        log "no deploy log at $DEPLOY_LOG"
        return 0
    fi
    while IFS= read -r line; do
        case $line in
            *" deployed "*) ok=$((ok + 1)) ;;
            *" ERROR: "*) failed=$((failed + 1)) ;;
        esac
    done < "$DEPLOY_LOG"
    echo "deploys: $ok succeeded, $failed failed (from $DEPLOY_LOG)"
}

print_text() {
    local version deployed_at deployed_by is_current count=0
    printf '%-9s %-21s %-6s %s\n' RELEASE DEPLOYED_AT BY CURRENT
    while IFS=$'\t' read -r version deployed_at deployed_by is_current; do
        printf '%-9s %-21s %-6s %s\n' "$version" "$deployed_at" "$deployed_by" "$is_current"
        count=$((count + 1))
    done < <(release_rows)
    echo "$count releases"
    deploy_counts
}

print_json() {
    release_rows | jq -Rn '[inputs | split("\t") | {
        version: .[0],
        deployed_at: (if .[1] == "-" then null else .[1] end),
        deployed_by: (if .[2] == "-" then null else .[2] end),
        current: (.[3] == "yes")
    }]'
}

main() {
    case ${1:-} in
        "") print_text ;;
        --json) print_json ;;
        -h | --help) usage ;;
        *)
            usage
            exit 2
            ;;
    esac
}

main "$@"; exit

The data is produced once and presented in two forms: release_rows writes one tab-separated line per release, print_text prints those lines as a table and print_json prints them as JSON through jq. The loop in print_text is fed by < <(release_rows), so count is correct after the loop. Tab counts as whitespace for IFS, so two tabs in a row are read as a single separator and the empty field disappears; that is why missing values are written as - and turned into null in the JSON. jq's -R reads the input as raw text lines instead of JSON, and inputs together with -n collects all lines into a single array.

$ ./scripts/bookstore-ctl.sh report
RELEASE   DEPLOYED_AT           BY     CURRENT
1.4.0     -                     -      no
1.4.1     -                     -      no
1.4.2     -                     -      no
1.4.3     2026-11-15T04:10:28Z  user   yes
4 releases
deploys: 1 succeeded, 1 failed (from /var/log/bookstore/deploy.log)
$ ./scripts/bookstore-report.sh --json | jq -c '.[]'
{"version":"1.4.0","deployed_at":null,"deployed_by":null,"current":false}
{"version":"1.4.1","deployed_at":null,"deployed_by":null,"current":false}
{"version":"1.4.2","deployed_at":null,"deployed_by":null,"current":false}
{"version":"1.4.3","deployed_at":"2026-11-15T04:10:28Z","deployed_by":"user","current":true}
$ ./scripts/bookstore-report.sh --json | jq -r '.[] | select(.current) | .version'
1.4.3
$ DEPLOY_LOG=/tmp/none.log ./scripts/bookstore-report.sh > /tmp/report.txt; tail -n 2 /tmp/report.txt
2026-11-15T04:10:28Z no deploy log at /tmp/none.log
1.4.3     2026-11-15T04:10:28Z  user   yes
4 releases

In the last command the message stayed in the terminal and only the report went to the file. That separation is what makes it possible for another script or a monitoring job to read the report. 1.4.0 to 1.4.2 have no RELEASE file because they were deployed before this post.

In Production

The moment another script starts reading a script's stdout, that output becomes an interface. If you want to be free to change the text table for people, give machines a separate, stable format (--json), so that a CI step or a monitoring check does not depend on column alignment.

A log file does not shrink by itself. Write a logrotate rule for /var/log/bookstore/deploy.log; how to write one is in the journalctl and logrotate post of the Linux and the Shell series. If you run the script as a systemd service or timer, systemd already writes stdout and stderr to the journal; in that case exec with tee is usually unnecessary and you read the output with journalctl -u.

If you generate a file that contains a secret with a here-doc, set the permissions before the file exists: (umask 077; release_secret > file) creates the file with mode 600; writing first and running chmod afterwards leaves a moment in which the file is readable. Since Bash 5.1, small here-docs are passed through a pipe, while those larger than the pipe buffer are still written to a temporary file; in older versions every here-doc meant a temporary file. That is another reason to keep a secret in a file with the right permissions rather than in a here-doc.

When you raise parallelism with xargs -P, think about the other side: copying files to ten servers at once strains the network, and ten parallel queries against the same database strain the database. Put the limit in a variable (PARALLEL=${PARALLEL:-4}) and lower it when needed.

Common Pitfalls

pipefail with grep -q. Once grep -q finds a match it exits immediately, the command on the left receives SIGPIPE, and the condition gives the wrong answer:

#!/usr/bin/env bash
set -o pipefail
if seq 1 100000 | grep -q '^1$'; then
    echo "found"
else
    echo "not found, PIPESTATUS=${PIPESTATUS[*]}"
fi
$ ./grep-q.sh
not found, PIPESTATUS=141 0

The line was found and grep returned 0, yet the pipeline "failed" with 141. If the source is a file, read the file directly, as in grep -q '^1$' file; if it is a command, capture its output in a variable first.

xargs without -0. By default it splits on spaces and newlines, so a file name with spaces becomes three arguments:

$ find /tmp/notes -name '*.md' | xargs ls; echo "exit=$?"
ls: cannot access '/tmp/notes/release': No such file or directory
ls: cannot access 'notes': No such file or directory
ls: cannot access '1.4.3.md': No such file or directory
exit=123
$ find /tmp/notes -name '*.md' -print0 | xargs -0 ls
/tmp/notes/release notes 1.4.3.md

A variable lost in a pipe-fed loop and a $ expanded by accident in a here-doc do not produce errors, as we saw above; ShellCheck catches both: SC2031 for the variable that changed in a subshell and SC2154 for the variable that was never assigned. That is why ShellCheck should run on every commit, not only in the editor; in the last post of the series we wire it into CI.

Using a function's stdout for both data and messages. As in the extract example, the value gets corrupted, and the error usually appears much later, where the corrupted value is used. If you capture a function's stdout with $(...), know where every command it calls writes.

A Small Experiment

★☆☆ Add the disk space each release uses as size_kb to the output of bookstore-report.sh --json (du -sk). The value should go into the JSON as a number, not a string; jq's tonumber helps.

★★☆ Add status --all to bookstore-ctl.sh: it runs the health checks of the three environments in parallel with xargs -P 3 but always shows the output in the order local, staging, production. Hint: write each call's output to its own file in a folder created with mktemp -d, then read them in order.

★★★ Keep the deploy log in JSON Lines format as well (one JSON object per line): add a log_event function based on jq -nc --arg to the library and have it write the start, the end and the failure of a deploy as {"time":…,"event":…,"version":…}. Then change deploy_counts to read that file with jq -s instead of searching text.

While preparing this post I ran one attempt in which I had forgotten to add start_log to the library: the script printed start_log: command not found and carried on with the deploy as if nothing had happened. That is exactly the topic of the next post: what set -euo pipefail really catches and what it misses, cleanup with trap, retries, a single deploy at a time with flock, and making sure an interrupted deploy does not leave the server half-done.