shellstreak

set -euo pipefail: bash scripts that fail safely

Updated · Every command on this page ran in our practice shell

Short answer

Start scripts that change things with set -euo pipefail: -e stops at the first failing command, -u makes unset variables an error, and pipefail makes a pipeline fail if any part fails. Add ${VAR:?} for paths you delete (rm -rf -- "${BUILD_DIR:?}"/*), pass arguments on with "$@", and make temp files with mktemp.

Out of the box, bash keeps going after a command fails, treats a typo'd variable as an empty string, and ignores failures in the middle of a pipeline. That's friendly at the prompt and dangerous in a script that deletes or deploys. A few lines at the top change it. The examples below ran in a practice shell on a bakery's clean-up script.

The bug these settings prevent

This script empties a build folder. If BUILD_DIR was never set, the rm line becomes rm -rf /*.

example
learner@shellstreak:~/bakery$ cat scripts/clean.sh && echo "rm -rf $BUILD_DIR/*"
#!/bin/bash
# Empty the build folder before a new build
rm -rf $BUILD_DIR/*
echo "Cleaned $BUILD_DIR"
rm -rf /*

set -u: unset variables are errors

With set -u (nounset), using a variable that was never set stops the script with unbound variable instead of quietly expanding to nothing.

example
learner@shellstreak:~/bakery$ bash -c 'set -u; echo start; echo "rm -rf $BUILD_DIR/*"; echo done' || echo "stopped early"
start
stopped early
bash: BUILD_DIR: unbound variable

${VAR:?}: stop if empty

set -u doesn't catch a variable that is set but empty. ${BUILD_DIR:?} stops in both cases, and ${BUILD_DIR:?set BUILD_DIR first} prints your message. Together with --, which ends rm's options, the safe line is rm -rf -- "${BUILD_DIR:?}"/*.

example
learner@shellstreak:~/bakery$ BUILD_DIR=build bash -c 'rm -rf -- "${BUILD_DIR:?}"/* && ls build && echo emptied'
emptied

set -e: stop on the first error

With set -e (errexit) the script exits as soon as a command fails, instead of carrying on with half its work done. It has exceptions: failures inside if tests and on the left of && or || don't count, because those are expected to fail sometimes.

set -o pipefail: failures inside pipelines count

A pipeline's exit code is normally the last command's, so false | true succeeds. With pipefail it fails if any part fails, so a broken grep or download in the middle of a pipeline is noticed (and, with set -e, stops the script).

example
learner@shellstreak:~/bakery$ false | true; echo "without: $?"; set -o pipefail; false | true; echo "with: $?"
without: 0
with: 1

Pass arguments on with "$@"

"$@" expands to every argument the script got, each as its own word, spaces intact. Unquoted $@ splits march 01.txt in two, and "$*" glues everything into one string. Put -- before them when they're file names, so a file called -r can't become a flag: cp -- "$@" backups/.

example
learner@shellstreak:~/bakery$ set -- "march 01.txt" notes.txt; for a in "$@"; do echo "[$a]"; done; for a in $@; do echo "<$a>"; done
[march 01.txt]
[notes.txt]
<march>
<01.txt>
<notes.txt>

Temp files: mktemp and trap

tmp=$(mktemp) creates a new file with a unique name and private permissions (600). trap 'rm -f "$tmp"' EXIT deletes it when the script ends for any reason, errors included. (The practice shell doesn't support trap.) Never write sort file > file: the shell empties the file before sort reads it. Sort into the temp file, then move it into place.

example
learner@shellstreak:~/bakery$ tmp=$(mktemp) && sort data/names.txt > "$tmp" && chmod 644 "$tmp" && mv "$tmp" data/names.txt && cat data/names.txt
Ana
Ben
Chloe
Dev

Check yourself

One real exercise from the “Shell scripting” unit. No account needed.

Use bash -c to run echo hi in a fresh bash.

What's checked: the output of your last command.

Common mistakes

  • Expecting set -e to catch a failure inside an if test or on the left of && or ||. Those are exempt, so check them yourself.

  • Relying on set -u alone for rm paths. An empty variable passes; use ${VAR:?} too.

  • Forgetting that a file made by mktemp is mode 600. Moving it over a shared file makes that file private; chmod it first.

  • Writing $@ or $* without quotes when passing arguments on.

Questions

Should every bash script start with set -euo pipefail?+

It's a good default for scripts that change things. Know the exceptions of `-e` (commands in `if`, `&&` and `||` don't trigger it), and run shellcheck, which catches most quoting mistakes.

What's the difference between $@ and $*?+

Quoted, `"$@"` gives each argument as a separate word and `"$*"` joins them into one string. Unquoted, both split on spaces. Use `"$@"` to pass arguments on.

Does set -e work in sh as well as bash?+

`set -e` and `set -u` are POSIX and work in `sh`. `set -o pipefail` is newer: bash, zsh and ksh have long had it and it's now in POSIX, but some older `sh` implementations reject it, so use `#!/bin/bash` when you rely on it.

Reading is step one. Now make it stick.

Practise this in a five-minute lesson with instant feedback. No account needed.

Practise it now