Wow, it took me a long time to find these bugs, but I’ve been working on various bash scripts and I finally figure something out that is hard to know:
There are a bunch of of bash safety checks that you can turn on. Most see great, but there are real gotchas. Here are two difficult ones. I hope you never encounter them:
set -e and the && construct
- We use a restrictive set of checks here
set -ueo pipefailwhich means the scripts that the scripts stop is any command returns a non-zero exit code, or if anything in a pipeline fails or if you use an unset variable. -
I have been using a common construct to print out errors,
$VERBOSE && echo start edits
But this has a series of consequences.
- What is the return value of the
$VERBOSE && echo fooif VERBOSE if set to false. Well, it is not the same asif $VERBOSE; then echo foo; fias I thought. The former returns 1 while the later returns 0.
So you can’t use the pipe on success as a synonym for the if-then. In particular if you use set -e, the script will mysteriously fail in normal mode but run find in debug or verbose mode!
In fact any multiple pipe like ls -l && ls -l ../lib will fail
- There is another common thing
test || echo false && echo truebut be careful this is not equivalent to an if-then-else becaue if the second statement fails, then you will never get to the final thing.
The effect is more subtle actually, as long as you use just the || operator you won’t have a problem since it only tests for false, so ssh git@github.com || true is a simple way to run something if something fails. This is ok, but the other version ssh git@github.com && echo works" does not because you get an error code as the return value, you can tryssh git@github.com && echo works || false` but only if you can guarantee that the “echo works” will never fail which is of course pretty impossible :-0
set -o pipefail causes ssh to hang up and behavies differently in single step
The next one is even more insidious, one good suggestion is to turn on -o pipefail. This makes sure that in a long pipeline, you don’t mask errors, so for instance false | true will be ok with pipefail, but will fail with it. this helps you check long pipelines. There are two things that make it unusable at least for me on Ubuntu 14.04:
- If you grep for something and don’t find it, there is no way to have the thing not fail. You can’t
grep home /etc/bar | cut -f 1for instance, you have to cut it into pieces because of the grep fails if it doesn’t find things, so you have to have temporary variables everywherex=$(grep home /etc/bar) if echo $x | cut -f 1…which is really a mess.
Things get very strange with ssh and interactive traps. But if you have pipefail on and are in an interactive ssh session, it will close the ssh session if you encounter an intermediate pipeline failure. There isn’t any recoverly. Also if you use trap DEBUG to catch errors, then it will work fine because the pipefail is cause by the trap. So it is very hard to debug.
Net, net -o pipefail is optional and I’m glad it is. I’m taking it out of my stuff now.
Net, net the lesson is ignore the various google queries that are clever about && and || for pipes, use if-then-else. Sigh, I have a lot of code to fix!
