Some notes on writing shell scripts

I use shell scripts for a number of things:

• Automating common tasks, and sets of commands
• Wrapping other tools, such as setting up environment
• Collecting information
• Creating “checklists” ¹

I follow the common touted advice of not using shell scripts for anything too complicated, logic heavy or where strict error handling matters. Sometimes shell scripts do need to be used in these areas, and I shudder thinking how about poor the error handling in shell scripts are. But for cases where you’re just glueing other tools together, and can handle the occasional edge case not working properly, they’re very useful.

My advice isn’t born from a long career writing shell scripts, and certainly isn’t going to be relevant everywhere. If you find a mistake I can be reached at eric AT ericproberts DOT dev.

What shell to use

Almost every script I write starts out with the following line,

#!/usr/bin/env zsh

Yes, in the shell wars I’m a zsh fan. But it’s not just that I prefer using zsh to other shells (frankly, I haven’t tried many), but using something specific allows me to take full advantage of my shell. If I were to start with #!/bin/sh I would be forced to write in a maximally portable way. For instance, I could no longer rely on [[ "$A" = "$B" ]], instead being forced to write the antiquated [ "x$A" = "x$B" ].

Worse, #!/bin/sh doesn’t usually mean what you think it does: that you’re running the original Bourne Shell that will behave the same everywhere. Often this is just bash or zsh in some emulation mode, meaning you are unlikely to get the same behaviour everywhere – hence you need to be careful with portability.

Differences between shells can be subtle, and choosing a single one makes life much easier. I’ll illustrate this by going through examples that you might not catch testing your script.

First, let’s try fixing some permissions. Guess the ... output (and how it differs between shells).

$ ls -al
total 8
drwxr-xr-x 2 eric eric 4096 May 30 15:22 .
d-wx--x--x 4 eric eric 4096 May 30 15:22 ..
-rw-r--r-- 1 eric eric    0 May 30 15:22 a
-rw-r--r-- 1 eric eric    0 May 30 15:22 b
--w------- 1 eric eric    0 May 30 15:22 .c
--w------- 1 eric eric    0 May 30 15:22 .d

$ cat a b .c .d
cat: .c: Permission denied
cat: .d: Permission denied

$ chmod +r -- .*

$ ls -al
...

On zsh this works as I would expect, with the last output being

total 8
drwxr-xr-x 2 eric eric 4096 May 30 15:22 .
d-wx--x--x 4 eric eric 4096 May 30 15:22 ..
-rw-r--r-- 1 eric eric    0 May 30 15:22 a
-rw-r--r-- 1 eric eric    0 May 30 15:22 b
-rw-r--r-- 1 eric eric    0 May 30 15:22 .c
-rw-r--r-- 1 eric eric    0 May 30 15:22 .d

Compare to bash,

total 8
drwxr-xr-x 2 eric eric 4096 May 30 15:22 .
drwxr-xr-x 4 eric eric 4096 May 30 15:22 ..
-rw-r--r-- 1 eric eric    0 May 30 15:22 a
-rw-r--r-- 1 eric eric    0 May 30 15:22 b
-rw-r--r-- 1 eric eric    0 May 30 15:22 .c
-rw-r--r-- 1 eric eric    0 May 30 15:22 .d

If you lack keen eyes look at the parent directory’s permissions. In bash the glob .* expands to . .. .c .d, but zsh leaves out the first two items. We probably meant the zsh behaviour, and I know this difference because it’s bitten me before. (From testing with other shells ksh, which zsh is based, follow’s zsh and sh, which on my Debian machine uses zsh’s emulation mode, follow’s bash’s behaviour).

To be clear, I’m not saying the bash behaviour is wrong, it’s actually more consistent and maybe less surprising than zsh (although I find the zsh behaviour much more useful). But if you’re trying to be portable you might not realize you have to account for this.

Another difference that I find annoying has to do with variable expansion. Like before, try to guess the output,

$ cat 1.sh
./2.sh $1 $2
$ cat 2.sh
echo 1 $1
echo 2 $2
$ $SHELL ./1.sh '/path to file' '/path to file'
...

If $SHELL is bash this will actually print 1 /path and 2 to, while zsh prints what you’d expect (ie, the two paths). I think in both these cases zsh wins out, however in either case with some testing you’d be able to suss out these differences. The point is they’re there, and if you insist on portability you’d need to account for them.

I’ll also throw in here that zsh has a bunch of powerful glob patterns that, as far as I know, don’t have easy equivalent in bash. For instance,

$ echo *(Lm+1)
...list files over 1MB

Error Handling

One of the hardest parts of programming, including shell scripting, is dealing with weird cases and error states. My approach to shell scripting: don’t try too hard. But there’s certainly some low effort things that can make a big difference.

(If you Google around there’s plenty of good boilerplate scripts. A lot of the stuff is overkill I think for the things I use shell scripting for, but it’s often good to know they exist).

The lowest effort error handling is to set some options that give more reasonable behaviour for scripts.

# These lines do the same thing, but both forms are common
set -o nounset -o errexit -o pipefail
set -euo pipefail

Now commands like false, false | true and echo $MISPELLING casue the script to exit.

But this isn’t foolproof. Take the script below,

#!/usr/bin/env zsh
set -euo pipefail
f() { false; echo 1 }
[[ 1 -eq "$(f)" ]] && echo True

The output? True

Generally errexit means false should fail but because of the environment it’s being called in we just move past the false entirely. One thing you can do to beef up command you don’t want to fail is to put || exit 1 after everything.

There’s also other situations that are more difficult to catch,

git ref-parse --git-dir || (
    echo "Not in git repo... creating"
    git init
)

In this case rev-parse is misspelled. A more careful scripter might catch this case.

git ref-parse --git-dir || [[ $? == 128 ]] && (
    echo "Not in git repo... creating"
    git init
)

You’d probably catch it if you misspelled a git subcommand, so handling this case seems extraneous. But the idea you might catch one error thinking it’s another is not uncommon, and consider that some git subcommands like git-switch are still not supported by many widely used git versions.

We catch this by exploiting that git returns a different error code for using the wrong command than not being in a git repository, but again I think that’s overkill.

Should you take tons of strict precautionary measures when writing shell scripts? No, unless you really have to. The number of corner cases and things that can go wrong is high. If that level of error handling becomes necessary, usually python’s an option.

What adds another layer of impossibility to shell error handling is you’re stuck in an environment you can’t control. If you’re writing a zsh script then you can assume the user has zsh. But do they have GNU awk? coreutils? gcc? Are they the right version? These are things you can check, but it’s probably not worth the hassle. You should generally assume there’s a reasonable environment, and handle the cases where you need something special they don’t have. In something like python this isn’t (as much of) a problem: you’re generally working within python where you’re generally not calling out to an external program every other line.

You also shouldn’t be controlling the user’s environment to fix this. If they have a non-standard awk that doesn’t follow the rules in their $PATH that’s on them (it’s up to you to make sure if you need GNU awk to use gawk).

Options Parsing

I like zparseopts, and I recommend you use it. It supports long options and requires less boilerplate than getopts. It’s zsh-specific and IMO the killer feature of writing shell scripts in zsh.

#!/usr/bin/env zsh
set -euo pipefail
zparseopts -D -E -F -a ARGV -help
[[ -n "${ARGV[0]}" ]] && cat <

#!/usr/bin/env zsh
set -euo pipefail

die() { echo "$1" >&2; exit 1 }

zparseopts -D -E -F i:=INPUT_FILE o:=OUTPUT_FILE -help=HELP
[[ -n "$HELP" ]] && echo help text && exit 0
INPUT_FILE="${INPUT_FILE:1}"
[[ -z "$INPUT_FILE" ]] && die 'Input file required'
[[ -z "${1:-}" ]] && die 'Positional required'
: "${OUTPUT_FILE:=/dev/stdout}"

echo "Input File: ${INPUT_FILE:1}"
echo "Output File: $OUTPUT_FILE"
echo "Positional Arg: $1"

$ ./command
Input file required
$ ./command -a -b -c
./command:zparseopts:6: bad option: a
$ ./command -ifirst -i override positional
Input File: override
Output File: /dev/stdout
Positional Arg: positional
$ ./command front_positional -i first
Input File: override
Output File: /dev/stdout
Positional Arg: front_positional
$ ./command --help
help text

html = httplib.get("https://example.com/stream.html")
stream_url = htmllib.find_selector("video", html).get_attr("src")
downloaded_file = m3u8lib.download_stream(stream_url, to_file=True)
sqlite3lib.run("INSERT INTO downloads(url,filename) VALUES (?,?);", stream_url, downloaded_filed)
print_line("DONE")

sqlite3 file.db "INSERT INTO(a,b) VALUES ('$A', '$B');"