Tag: shell script

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why don't the options in "man time" work?
time -f %w myapp

Short answer: ‘time’ runs the builtin version, ‘man time’ shows the external version

time is a builtin in the shell, as well as an external command (this also goes for kill, pwd, and test). The man time shows info about the external command, while help time shows the internal one.

To run the external version, one can use command time or /usr/bin/time or just \time.

The reason why time is built in is so timing pipelines will work properly. time true | sleep 10 would say 0 seconds with an external command (which can’t know what it’s being piped into), and while the internal version can say 10 seconds since it knows about the whole pipeline.

POSIX leaves the behaviour of time a | b undefined.

# This finds the full path to ls. Why isn't there a 'man type'?
type -P ls

type is a bash builtin, not an external command. This allows it to take shell functions and aliases into account, something whereis can’t.

Builtins are documented in man bash, or more conveniently, “help type” (help is also a builtin).

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why can't you use variables in {}?
for i in {0..$n}; do ..

Short answer: {..} is expanded before $n

Shell execution is based on successive expansions. {..} is evaluated early, before variable expansion, and thus you can’t use variables in them.

This also implies that you can use {..} in variable names, like a1=foo; a2=bar; echo $a{1,2}.

Instead, use for ((i=0; i<n; i++)); do ....

# Why aren't any of the Linux rename tools like Windows 'ren'?
touch foo0001.jpg foo0002.jpg
ren foo*.jpg bar*.jpg        # Windows
rename foo bar foo*.jpg      # Coreutils rename
rename 's/foo/bar/' foo*.jpg # Perl (debian) rename

Short answer: globs are expanded before the command sees them

Bash expands globs before running the command. This means that running rename foo*.jpg bar*.jpg is exactly the same as running rename foo0000.jpg foo0001.jpg .... Since rename can’t know what pattern was originally used, it has to use an alternative syntax.

Of course, you could write a rename where you quote the globs, like rename "foo*.jpg" "bar*.jpg", but that’s not simpler than the coreutils version. It just adds complexity, edge cases and general confusion.

There have been proposals for environment variables to set so that commands can see the shell arguments with globs intact, but that has its own problems so they weren’t widely used.

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why does esc contain "x1B" instead of the escape char?
esc=`printf \\x1B`

Short answer: “ requires another level of backslash escaping

To embed a backtick inside “, you escape it with a backslash. To embed a backslash, you escape it with another backslash. So `printf \\x1B` actually runs printf \x1B, and the shell interprets \x as a literal x with a superfluous escape. In other words, printf just sees “x1B”, and that’s what you get.

The problem grows exponentially as you try to nest ` `.

$(..) has distinct start and stop characters, so they can be used and nested without adding layers of backslashes. In this particular case you’d use esc=$'\x1B', and in general you could use esc=$(printf \\x1B).

# Why is newline empty instead of containing the line feed character?
newline=$(printf "\n")
echo "hello${newline}world"

Short answer: “ and $(..) strips trailing line feeds

$(..) and “ always strip trailing line feeds from command output. This is that special kind of magic that works so well you never think about it. echo “Hello $(whoami), how are you?” comes out as one line even though “whoami” (and basically all other commands) writes the username followed by a line feed.

This causes problems here because the output is only a single \n, i.e. the empty string followed by a trailing line feed. In this case, you’d again use newline=$'\n', but you could also have done newline=$(printf '\n'; printf x); newline=${newline#x} (append a x and then remove it), so that the line feeds in the output aren’t trailing.

# Bonus: Exercise for the reader
# Try adding the appropriate quotes and
# escapes to this quote normalization example
normalized=`echo ``Go on'', he said | sed -e s/`/'/g`

The variable should contain ''Go on'', he said. Can you get it right on the first try?

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

 # Why doesn't || work in [ ] ?
 if [ -f /etc/inetd.conf || -d /etc/xinetd.d ]; then .. 

# And why does it work in [[ ]] ?
 if [[ -f /etc/inetd.conf || -d /etc/xinetd.d ]]; then ..

Short answer: [ is a regular command, and can’t override ||. [[ is shell syntax.

[ is a pseudo-syntactical command. That is, it’s a regular command much like cp or grep, the name just happens to be a single opening square bracket (try ls -l /usr/bin/[).

Seeing as how it’s a regular command, it can’t affect shell grammar. In grep -q kittens file || echo kittens >> file, grep can’t know or do anything about the fact that it’s being used with “||”. The same goes for [ in the example.

In Bash (but not necessarily other shells), [ is now a builtin command emulating /usr/bin/[ for efficiency. There’s no reason why Bash couldn’t make [ a || b ] work, but this would break compatibility.

Instead, we have [[ which is not bound by legacy, and does in fact alter shell syntax. [[ interprets ||, &&, globs and unquoted variable expansions in ways that an external [-command couldn’t, and an internal [ therefore can’t either.

To get around it without using [[, we’d do [ a ] || [ b ] or [ a -o b ].

# Why is this always true (should be false when foo is empty)?
if [ -n $foo ]

Short answer: $foo disappears and [ -n ] is shorthand for [ -n “-n” ], which is true

This comes back to the fact that [ is a regular command. If $foo is empty, the shell runs [ -n ], and there’s no way for [ to know that there was a variable that was expanded out.

[ -n x ] checks that x is not empty. [ x ] is shorthand for the same. [ -n ] therefore checks if dash-n is empty, which it isn’t, and thus the expression is always true.

Why doesn’t [ complain if you have -n without a parameter? Same thing again. It wouldn’t be able to tell that you said [ $1 ] rather than [ -n ], and then scripts would break whenever you work with your parameter list or any other strings starting with a dash.

Instead, you’d use [ -n "$foo" ], which will prevent bash from removing foo when it’s empty, and instead send in a zero-length argument. Or you could use [[ -n $foo ]], since [[ is built into the shell grammar and can tell that there is a variable there.

# Why doesn't this work?
if [ grep -q text myfile ]

Short answer: [ is a command, grep is a command. Choose one.

if [ -n "$foo" ] is the same as if test -n "$foo", which makes it more obvious how to use other commands (if grep -q text myfile).

Pseudo-syntactical elements makes it harder to tell language from implementation for better and for worse. It makes code look neat, but can be confusing. Especially when basically all if-statements include [.

A similar effect is seen in here-documents (cmd << EOF), where many people only ever see the delimiter “EOF” and think it’s some kind of keyword. Next time, use cmd << KITTENS to tip off anyone reading your script.