Applications often behave differently in subtle ways when stdout is not a terminal. Most of the time, this is done so smoothly that the user isn’t even aware of it.
When it works like magic
Consider ls:
vidar@vidarholen ~/src $ ls PyYAML-3.09 bsd-games-2.17 nltk-2.0b9 alsa-lib-1.0.23 libsamplerate-0.1.7 pulseaudio-0.9.21 bash-4.0 linux tmp bitlbee-1.2.8 linux-2.6.32.8 vidar@vidarholen ~/src $
Now, say we want a list of projects in our ~/src dir, ignoring version numbers:
# For novelty purposes only; parsing ls is a bad idea vidar@vidarholen ~/src $ ls | sed -n 's/-[^-]*$//p' PyYAML alsa-lib bash bitlbee bsd-games libsamplerate linux nltk pulseaudio vidar@vidarholen ~/src $
Piece of cake, right?
But think about the magic that actually happened there: We started out with three lines of coloured text, ran it through sed to search&replace on each line, and ended up with nine lines of uncoloured text.
How did sed filter the colours? How did it put each filename a separate line, when the same does not happen for echo "foo bar" | sed ..?
The answer, of course, is that it didn’t. ls detected that output wasn’t a terminal and altered its output accordingly.
When outputting to a terminal, you can be fairly sure that the user will be reading it directly, so you can make it as pretty and unparsable as you want. When output is not a terminal, it’s likely going to some program or file where pretty output will just complicate things.
Life without magic
Try the previous example with ls -C --color=always instead of just ls, and see how different life would have been without this terminal detection. You can also try this with xargs, to see how colours could break things:
vidar@vidarholen ~/src $ ls -C --color=always | xargs ls -ld ls: cannot access PyYAML-3.09: No such file or directory ls: cannot access alsa-lib-1.0.23: No such file or directory ...
The directories obviously exist, but the ANSI escape codes that give them that cute colour also prevents utilities from working with them. For additional fun, copy-pasting this error message from a terminal strips the colours, so anyone you reported it to would be quite stumped.
Magic efficiency tricks
It’s not all about making output pretty or parsable depending on the situation. Read/write syscalls are notoriously expensive; reading anything less than about 4k bytes at a time will make disk reads CPU bound.
glibc knows this, and will alter write buffering depending on the context. If the output is a terminal, a user is probably watching and waiting for it, so it will flush output immediately. If it’s a file, it’s better to buffer it up for efficiency:
vidar@kelvin ~ $ strace -e write -o log grep God text/bible12.txt
01:001:001 In the beginning God created the heaven and the earth.
...
vidar@kelvin ~ $ wc -l log
3948 log
In other words, grep wrote about god 3948 times (insert your own bible forum jokes).
vidar@kelvin ~ $ strace -e write -o log grep God text/bible12.txt > tmp
vidar@kelvin ~ $ wc -l log
64 log
This time, grep produced the exact same output, but wrote to a file instead. This resulted in 64 writes – about 1% of the more interactive mode!
Spells of confusion
Sometimes magic can confuse and astound. What if output is kinda like a terminal, only not?
ls -l gives the user pretty colours. ls -l | more does not. The reason is not at all obvious for users who just consider ” | more” a way to scroll in output. But it works, even if it’s not as pretty as we’d like.
Here’s a much more confusing example (just go along with the simplified grep):
# Show apache traffic (works) cat access.log # Show 404 errors with line numbers (works) cat access.log | grep 404 | nl
Basic stuff.
# Show apache traffic in realtime (works) tail -f access.log # Show 404 errors with line numbers in realtime (FAILS) tail -f access.log | grep 404 | nl
While the logic is the same as before, our realtime error log doesn’t show anything!
Why? Because grep’s output isn’t a terminal, so it will buffer up about 4k worth of data before writing it all in one go. In the mean time, the command will just seem to hang for no apparent reason!
(Observant readers might ask, “Isn’t tail buffering?”. And it might be or it might not. It depends on your version and distro patches.)
Mastering magic
Ok, so what can we do to take charge of these useful peculiarities?
Many apps have flags for this, though none of them are POSIX.
GNU ls lets you specify -C for columned mode, and --color=always for colours, regardless of the nature of stdout.
sed has -u, grep has a --line-buffered. awk has a fflush function. tail, if yours buffers at all, has a -u since about 2008 which as of now isn’t in debian stable.
If your app doesn’t have such an option, there’s always unbuffer from Expect, the interactive tool scripting package.
unbuffer starts applications within its own pseudo-tty, much like how xterm and sshd does it. This usually tricks the application into not buffering (and maybe to prettify its output).
Obviously, this depends on the app using standard C stdio, or that it checks for a terminal itself. Apps can unintentionally be written to avoid this, like when setting Java’s System.Out to a BufferedOutputStream.
And finally… how can you create such behaviour yourself?
if [[ -t 1 ]] #if stdout is a terminal
then
tput setaf 3 #Set foreground to yellow
fi
echo "Pure gold"