Linux command line tricks you need to know: IO redirection and piping
Feb 14, 2012The linux command line has a rich set of utilities and functionality. One of its beauties is that there are many utilities with a simple purpose that are in themselves very useful, but can also be combined to do an amazing array of jobs. This is a fundamental part of linux with tricks built into the bash command line shell to make that very easy.
In this post I'm going to be talking about two such tricks that every command line warrior must know: IO redirection, and piping. Whether you're writing shell scripts, or doing one-off commands on the command line, both are very useful.
IO Redirection
IO redirection is useful, for example, if you want to store the output of a program for later reference. I like to log almost everything, so in my crontab instead of my output going into a black hole I redirect it to a file in my logging directory using the ">" operator.
To redirect the standard output of a command to a file:
echo "hey" > filename
This will create a new file or wipe the contents of an existing file and write the output of the command to it. If you would rather append to the file if it already exists use:
echo "how are you?" >> filename
Here's an example with a little bit more import. To do a regex substitution and save the results in a new file:
sed 's/obama/palin/g' liberal.txt > conservative.txt
What about input?
You can use a file as the source of input to any command. The "<" operator will send the contents of a file to the standard input of a command.
sort < christmas_list.txt
This will send the contents of the file "christmas_list.txt" to the "sort" command, which will display the sorted results. You can also chain the different io redirection operators. For example:
sort < christmas_list.txt > sorted_christmas_list.txt
What about standard error?
You can also redirect the standard error (stderr) output if you want to capture error messages.
command 2> error.log
This will redirect the stderr to a file. "2" always represents the standard error ("1" represents the standard output). If you want to redirect both stdout and stderr to the same file:
command > filename 2>&1
Here's an example of a poor man's daemonize that'll run a command in the background and capture the standard output and error to a file.
nohup command > nohup.log 2>&1 &
nohup (short for nohangup) makes the process ignore interrupts, quit signals, and hangups that may accidentally cause the process to abort. The final "&" means to run it in the background.
The abyss: /dev/null
If you don't care to see the output you could redirect the output to /dev/null.
command > /dev/null 2>&1
/dev/null is a special file that discards all the data written to it.
Pipes
Pipes are another useful way to manipulate the input and output of a program. They are used to send the output of one program to the input of another.
To pipe something use the pipe "|" character and the standard output of one command will be sent to the standard input of the next command.
As a quick example, you can pipe the "history" command through "grep" to see only the history you're interested in.
history | grep "python"
This will only display your previous python commands for the current session. To pipe the bash history of your session through the "more" program, which allows you to page through a screenful of output at a time, you use the following:
history | more
The same can be done with any program that may produce a large amount of output (eg. "ls", "find" etc).
More examples
Sort the contents of a file alphabetically and save the result to a new file.
cat christmas_list.txt | sort > christmas_list.txt
Display processes with the highest memory usage:
ps aux | sort -nrk 4 | head
If you want the output to go to both the display and to a file:
command | tee filename
Simple tools, but limitless possibilities
I'm only really scratching the surface with this introduction. A really cool feature I didn't even touch upon are named pipes. The amount of things you can do on the command line makes linux a deep user experience because there's so much you can discover. I'm constantly discovering great new tricks I didn't know about before. The possibilities are limitless.
