Speeding up archive operations on OS X and Linux
I work with a lot of virtual machine images and other big data sets that needed to be archived, or moved around. If I run a command like tar -cjf archive.tar.bz2 directory it will churn away on a single core.
Given that all my machines have a lot of cores, it would be better if the tar command made use of all available cores. Thankfully someone created a tool called pigz (pronounced "pig-zee") that will allow the use of these CPU cores.
There are 2 parts to the setup for pigz: get the utility, and force tar to use it. On my Mac I grabbed pigz from brew:
brew install pigz
Then I added an alias so that tar uses this by default:
alias tar='tar --use-compress-program=pigz'
Remember that you might not want to use an alias if your profile travels with you to environments where the use of all cores could create problems.