Encrypt and decrypt in parallel with GPG

As a systems administrator or general hacker, encrypting files will be a fairly common task for backups and other sensitive information. Luckily, GPG is incredibly easy to use, but like most shell programs, runs on one core on your machine. Using GNU Parallel, however, we can run multiple GPG encryption processes at the same time, given a queue of files to encrypt or decrypt.

Encrypt Multiple Files at Once

Encrypting multiple files at once is fairly easy using GNU Parallel:

# find all files in the tree of this working directory and encrypt them
find . -type f -not -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    echo "Encrypting {}..."
    gpg --encrypt --recipient "me@mail.com" "{}"

The above command uses 8 processes (-j 8) to encrypt the files. You can change this to suit your needs and core count.

Decrypt Multiple Files at Once

Likewise, decrypting multiple files at once is also fairly straightforward:

# find all encrypted files and decrypt them
find . -type f -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    echo "Decrypting $file..."
    gpg --decrypt --output "${file%.gpg}" "$file"

Verifying Multiple Files at Once

Being a bit paranoid from some bad experiences with failed encryption runs, I prefer to validate my backups to make sure they’re restorable. To this end, I calculate the SHA-256 sum of the original source files, then decrypt the GPG files and compare the hashes. This is all done in-memory. This will validate two things:

  1. That your backup is restorable, ie: GPG can successfully decrypt all of the data, it didn’t make a mistake in encryption.
  2. That your backup is intact, ie: the encrypted file content is the same as the original file content.

This is fairly similar to the examples above, except we’re using flock to make sure that script output doesn’t occur at the same time:

# verify all GPG files against their source files
find . -type f -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    raw_hash="$(sha256sum "${file%.gpg}" | cut -c 1-64)"
    gpg_hash="$(gpg --batch --decrypt "$file" 2>/dev/null | sha256sum | cut -c 1-64)"
    # lock script output to one process at a time
        flock -e 200    

        echo "$file:"
        echo "================================"
        echo "Raw Hash: $raw_hash"
        echo "GPG Hash: $gpg_hash"
        if [ "$raw_hash" != "$gpg_hash" ]; then
            echo "file verification FAILED, INVALID CHECKSUM"
            echo "file verification succeeded, valid checksum"

        echo ""
    ) 200>/tmp/hashcheck.lock

I like to run this script through tee to be able to analyze it later:

./verify-backups.sh | tee VERIFY

Adding a Certificate Authority to the Trusted List in Ubuntu

Sometimes, working with SSL certificates isn’t all it’s cropped up to be. Heck, most of the time it’s not. It’s painful, time-consuming work.

However, it’s actually not so hard to install a self-signed certificate authority in Ubuntu, using a few commands.

First, install libnss3-tools, which contains the certutil command:

sudo apt-get install libnss3-tools

Next, we’ll copy the public certificate authority file to the certificate store:

sudo cp my_ca.crt /usr/share/ca-certificates/

We’ll now recompile the SSL CA list for Ubuntu, adding our certificate:

sudo dpkg-reconfigure ca-certificates

This will lead to a ncurses menu. In this menu, choose ask, and scroll through the long list of trusted CAs until you find your ‘my_ca.crt’ certificate authority file. Mark it for inclusion with Space, then hit Tab then Enter to finish up.

The last step is to install the certificate into Google Chrome’s registry. (If you’re using Firefox or otherwise, your mileage may vary.) Let’s add it with this command:

certutil -d sql:$HOME/.pki/nssdb -A -t "C,," -n "My Homemade CA" -i my_ca.crt

Great! Now restart Google Chrome and you should now see your sites signed with this CA as being trusted :)

Intel Graphics on a 2011 MacBook Pro in Linux

TerminalOne of the headaches of running Linux on a 2011 MacBook Pro is the bad battery life, heat generation, and the nearly incessant fan noise. As it turns out, this is largely caused by using the dedicated ATI high-power graphics card all the time, as it’s not as easy to get graphics-switching configured on Linux. On OSX, this feature comes out of the box, which is why battery life and heat-generation is so good. I finally got fed up with the heat, noise, and short battery life and took the plunge into configuring onboard Intel Graphics on my MacBook Pro.

Continue reading

Quickly Creating Python and Django Projects with Buildout

Buildout is an awesome build management tool for Python projects, managing dependencies and featuring rich recipes for accomplishing advanced tasks. The main problem seems to be that documentation is lacking and/or terrible. Plus, project setup is painstaking and difficult to get right. Since I have years of experience in Maven, where creating a buildable project takes (literally) seconds to accomplish, this initially made Buildout too much of a hurdle. However, that’s now changed, as I’ve recently released version 0.1.0 of buildout-starter, an installable Python script to make creating Python and Django projects with Buildout a snap.

Continue reading

Adding Icons to a Theme in Elementary Luna

Elementary Luna IconSince I’m using Elementary Luna as my daily driver, I’m using it with a lot of apps not designed with it in mind. Rather, the Elementary icon set doesn’t include icons for these programs, such as Eclipse, NVIDIA X Server Settings, Skype, etc. Luckily, a beautiful icon set called Faenza does. By installing Faenza and patching our Elementary icon set with links to Faenza icons, we can fill in the gaps in our icon set.

Continue reading

Creating Long-Term Backups with Amazon Glacier on Linux

Amazon Glacier LogoIf you haven’t heard about Amazon Glacier already, it’s definitely something to be excited about. Amazon Glacier is a service that makes it extremely affordable to store gigabytes upon gigabytes of data for the long term in the cloud. Your data is stored immediately, but retrieval requests take at least 4 hours to make your data available again for your downloading. Let’s back up a ton of files on Linux to Glacier.

Continue reading

A Better Magic Trackpad Experience in Linux

If you’re like me, you own a nice Apple Magic Trackpad. You’ve also paired it with your Linux box and it’s working great. However, it could probably work better. The defaults for the device are, in my opinion, pretty unresponsive and at times really bizarre. Let’s walk through some hacks and fixes in order to get things working better.

Continue reading

Patching btusb to enable compatibility with the BCM20702A0 chip

I just recently built a brand new machine and noticed that I wasn’t seeing my Bluetooth chip. After a bunch of digging, I found that my Broadcom combo chip which came with my ASUS Maximus V motherboard hosted both a BCM43228 WiFi module and a BCM20702A0 Bluetooth module. Stranger still is that the Bluetooth chip is actually an embedded USB device running on the PCI port.

Continue reading