Encrypt and decrypt in parallel with GPG

As a systems administrator or general hacker, encrypting files will be a fairly common task for backups and other sensitive information. Luckily, GPG is incredibly easy to use, but like most shell programs, runs on one core on your machine. Using GNU Parallel, however, we can run multiple GPG encryption processes at the same time, given a queue of files to encrypt or decrypt.

Encrypt Multiple Files at Once

Encrypting multiple files at once is fairly easy using GNU Parallel:

# find all files in the tree of this working directory and encrypt them
find . -type f -not -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    echo "Encrypting {}..."
    gpg --encrypt --recipient "me@mail.com" "{}"
'

The above command uses 8 processes (-j 8) to encrypt the files. You can change this to suit your needs and core count.

Decrypt Multiple Files at Once

Likewise, decrypting multiple files at once is also fairly straightforward:

# find all encrypted files and decrypt them
find . -type f -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    file="{}"
    echo "Decrypting $file..."
    gpg --decrypt --output "${file%.gpg}" "$file"
'

Verifying Multiple Files at Once

Being a bit paranoid from some bad experiences with failed encryption runs, I prefer to validate my backups to make sure they’re restorable. To this end, I calculate the SHA-256 sum of the original source files, then decrypt the GPG files and compare the hashes. This is all done in-memory. This will validate two things:

  1. That your backup is restorable, ie: GPG can successfully decrypt all of the data, it didn’t make a mistake in encryption.
  2. That your backup is intact, ie: the encrypted file content is the same as the original file content.

This is fairly similar to the examples above, except we’re using flock to make sure that script output doesn’t occur at the same time:

# verify all GPG files against their source files
find . -type f -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    file="{}"
    raw_hash="$(sha256sum "${file%.gpg}" | cut -c 1-64)"
    gpg_hash="$(gpg --batch --decrypt "$file" 2>/dev/null | sha256sum | cut -c 1-64)"
    
    # lock script output to one process at a time
    (
        flock -e 200    

        echo "$file:"
        echo "================================"
        echo "Raw Hash: $raw_hash"
        echo "GPG Hash: $gpg_hash"
    
        if [ "$raw_hash" != "$gpg_hash" ]; then
            echo "file verification FAILED, INVALID CHECKSUM"
        else
            echo "file verification succeeded, valid checksum"
        fi

        echo ""
    ) 200>/tmp/hashcheck.lock
'

I like to run this script through tee to be able to analyze it later:

./verify-backups.sh | tee VERIFY

MacBook Pro 8,1/8,2/8,3 Bluetooth Issues on Linux


Oh, by the way, I just got a shiny new MacBook Pro to run Linux on! Hooray! It’s an awesome machine, but I just wanted to post this bug to the general public to save people the 3 days I’ve spent trying to figure it out.

Essentially, what it boils down to is this: the b43 Linux driver that you’ve compiled and installed interferes hardcore with the adjacent Bluetooth chip on your MacBook.

Let’s do a little test to demonstrate. While connected to a WiFi network with b43, put a Bluetooth device into pairing mode near your machine. Make sure the bluez package is installed. Next, run the following:

$ hcitool scan
Scanning...

You’ll probably see it say Scanning... and then… nothing. It won’t see your device. Period.

Let’s continue with our experiment. Unload the b43 module with sudo rmmod b43. Your internet will go down if you’re on WiFi. Now, run hcitool scan again.

$ hcitool scan
Scanning...
    XX:XX:XX:XX:XX:XX    Nexus One

Aha! Found you! So, it seems that the problem is with the driver itself. This is kind of a bummer, to say the least, but hopefully they’ll have it patched soon. Until then, sudo modprobe b43 and continue on, weary warrior.

For the record and for Google, I’m running a 2011 MacBook Pro 8,3 with Ubuntu 11.10 Oneiric Ocelot 64bit/amd64.

Update: A Working Workaround!

As noted by Benoit in the comments below, you can actually get things working with a little workaround. First, unload the b43 driver from the kernel:

$ sudo rmmod b43

Next, reload the module, turning Bluetooth coexistence support off:

$ sudo modprobe b43 btcoex=0

As counterintuitive as it seems, it works! Hooray! I’m now able to listen to music with my Bluetooth headphones and use the built in B4331 wireless card in my MacBook at the same time!!! The only issue I’ve encountered is that when you’re spiking and getting really high upload/download rates, you’ll notice that Bluetooth audio skips a bit. Luckily, this is a pretty minimal problem and doesn’t prevent you from using your Bluetooth or your B4331 WiFi card in your MacBook Pro.

Run Wowza Media Server as a Non-root User on Linux

I recently had an interesting scenario where I needed to run Wowza as a non-root user in order to keep file permissions the same across a number of servers, as I was recording video on one server and serving it over VOD on another.

Unix Permissions have always seriously confused me and vexed me to no end, but I found that the actual Wowza Media Server User’s Guide gives instructions on how to run the server as non-root. For a given user user with a primary group of user:

chown user:user /usr/local/WowzaMediaServer
chown -R user:user /usr/local/WowzaMediaServer-2.2.4
chmod -R 775 /usr/local/WowzaMediaServer-2.2.4
rm -f /var/run/WowzaMediaServer.lock /var/run/WowzaMediaServer.pid

If you’re a l33t h4ck3r like me, you can combine the last line like:

rm -f /var/run/WowzaMediaServer.{lock,pid}

…but only if you’re using Bash.

After you do this, modify both /usr/local/WowzaMediaServer/bin/startup.sh and /usr/local/WowzaMediaServer/bin/wms.sh to prepend sudo -u user to the 24th (or last) line. Basically, for the line that starts with $_EXECJAVA ..., it should now start with sudo -u user $_EXECJAVA ....

I’ve also put the Wowza docs on my Google Docs account so as to make them a lot easier to access. Find them below:

Terminal Linux: Clipboard Magic

Just posted an awesome new little tutorial on TerminalLinux.org on getting your awesome on with the X clipboard on your Linux machine. Automate and mass-transform your clipboard contents from your handy-dandy terminal!

# insert a tab at the beginning of each line of the contents of the clipboard 
xclip -o | sed 's:^:\t:g' | xclip -selection clipboard

Installing Adobe AIR 1.x on 64bit Linux

Adobe AIRAdobe currently offers an installer for AIR for almost all operating systems. Windows, Mac, and Linux, all of the major platforms are covered. However, one installer is missing… a 64bit one for Linux. For those of us who are die-hard Linux users who also prefer the speed and power of running 64bit operating systems, Adobe AIR won’t just install out of the box and work. This article is to help you get Adobe AIR installed and running on your 64bit Linux system in no time flat.

Continue reading

Ripping DVD’s and CD’s to ISO format in Linux

There are many reasons why one would want to rip a disc to an ISO file. The ISO format stores the disc locally on your hard drive in a way that it can be easily and quickly burnt again to removable media. If you’ve ever wanted to keep a backup copy of some software on your computer, or you would like to store entire DVDs locally on your filesystem, Linux can help you out with that. The program “dd,” provided by most Linux distributions by default, allows for quick, simple ripping of CD’s and DVD’s from a terminal. Let’s get ripping. Continue reading

Music Management in Linux: Ripping CD’s with abcde

When one sets out to rip a CD in Linux, he is confronted by many options. The user can use a graphical music player such as Rhythmbox, Banshee, or Amarok to rip said CD, along with many other dedicated ripping solutions. I recently purchased Rosetta Stone Hebrew Levels 1 – 3 which comes with 12 (read ‘em, TWELVE) discs of audio companion material. Let’s just say I needed a way to rip 12 discs fast, preferably in a terminal. Enter abcde. Continue reading

Flash Player on Linux – Installing Native 64bit Flash Player

It seems that installing Flash Player on an Ubuntu desktop is getting easier and easier with every release of the popular Linux operating system. However, its still very apparent that there are still problems pairing the two together in some areas. In the past, it has been less-than-easy to install the 64bit Flash Player plugin in Linux. However, Adobe just released ¬†refresh to the alpha, so let’s see if it has taken care of some of the ease-of-installation problems. Continue reading