Encrypt and decrypt in parallel with GPG

As a systems administrator or general hacker, encrypting files will be a fairly common task for backups and other sensitive information. Luckily, GPG is incredibly easy to use, but like most shell programs, runs on one core on your machine. Using GNU Parallel, however, we can run multiple GPG encryption processes at the same time, given a queue of files to encrypt or decrypt.

Encrypt Multiple Files at Once

Encrypting multiple files at once is fairly easy using GNU Parallel:

# find all files in the tree of this working directory and encrypt them
find . -type f -not -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    echo "Encrypting {}..."
    gpg --encrypt --recipient "me@mail.com" "{}"
'

The above command uses 8 processes (-j 8) to encrypt the files. You can change this to suit your needs and core count.

Decrypt Multiple Files at Once

Likewise, decrypting multiple files at once is also fairly straightforward:

# find all encrypted files and decrypt them
find . -type f -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    file="{}"
    echo "Decrypting $file..."
    gpg --decrypt --output "${file%.gpg}" "$file"
'

Verifying Multiple Files at Once

Being a bit paranoid from some bad experiences with failed encryption runs, I prefer to validate my backups to make sure they’re restorable. To this end, I calculate the SHA-256 sum of the original source files, then decrypt the GPG files and compare the hashes. This is all done in-memory. This will validate two things:

  1. That your backup is restorable, ie: GPG can successfully decrypt all of the data, it didn’t make a mistake in encryption.
  2. That your backup is intact, ie: the encrypted file content is the same as the original file content.

This is fairly similar to the examples above, except we’re using flock to make sure that script output doesn’t occur at the same time:

# verify all GPG files against their source files
find . -type f -iname "*.gpg" | sort | parallel --gnu -j 8 --workdir "$PWD" '
    file="{}"
    raw_hash="$(sha256sum "${file%.gpg}" | cut -c 1-64)"
    gpg_hash="$(gpg --batch --decrypt "$file" 2>/dev/null | sha256sum | cut -c 1-64)"
    
    # lock script output to one process at a time
    (
        flock -e 200    

        echo "$file:"
        echo "================================"
        echo "Raw Hash: $raw_hash"
        echo "GPG Hash: $gpg_hash"
    
        if [ "$raw_hash" != "$gpg_hash" ]; then
            echo "file verification FAILED, INVALID CHECKSUM"
        else
            echo "file verification succeeded, valid checksum"
        fi

        echo ""
    ) 200>/tmp/hashcheck.lock
'

I like to run this script through tee to be able to analyze it later:

./verify-backups.sh | tee VERIFY

Creating Long-Term Backups with Amazon Glacier on Linux

Amazon Glacier LogoIf you haven’t heard about Amazon Glacier already, it’s definitely something to be excited about. Amazon Glacier is a service that makes it extremely affordable to store gigabytes upon gigabytes of data for the long term in the cloud. Your data is stored immediately, but retrieval requests take at least 4 hours to make your data available again for your downloading. Let’s back up a ton of files on Linux to Glacier.

Continue reading

The State of Freedom on the Web

Before you write this off as a political/religious/philosophical rant… well, you should probably read it. As both a consumer who purchases hardware and as a producer who creates software and media, I deal with the concept of intellectual property and “free as in freedom” when it comes to ownership and licensing. While I’m not completely “frum” when it comes to using only free software, the issue of SaaS is a major issue today for intellectual freedom. This will be a multi-part article, covering the various facets of the issue. This first article will serve as the introduction to what freedom means in terms of the web and your content, meaning your media, when you share it on the web.

Continue reading

New Site! Enter TerminalLinux.org

Ever wanted to learn to make the most out of your Linux? Enter TerminallLinux.org, a site dedicated to teaching you some of the most incredible things that can be done with Linux. This is a personal learning project for me and as I learn new commands, you will too. Be sure to subscribe to the RSS feed and to follow us on Twitter!

Terminal Linux: Tips, Tricks, and Code to Rock Your World.

as3signals – An Awesome Solution to Events/Signals in AS3

There has been quite a lot of buzz lately in the community about a new library by Robert Penner called as3-signals. John Lindquist recently posted a video tutorial on using it, and I thought I’d follow up with a nice text-based tutorial explaining the common ins-and-outs of the library.
Continue reading

Flex 4, Ant, and an OutOfMemoryError

I just started messing around with the new Flex 4 SDK beta, and man is it cool. The rework of the states system is reason alone to move now to Flex 4. Two-way data-binding, virtualization (I have not a clue as to what that means in Flex), and tons of other goodies look very promising for the Flex product line. Adobe, hats off to you for one-upping Flex 3, which indeed must have been a challenging feat.

Today, however, I ran into a problem with running the Flex MXMLC Ant task to compile my application. I kept getting a Java OutOfMemoryError whenever I’d try to compile. Initially, I wasn’t seeing this problem, but as my application got more complex, like it always does, I couldn’t be rid of the problem.

Luckily for you and I, a few people have run into this before and so I got a few pointers from a couple colleagues on how to work around the problem.

The first workaround I stumbled upon was a Windows-specific fix (for running ant from the command-line): http://n1aub0.cowurl.com | Apparently, you can set an environment variable “ANT_OPT” to include default JVM arguments when running Ant from the console.

The workaround that really fixed the problem for me in Eclipse was found on Sönke Rohde’s blog. Mad props to Sönke for finding a way to fix this. Again this fix isn’t something new to me, but it’s just nice to have these links laying around whenever you run into them. Adjusting Java’s memory size isn’t necessarily an “easy” thing to do, so it helps to have a reference.

Lametarded Firefox Flash Player Crash

http://bugs.adobe.com/jira/browse/FP-17

This is the lamest thing in the history of forever. It will crash Firefox 90% of the time. How does it happen?

Basically the scenario is this: if Flash Player originates a resizing of itself on the page, Firefox will immediately crash and close. Can’t be any more clear than that. Why was I trying to get Flash to resize? My current project needed to have a “sidebar” which would expand when necessary, displaying content when it was needed, hidden when it wasn’t. I used a really sweet Javascript tween library as well as JQuery to make it all happen, and it worked fine when the animation was triggered by a HTML button tag.

But as soon as I tried doing it with Flash Player, I was presented with more than an hour of crashing Firefox. There’s no consistent way to make it work, period.

If you are suffering from this horrible Flash Player plague, I feel your pain. This really sucks.

Dependency Injection via Spring

I’ve been doing a lot of work with Spring and java persistence lately, and I’ve definitely been loving it. Dependency injection is one of the coolest things I have ever seen in developing Java systems and Spring allows you to almost externalize all of your classes’ needs and eliminate the ‘new’ operator altogether. This is amazing to say the least. Having started all of my coding in Actionscript 2, the world of Java is somewhat foreign to me, but refreshing to say the least.

I built some of my first generic classes and interfaces the other day and I take what I said about generics back, they are extremely useful. By extending and implementing these classes and interfaces, I literally can save myself around an hour per DAO I create, and let Spring manage everything without skipping a beat. It makes life so much easier!

While I am still totally a newbie at Spring, I’m so glad for Red5′s use of it. It makes development so much easier and really helps in building modular applications. “Spring in Action” is on my to-get list on Amazon and I can’t wait to get it and learn a lot.