facti.net

Backing up is one of the most important things to be done with a computer. Failure will happen, it is only a question of when, and information loss can be quite painful. Unfortunately the amount of information available now makes it more difficult to effectively manage backups and archives.

Even though my main backup medium of choice (external hard drives) is getting cheaper and bigger about as fast as my information growth, the pipe the information must flow through (USB or FireWire, or DVD for archives) is not keeping up. This makes it slower and more laborious to keep backups up-to-date as more information is added.

If you have a good strategy leave me a comment and let me know what it is, I would love to hear about it. My current strategy is to have a current view and a historical view of important data. I don't backup the OS and installed applications since I always ensure I can rebuild the OS and applications from my install media.

The current view is backed up to external hard drives, one of which I keep locally in an encrypted live file system for fast recovery, and another of which I rotate off-site for disaster recovery.  In this way I can recover quickly any current stuff.  Making this backup is easy and fast using a script that invokes rsync for each of the folders being backed up, e.g.

rsync -av --delete /home/ /mnt/backup/home/

The historical view is burned to DVD. By creating and keeping DVDs I can reconstruct a file that may have been deleted from my current view and removed from the current backups because it was no longer needed, or not detected before a new backup was made. The capacity and creation time of DVDs is much slower than hard drives, and the historical archives will not be needed as often, so I compress and encrypt the DVD archives. This makes it much slower to recover a file, but also saves a great deal of space.

To create the compressed and encrypted archive files for the full-image archive I use tar, gpg, and split like this :

tar c home/ | gpg -c | split --bytes=1400m --numeric-suffixes - archives/home.dat.

The tar reads the folder and passes the output to gpg using a pipe, gpg then encrypts the stream and passes to split which splits the output stream to files that are 1.4GB in length. I use 1.4GB files since they are smaller than the maximum regular ISO file size allowed (just over 2GB)  and will fit evenly into a normal DVD.

Before burning I create a listing of the file names that will go into an archive so that I can tell if the file I want is in a given archive, and I store these listings in the current system view for easy searching. To create a local listing, simply use ls, something like this:

ls -lR folder/ > archives/archive-20080313-folder.txt

The archive files are then burned, 3 to a DVD, along with a copy of the listing. Ideally this should be burned twice since any corruption to the gpg file will result in an invalid archive.