Update 2016-02-13: I’m now making use of Duplicity’s “–full-if-older-than” backup option and the “remove-all-but-n-full” clean up mode to keep a moving window of 3 months of backups. This way your backups don’t keep growing and growing. I’ve updated the examples in below.

Duplicity – Pleasure and Simplicity in Your Backups

If you’re like me, you’ve spent a long time, in many different configurations, trying to come up with a simple yet flexible, easy to use yet space efficient backup solution for a long, long time. Well, I think I’ve finally jumped ship from shell scripts and tarballs to duplicity: bandwidth-efficient backup using the rsync algorithm. Although duplicity talks up encryption, and it’s great and makes it easy, it’s not required and I don’t use it. Here’s some tips and tricks to get you started…

Before we get into duplicity, I’d like to state my personal goals with backup. It’s relatively simple and yet if you’re a system admin of any vintage you know how maddening it can be to come up with the perfect backup script. I’ve gone from dead simple 7 day rotations to complicated programs with their own opaque backends. The good news is duplicity allows me to write a simple shell script in the way that I want to write it and it takes care of all the rest of it like archiving, compressing, incremental backups, etc. What’s better still is it has easy commands to do things like: full backups, incremental backups, status of your backups, file lists, extraction, efficient extraction of specific files (Woohoo! No more 3 hour waits while you’re 6GB tarball decompresses when all you wanted was one folder!), verify/diff backup against source, clean up old backups with an age argument, using or not using encryption, and efficient rsync local and remote copying.

Duplicity hasn’t gotten much love in past years but it’s gotten a refresh and its latest release at the time of writing is May 6, 2015, just a few weeks ago.

So, here’s a quick primer of the most useful commands I use with duplicity. You can find the whole manual with the `man` command:

# First and subsequent backups without encryption
# - First is a full backup, subsequent are incremental
duplicity /etc file:///backup/duplicity/etc --no-encryption

# Explicit Full backup
duplicity full /etc file:///backup/duplicity/etc --no-encryption

# Explicit incremental backup (will abort if no old signatures found)
duplicity incr /etc file:///backup/duplicity/etc --no-encryption

# Check status of backup (set count, volume count, full backup count,
# incremental backup count, orphaned or incomplete backup sets, etc.)
duplicity collection-status file:///backup/duplicity/etc/ --no-encryption

# List current files in backup
duplicity list-current-files file:///backup/duplicity/etc/ --no-encryption

# Extract backup to a given directory
duplicity restore file:///backup/duplicity/etc /tmp/etc.old --no-encryption

# Extract specific file/directory from backup
duplicity restore --file-to-restore passwd file:///backup/duplicity/etc/ /tmp/ --no-encryption

# Extract specific point-in-time file/directory from backup
# - See TIME FORMATS in `man duplicity`
#   ^ ISO 8601 Date-Time Format
duplicity restore --file-to-restore passwd --time '2017/01/24' file:///backup/duplicity/etc/ /tmp/ --no-encryption

# Verify backup against source (show diffs in backup versus source)
duplicity verify file://duplicity/etc /etc --no-encryption

# Remove backup sets older-than / Clean up / Consolidate backups
# - See `man duplicity` TIME FORMATS
# - 7D = 7 Days
duplicity remove-older-than 7D file:///backup/duplicity/etc/ --no-encryption

# Disable encryption add argument:
--no-encryption

If you’re wondering what my backup script looks like, the 15 line script (which includes a database dump and rsync to remote backup), looks something like this:

#!/bin/sh

# Backup
duplicity /etc file:///backup/duplicity/etc --no-encryption --full-if-older-than 3M
duplicity /root file:///backup/duplicity/root --no-encryption --full-if-older-than 3M
duplicity /home/matt file:///backup/duplicity/matt --no-encryption --full-if-older-than 3M

# Clean up
duplicity remove-all-but-n-full 1 --force file:///backup/duplicity/root --no-encryption
duplicity remove-all-but-n-full 1 --force file:///backup/duplicity/etc --no-encryption
duplicity remove-all-but-n-full 1 --force file:///backup/duplicity/matt --no-encryption

# Backup database
...backup database...

# Sync off-site
...rsync to rsync.net - great service by the way...

I hope you find some value in duplicity and I hope it helps you get a little sanity back in your backups by providing a simple frontend with a powerful backend.