Space Efficient Mysqldump Backups Using Incremental Patches

Update 2015-08-18: Boy do I feel silly! It turns out there’s a much simpler and much more robust way of doing what I’ve done with the scripts below. It turns out that, using any revision control system (eg. cvs, git, svn) that stores revisions as deltas (and most if not all do), all you need to do is copy anything into a revision control repository and commit it. Tada! The rcs takes care of the incremental part for you by its use of revision deltas (ie. patches). As a big fan of git I was hoping there was a way for it to fill this role. I had mistakenly thought that git stores whole files without diffs/deltas for every revision. This is true until git garbage collects as I found out with my Stack Overflow question: Can git use patch/diff based storage? There’s some great reading there, check it out. Simply garbage collect after adding and committing in git and you automatically get space efficient incremental backups with the bonus of the robustness and reliability of git (or whatever rcs you choose). Bonus: You can delta anything you can store in an rcs repository meaning files, binary or text, archives, images, etc. You still get the space savings! So, quite literally, my database backup is now something like this: (1) mysql dump, (2) git add dump, (3) git commit dump, (4) git gc. Simple, powerful, elegant, beautiful. As it should be!

Space Efficient Mysqldump Backups Using Incremental Patches

I’m now using Duplicity for super convenient one-liner style incremental backup commands in a simple shell script (seriously, it’s like three commands long) but what I’m missing is incremental space-savings on my database dump. Right now my mysqldump produces about a 40MB file, about 10MB compressed. It’s irked me for some time that there’s no simple way to do intra-file incremental backups. I’ve also wanted to do intra-day, not just daily, backups. Duplicity’s incremental backups allow for that but full database backups add up quickly. Well, I finally went ahead and wrote a shell script to do it and a recover script that can recover to any date in the series of backups – just like duplicity. The key was interdiff for incremental patches. Here’s how I did it…

Continue reading “Space Efficient Mysqldump Backups Using Incremental Patches”

Duplicity – Pleasure and Simplicity in Your Backups

Update 2016-02-13: I’m now making use of Duplicity’s “–full-if-older-than” backup option and the “remove-all-but-n-full” clean up mode to keep a moving window of 3 months of backups. This way your backups don’t keep growing and growing. I’ve updated the examples in below.

Duplicity – Pleasure and Simplicity in Your Backups

If you’re like me, you’ve spent a long time, in many different configurations, trying to come up with a simple yet flexible, easy to use yet space efficient backup solution for a long, long time. Well, I think I’ve finally jumped ship from shell scripts and tarballs to duplicity: bandwidth-efficient backup using the rsync algorithm. Although duplicity talks up encryption, and it’s great and makes it easy, it’s not required and I don’t use it. Here’s some tips and tricks to get you started…

Continue reading “Duplicity – Pleasure and Simplicity in Your Backups”

Better Colour Schemes for PuTTY

Update 2014-09-20: Try the Consolas font instead of the default Courier New. It’s slightly smaller but still quite readable and permits a narrow 80×40 window which I can lay out side-by-side 3 times on my desktop (1920×1080) without overlapping. That’s something you can’t do with Courier New at 80×40. Give it a try under Putty Configuration > Window > Appearance > Font Settings.

Better Colour Schemes for PuTTY

Check out the igvita.com colour schemes for putty. They’re very good and a big improvement on the default colour scheme. If I have to use a terminal on windows, putty is it and these colour schemes really make a difference in the usability of the terminal.