The Association of Mad Scientists

Backing up my deployments

Recently, I've been writing about my deployment architecture, and a key component to that is making sure that if you have made a mistake that you can at the very least recover your users' data. For example, when deploying a database, it often creates files owned by root. If these files lie in a bind-mount, and the application isn't started by the root user, they won't persist a reboot of the server. This is an easy mistake to make, and one I've made myself.

There are many more reasons why your data is always at risk, the first of which that come to mind are hardware decay and malicious manipulation of the data. A good backup strategy makes sure to keep track of more than one version of a file, just in case a mistake doesn't get caught before the next backup cycle, and is stored on a physically separate medium from the drive that is actually serving the data.

It's also a good idea to take multiple backups, but this article only concerns a single method of automated, incremental backups. We'll be taking the example of the Traefik reverse proxy I previously covered, using a script called rsnapshot, on Ubuntu 18.04.

The rsnapshot application is configured with a configuration file that is pointed to when it is run, which we will be triggering with a cron job. After you have rsnapshot installed (it's available from the default repositories in most major Linux distributions), open /etc/rsnapshot.conf for reference. You can copy most of the file as-is, except the file locations, to your liking. I made the following changes.

  • snapshot_root should be set to the place you want your backups to be placed. Ideally this should be an external drive. If snapshot_root is on an external drive, you should set no_create_root to 1.
  • I replaced all of the cmd_* entries with the output of which for the respective commands. For example, cmd_cp should be set to the output of which cp.
  • I changed the names of the backup levels, like so:
#########################################
#     BACKUP LEVELS / INTERVALS         #
# Must be unique and in ascending order #
# e.g. alpha, beta, gamma, etc.         #
#########################################

retain	daily	6
retain	weekly	6
# for monthly snapshots change weekly to 4
# and uncomment the following line
#retain	monthly	6
  • I set a lock- and logfile for this particular instance of rsnapshot, as I intend to have others that may run concurrently.
  • Finally, the backup directives need set. These are just tab-separated entries, like my Traefik backup:
backup		/opt/docker/traefik/mounts 	traefik/

Once the configuration file is all set up, you have to set up a crontab entry. For more on how crontabs work, see man crontab, but to summarize briefly, it's a table of scripts to run on a schedule. Each line contains a script, and five whitespace-separated time values, in the order

  1. Minute
  2. Hour
  3. Day of the month
  4. Month of the year
  5. Day of the week

So the following crontab entry tells the cron daemon to run the command command at 4:59PM on Tuesdays in March, you would add this to your crontab entry:

59	16	*	3	2	command

In our case, we want to run the rsnapshot command, and tell it to take the daily snapshots Monday through Saturday, then a weekly snapshot on Sunday. This takes two crontab entries:

# daily backups
30  06  *   *   1-6 rsnapshot -c /home/scott/Documents/docker/docker/rsnapshot.conf daily
# weekly backups 
00  07  *   *   7   rsnapshot -c /home/scott/Documents/docker/docker/rsnapshot.conf weekly

These run at 06:30 and 7AM, the times when my git commit history shows I'm least likely to be up.

The rsnapshot application works by storing hard links to files that already exist, meaning that only your most recent snapshot is the full size of your mount, and the others will be smaller.

These are the filesizes of my current backups:

$>> du -hd1 docker-mounts     
	1.9G	docker-mounts/daily.0
	82M	docker-mounts/daily.1
	5.3M	docker-mounts/daily.2
	159M	docker-mounts/daily.3
	55M	docker-mounts/daily.4
	71M	docker-mounts/daily.5
	2.3G	docker-mounts