Vašek's blog

Atomic backups with restic and btrfs

Published on

I want to backup a disk on my home server used by several databases and background synchronization tools. The files keep changing in the background all the time and blindly copying the files would lead to inconsistent data. Restic, one of the backup tools I use, does not guarantee backup consistency by itself. That's why I paired it up with btrfs snapshots to get a consistent view of the file system.

The problem definition

I have a btrfs filesystem using the flat subvolume layout (the root subvolume is used only for other subvolumes). The root of my filesystem is mounted at /mnt/biggadisk.root and all important data reside in a subvolume called @biggadisk mounted at /mnt/biggadisk.

I want to:

  • backup the /mnt/biggadisk directory with restic
  • backup an atomic snapshot of the file system
  • restic has to see the file at /mnt/biggadisk, not a different directory

In general, the same problem can be found discussed on restic forums here.

The solution

The first part of the solution is that we need to create a snapshot before backing up the disk. However, just by itself, the snapshot would be at a different location and the backup tool would see that. This is where mount namespaces can save us. We can run the backup process in a separate mount namespace and mount the correct snapshot to the expected directory. So the backup would run roughly like this:

  1. clone the current mount namespace and start the backup shell script in it
  2. create a btrfs snapshot (and delete a stale one if present)
  3. remount the backed-up directory with the snapshot
  4. run restic
  5. when all of this finishes, the mount namespace will be removed automatically

My implementation

I am using systemd to start the backup, so creating a private mount namespace is just a matter of a single configuration directive in the .service unit.

[Unit]
Description=Restic backup service
After=mnt-biggadisk.mount
Requires=mnt-biggadisk.mount

[Service]
Type=oneshot
# this creates a new mount namespace, so we can remount things safely
PrivateMounts=true

EnvironmentFile=/etc/restic.env
ExecStart=bash /etc/restic-backup.sh

The backup script looks like this:

#/bin/bash
set -Eeuxo pipefail

CACHE="--cache-dir /mnt/biggadisk.root/@cache/restic"

# clean old snapshot
if btrfs subvolume delete /mnt/biggadisk.root/@restic-snapshot; then
    echo "WARNING: previous run did not cleanly finish, removing old snapshot"
fi


# create a snapshot
SNAPSHOT_NAME="@restic-snapshot"
trap "btrfs subvolume delete /mnt/biggadisk.root/$SNAPSHOT_NAME" EXIT
btrfs subvolume snapshot -r /mnt/biggadisk.root/@biggadisk /mnt/biggadisk.root/$SNAPSHOT_NAME

# we should be running in a private mount namespace, so remounting things should be safe
# link the snapshot to /mnt/biggadisk, so that it appears normal
umount /mnt/biggadisk
mount -t btrfs -o subvol=$SNAPSHOT_NAME /dev/mapper/biggadisk /mnt/biggadisk

# run the actual backup
restic $CACHE backup --verbose --one-file-system --tag systemd.timer $BACKUP_EXCLUDES $BACKUP_PATHS
restic $CACHE forget --prune --max-unused 5G --verbose --tag systemd.timer --group-by "paths,tags" --keep-daily $RETENTION_DAYS --keep-weekly $RETENTION_WEEKS --keep-monthly $RETENTION_MONTHS --keep-yearly $RETENTION_YEARS

The environment file like this:

BACKUP_PATHS=/mnt/biggadisk
BACKUP_EXCLUDES="--exclude-file /mnt/biggadisk/.restic_exclude --exclude-if-present .resticignore"
RETENTION_DAYS=7
RETENTION_WEEKS=4
RETENTION_MONTHS=6
RETENTION_YEARS=2

B2_ACCOUNT_ID=<redacted>
B2_ACCOUNT_KEY=<redacted>
RESTIC_REPOSITORY=<redacted>
RESTIC_PASSWORD=<redacted>

All started by this timer unit:

[Unit]
Description=Backup with restic daily
After=mnt-biggadisk.mount
Requires=mnt-biggadisk.mount

[Timer]
OnCalendar=*-*-* 03:00:00

[Install]
WantedBy=timers.target
Thoughts? Leave a comment