Use rsnapshot for backups

[nicojx]

I've used dietpi-backup to have two days of backups available. I was looking into setting up an additional remote backup and came across rsnapshot (https://rsnapshot.org/). This uses rsync and hard links to make space saving backups, and sets up daily, weekly, monthly or whatever copies. It's a neat setup, and I was wondering whether this should be the default for DietPi.

[MichaIng]

There are a lot of great backup tools out there, each with its pros and cons, depending on use case and needs. But I do not aim to replace dietpi-backup as minimal integrated solution, which aims to be quick, simple (little code, low complexity) and integrated in the first place. But we can make dietpi-backup better.

I had a look into their hard link usage, which is indeed clever for multiple backups, and easy to implement out end as well: https://manpages.debian.org/rsync#compare-dest=DIR
All we need to do is:

[[ -d $FP_TARGET/data_2 ]] && aRSYNC_RUN_OPTIONS_BACKUP+=("--link-dest=$FP_TARGET/data_2")

So if a file did not change between last and current backup iteration, the target file is created as hard link to the one in the last iteration. And this applies recursively over all iterations, i.e. if a file did not change at all between all daily backup iterations, they are all hard links to the same inode.

There is a notice at the --hard-links flag (which we use already): https://manpages.debian.org/rsync#hard-links

• If you specify a --link-dest directory that contains hard links, the linking of the destination files against the --link-dest files can cause some paths in the destination to become linked together due to the --link-dest associations.

Not such an easy sentence, and a little confusing in this context. But actually pretty natural and perfectly fine:

• Assume rsync --link-dest=/target_old /source /target
• Assume /target_old/file1 and /target_old/file2 are hard links to the same inode.
• Assume both files did not change since /target_old was created.
• So rsync now creates /target/file1 and /target/file2 as hard links to the respective files in /target_old.
• And since the two in /target_old are already hard links to the same inode, now all 4 files are hardlinks to the same inode.

Now I wonder why this is mentioned in to --hard-links flag section, even that this fact has nothing to do with it, at least not directly. I guess it is mentioned because of these two indirect relations:

• Normally, rsync creates each target file as distinct inode, and the --hard-links is the only more widely known way to have it creating hard links, and only if there are hard links between files in the source directory. In above case, hard links between files within the target directories are created even without --hard-links, or without the two files being hard linked in the current source. Hence I see this as reminder to avoid confusion and bug reports in case of unexpected hard links between files in the target dir.
• Additionally, hard links between files in /target_old are usually the result of --hard-links used in the first place, as multiple backup iterations are the common use case for --link-dest. So without using --hard-links, the above example case would never happen, but the only hard links would be between /target and /target_old, but not between files within each of them.

[MichaIng]

Done: 5bf0d2b
That was easy. Does not even need to be added conditionally, since rsync gracefully ignores it, if the directory does not exist.

Wow that works crazy well, and on my test VMs also the performance gain for the backups is massive.

Here effective disk usage, which represents the fully intact iterations, but overall disk usage of just a single backup + meta data:

root@VM-Trixie:/mnt/dietpi-backup# G_TREESIZE data_3
447.5 MiB data_3
403.5 MiB data_3/usr
26.5 MiB data_3/boot
13.4 MiB data_3/var
2.1 MiB data_3/etc
48.0 KiB data_3/root
24.0 KiB data_3/mnt
20.0 KiB data_3/home
4.0 KiB data_3/srv
4.0 KiB data_3/opt
root@VM-Trixie:/mnt/dietpi-backup# G_TREESIZE data_2
447.5 MiB data_2
403.5 MiB data_2/usr
26.5 MiB data_2/boot
13.4 MiB data_2/var
2.1 MiB data_2/etc
48.0 KiB data_2/root
24.0 KiB data_2/mnt
20.0 KiB data_2/home
4.0 KiB data_2/srv
4.0 KiB data_2/opt
root@VM-Trixie:/mnt/dietpi-backup# G_TREESIZE data
445.6 MiB data
403.5 MiB data/usr
26.5 MiB data/boot
13.4 MiB data/var
2.1 MiB data/etc
48.0 KiB data/root
24.0 KiB data/mnt
20.0 KiB data/home
4.0 KiB data/srv
4.0 KiB data/opt
root@VM-Trixie:/mnt/dietpi-backup# G_TREESIZE
465.5 MiB .
447.5 MiB ./data_2
9.0 MiB ./data_3
9.0 MiB ./data

And here the time savings, doing a very first/fresh backup, and then a second one without significant file changes in between:

root@VM-Trixie:/mnt/dietpi-backup# export G_INTERACTIVE=0; time dietpi-backup 1

 DietPi-Backup
─────────────────────────────────────────────────────
 Mode: Backup

[ INFO ] DietPi-Backup | Rotating backup archive
[  OK  ] DietPi-Backup | mkdir -p /mnt/dietpi-backup/data
[ SUB1 ] DietPi-Services > stop
[  OK  ] DietPi-Services | stop : cron
[  OK  ] DietPi-Services | stop : raspotify
[  OK  ] DietPi-Services | stop : avahi-daemon
[ INFO ] DietPi-Backup | Checking for required APT packages: rsync
[ INFO ] DietPi-Backup | Backup to /mnt/dietpi-backup in progress, please wait...
sending incremental file list
--link-dest arg does not exist: ../data_2
    422,505,400  99%   37.32MB/s    0:00:10 (xfr#14625, to-chk=0/18725)

sent 423,710,921 bytes  received 297,339 bytes  36,870,283.48 bytes/sec
total size is 426,475,183  speedup is 1.01
[  OK  ] DietPi-Backup | cp -a /etc/debian_version /mnt/dietpi-backup/data/etc/debian_version
[  OK  ] DietPi-Backup | touch /mnt/dietpi-backup/data
[ SUB1 ] DietPi-Services > start
[  OK  ] DietPi-Services | start : avahi-daemon
[  OK  ] DietPi-Services | start : raspotify
[  OK  ] DietPi-Services | start : cron
[  OK  ] DietPi-Backup: Backup | Completed
[ INFO ] DietPi-Backup | Backup completed:
 - /mnt/dietpi-backup

real    0m11.651s
user    0m2.167s
sys     0m12.543s
root@VM-Trixie:/mnt/dietpi-backup# export G_INTERACTIVE=0; time dietpi-backup 1

 DietPi-Backup
─────────────────────────────────────────────────────
 Mode: Backup

[ INFO ] DietPi-Backup | Rotating backup archive
[  OK  ] DietPi-Backup | mv /mnt/dietpi-backup/data /mnt/dietpi-backup/data_2
[  OK  ] DietPi-Backup | mkdir -p /mnt/dietpi-backup/data
[ SUB1 ] DietPi-Services > stop
[  OK  ] DietPi-Services | stop : cron
[  OK  ] DietPi-Services | stop : raspotify
[  OK  ] DietPi-Services | stop : avahi-daemon
[ INFO ] DietPi-Backup | Checking for required APT packages: rsync
[ INFO ] DietPi-Backup | Backup to /mnt/dietpi-backup in progress, please wait...
sending incremental file list
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/18725)

sent 482,642 bytes  received 2,391 bytes  194,013.20 bytes/sec
total size is 426,475,183  speedup is 879.27
[  OK  ] DietPi-Backup | cp -a /etc/debian_version /mnt/dietpi-backup/data/etc/debian_version
[  OK  ] DietPi-Backup | touch /mnt/dietpi-backup/data
[ SUB1 ] DietPi-Services > start
[  OK  ] DietPi-Services | start : avahi-daemon
[  OK  ] DietPi-Services | start : raspotify
[  OK  ] DietPi-Services | start : cron
[  OK  ] DietPi-Backup: Backup | Completed
[ INFO ] DietPi-Backup | Backup completed:
 - /mnt/dietpi-backup

real    0m2.371s
user    0m0.605s
sys     0m1.622s

11.6s vs 2.3s, with all dietpi-backup wrapping processing included (stopping/restarting services took a while). Of course this depends on the read vs write speeds of the filesystem and drive as well. But it shows that the comparison between source and old/comparison target is done pretty effectively, probably using some metadata hashes? Not sure how it is done technically, but impressive.

EDIT: Hmm, with this, there is actually not much point to use the oldest obsolete backup iteration as basis for the first, as it will have a larger difference to the current system then the last backup. But it does not cause much issues either. Removing it instead of moving it, and having rsync doing the removal of changed inodes does not make any difference, I guess.