Good practices for well-maintained backups

I’ve been testing both Duplicity and Duplicati for a while and decided to stick with the latter.

For duplicity, there are some recommendations in order to keep reliable backups (specially the remote ones), e.g. perform a full backup from time to time, verify the snapshots regularly, do not keep too much incremental versions between full backups, and so on.

My question is: Is there any such recommendations for Duplicati? Specifically: in an extreme case in which one performs an incremental backup, say, each hour that would be a bad practice? This is probably not recommended for Duplicity I guess , so I’m wondering how Duplicati goes on this matter.

Any feedback is greatly appreciated.

Welcome to the forum @abarros

There’s not anything just like that because the full/incremental split doesn’t exist in Duplicati 2.

Duplicati old site (going back to Duplicati 1) says previous Duplicati and duplicity were similar.
Does Block-based storage engine Background section on Duplicati 1 remind you of duplicity?
New Storage Format on Duplicati 2 says “There is no need to upload full backups regularly.”

That’s exactly how often I do backups, but I keep versions thinned with a retention policy option.
Letting versions grow to infinity is going to be larger and slower. Set a default or custom setting.

image

Choosing sizes in Duplicati has some other advice. In addition, large backup (maybe over 100 GB) performance slows at default 100 KB blocksize. Raise blocksize to stay below a few million blocks.

Other good practices are to test restores (ideally by Direct restore from backup files or at least with
no-local-blocks), keep at least one backup offsite, don’t trust valuable data to any single backup, etc.

I’m reluctant to get too far from the primary questions without knowing that you’re interested in more.

1 Like

@ts678 , thank you very much for such a detailed answer. Your post clarified all my points. Indeed I should have studied Duplicati’s documentation better before posting in here. Lesson learned. That bit about blocksize was particularly useful. Finally, I’m interested in more information so please go ahead.

Topic could use input from other people, but I’ll start with maintenance one could do on local Database.

VACUUM - performance improvement is huge is a post on this. Results may vary. Size may decrease.

Usage: vacuum <storage-URL> [<options>]

  Rebuilds the local database, repacking it into a minimal amount of disk
  space.

The storage URL can come from Export As Command-line for true CLI, or you can use Commandline.

You can use auto-vacuum with auto-vacuum-interval if you like vacuum. It’s not really a must-run thing.

Database loss (e.g. from a disaster) and Direct restore from backup files will make a partial temporary database from destination files, and testing helps ensure that still works. A more thorough (and slower) testing method is to occasionally use the Database Recreate button, and maybe plan on waiting awhile.
Looking at About → Show log → Live → Verbose will detail where you are, and demonstrate progresss.

Somewhat safer is to move the database to a backup name, and use the Repair button. This gives you simple recovery if something goes wrong. It won’t change the destination, so old database will still work, however you never want to be using a database obtained somehow (e.g. backup) not fitting destination.

Ordinarily rebuilding the database should get to about 70% on the progress bar. Any further can indicate trouble finding data. 90% to 100% is downloading the rest of the backup, and it’s usually not successful.

Checking on destination file health is good, but sometimes the default verification sample is sort of small.
Large backups or ones with destination trouble can set backup-test-samples or backup-test-percentage.
Corrupted files are hard to predict, but transfer error rates show up in job Complete log RetryAttempts which is intended to ride through some isolated issues (subject to the number-of-retries and retry-delay).

The TEST command can be used in some idle period, if you’d rather not slow down your backup cycling.

Unlike the job database, which can be rebuilt from destination data (except for logs), the server database (Duplicati-server.sqlite) should be protected by Export To File, then keep that safe. It’s not needed for the restore, but will save some trouble configuring that backup again on a replacement system after disaster.

If nothing else, make sure you save enough destination and passphrase information to Direct restore.

Reporting options are good. There are third-party monitors that build on them, to let you keep track easily.
Duplicati Monitoring and dupReport are examples, but people on the forum use all kinds of things for this.

There are loads of options (some probably not much-used therefore not well-proven), if you’re into those, however I’m trying to stay close to “well-maintained” topic including a bit on preparing for disaster restore.

EDIT:

Above “options” reference had Advanced Options in mind for those who like to tune, but in general there’s “good enough” versus levels aimed at extra care or proven needs. This depends on you and your system. Some people probably don’t do any of this speed-helping and extra-testing and all has worked out – so far.

2 Likes

I literally just created a user account here so that I could say thank you for your incredibly helpful and well written replies to this thread. You have set an example for how to promote the ideals of the FOSS community by freely contributing a wealth of knowledge toward the collective benefit and success of every user.

I really appreciate very much the time and effort you took to provide such an insightful response to the OP’s query. I will definitely begin testing some of the methods and solutions you mentioned, and if I encounter anything that might add something to this thread, I will post back with my experience.

2 Likes

Welcome to the forum @nativeit and I think you’ve very well said the spirit of the community effort.
Thanks for giving the appreciation. There’s loads of room for volunteers, and your voice is welcome.

Everybody likely has something to contribute, and a lot of people have specialized skills that are rare.
Even heavy Duplicati volunteers can’t get all skills and systems, so community help really does help.
Even for less difficult scenarios, people adding what they can helps free up others, and it can be fun.