Best way to verify backup after each incremental run

Dear Duplicati users,

I am running for several months an incremental encrypted Duplicati backup with a total size of about 1TB. My nightmare is, that at a certain point of time, I really need a recovery and the backup is somehow corrupted => Game over.

I know, that after each run, by default, Duplicati chooses a random snippet for verifcation backend files. This takes usually a few seconds. This will never do a full-verification even not with multiple runs.

I am looking now for a better verification method, which is a good trade-off between time consumption and verification. I would accept about 1-2 hours per run for verification process, that hopefully over time the whole backup is verified continously.

What would be your recommendation for options? From my understanding “backup-test-percentage” is exactly what I need? The higher the value, the longer it takes, and the lower the risk after serveral runs, that there are still some unchecked blocks?

I don’t really understand the cross-connection to “backup-test-samples”?

What does it for example mean, if I use:

  • backup-test-percentage = 10
  • backup-test-samples = 20

Could somebody explain? What is “one sample”? One of the zip-files according to my filesize-setting? Which of the two settings overrule? Minimum? Maximum?

Do you agree with my approach, or do you have a better idea?

Thank you!

Maybe not, but the game plan may change:

Duplicati.CommandLine.RecoveryTool.exe

This tool can be used in very specific situations, where you have to restore data from a corrupted backup. The procedure for recovering from this scenario is covered in Disaster Recovery.

The tool can’t perform miracles, but it’s less picky about things and can work around damaged spots.

A single sample set can fall behind creation of new ones. A percentage might keep up, but it’s hard to figure out exactly when each individual file has been verified. That might be a wasted exercise though.

This level of verification is mainly looking for corrupted files at the destination. Some are more reliable.
At a less intensive level than file downloading, file listings are done by default to see if files look “right”.
At a more intensive level on top of downloading, you can have Duplicati examine internal file structure.
The TEST command describes full-remote-verification which is prone to being falsely alarming.
The documentation answers an asked question with “A sample consists of 1 dlist, 1 dindex, 1 dblock.”.
Unfortunately this begs the question of what happens if a category runs short? I think once you get all, that’s all for the type, meaning it won’t download twice, but you might wind up with less than three files.
Backup Test block selection logic explains in more detail, but I’m not sure hyper-focus is the right path.
What is the destination and the connection to it, and is there any particular reliability concern with this?
While trying to test a backup regularly is great, keeping multiple backups done differently is even safer.

Some help text on the question (also in the manual and GUI):

  --backup-test-samples (Integer): The number of samples to test after a
    backup
    After a backup is completed, some (dblock, dindex, dlist) files from the
    remote backend are selected for verification. Use this option to change
    how many. If the backup-test-percentage option is also provided, the
    number of samples tested is the maximum implied by the two options. If
    this value is set to 0 or the option --no-backend-verification is set, no
    remote files are verified
    * default value: 1

  --backup-test-percentage (Integer): The percentage of samples to test after
    a backup
    After a backup is completed, some (dblock, dindex, dlist) files from the
    remote backend are selected for verification. Use this option to specify
    the percentage (between 0 and 100) of files to test. If the
    backup-test-samples option is also provided, the number of samples tested
    is the maximum implied by the two options. If the no-backend-verification
    option is provided, no remote files are verified.
    * default value: 0

I mentioned multiple backups (taking more time and expense…) for anything you’d really hate to lose. Maybe a more reliable backup program should be one of them. Duplicati is still in Beta and imperfect.
Don’t ask me which tool is perfect though. I think they’re all just varied levels, which is why two helps.

Good practices for well-maintained backups has a bunch of ideas (including actually testing restores).

If you’re determined to test-to-the-near-max automatically, you could probably do a database recreate following every backup. Probably do it to a test database instead of your real, to avoid failure damage.
I happen to run heavy instrumentation because I want data to debug a bug if my Duplicati ever breaks.
I’m doing both a DB recreate and a test with all and --full-remote-verification, Rather heavy. Because the backup is on B2 cloud storage, I don’t really want to download it all (slow and costly), so I rclone sync it down (fast and cheap) to the local drive, where Duplicati can also read it nice and fast…

Your backup is much bigger. I don’t know where it is or how fast it can download (or if download costs). Keeping it on (for example) a local USB drive is pretty fast, but it’s subject to local loss just like system.
If you’re able to get a default 50 MB dblock file (and more) in a few seconds, maybe you have this peril, however you won’t need to download the backup to local disk for several checks, as it’s already on one.

As a side note on recreate (and other operation) speed (or not), a 1 TB backup for good speeds needs blocksize of 1 MB, increased from default 100 KB. Without this, tracking 10 million blocks can get slow. Unfortunately blocksize can’t be changed once a backup is going. This needs to be chosen in advance.

It’s not clear if this is a disaster recovery, but those are when database rebuild can give nasty surprises previously not seen because you were doing ordinary restores that use the pre-existing local database. Hyper-focus on the sampled file-corruption test will miss possible logical issues that recovery might hit.

1 Like