Best way to verify backup after each incremental run

Maybe not, but the game plan may change:

Duplicati.CommandLine.RecoveryTool.exe

This tool can be used in very specific situations, where you have to restore data from a corrupted backup. The procedure for recovering from this scenario is covered in Disaster Recovery.

The tool can’t perform miracles, but it’s less picky about things and can work around damaged spots.

A single sample set can fall behind creation of new ones. A percentage might keep up, but it’s hard to figure out exactly when each individual file has been verified. That might be a wasted exercise though.

This level of verification is mainly looking for corrupted files at the destination. Some are more reliable.
At a less intensive level than file downloading, file listings are done by default to see if files look “right”.
At a more intensive level on top of downloading, you can have Duplicati examine internal file structure.
The TEST command describes full-remote-verification which is prone to being falsely alarming.
The documentation answers an asked question with “A sample consists of 1 dlist, 1 dindex, 1 dblock.”.
Unfortunately this begs the question of what happens if a category runs short? I think once you get all, that’s all for the type, meaning it won’t download twice, but you might wind up with less than three files.
Backup Test block selection logic explains in more detail, but I’m not sure hyper-focus is the right path.
What is the destination and the connection to it, and is there any particular reliability concern with this?
While trying to test a backup regularly is great, keeping multiple backups done differently is even safer.

Some help text on the question (also in the manual and GUI):

  --backup-test-samples (Integer): The number of samples to test after a
    backup
    After a backup is completed, some (dblock, dindex, dlist) files from the
    remote backend are selected for verification. Use this option to change
    how many. If the backup-test-percentage option is also provided, the
    number of samples tested is the maximum implied by the two options. If
    this value is set to 0 or the option --no-backend-verification is set, no
    remote files are verified
    * default value: 1

  --backup-test-percentage (Integer): The percentage of samples to test after
    a backup
    After a backup is completed, some (dblock, dindex, dlist) files from the
    remote backend are selected for verification. Use this option to specify
    the percentage (between 0 and 100) of files to test. If the
    backup-test-samples option is also provided, the number of samples tested
    is the maximum implied by the two options. If the no-backend-verification
    option is provided, no remote files are verified.
    * default value: 0

I mentioned multiple backups (taking more time and expense…) for anything you’d really hate to lose. Maybe a more reliable backup program should be one of them. Duplicati is still in Beta and imperfect.
Don’t ask me which tool is perfect though. I think they’re all just varied levels, which is why two helps.

Good practices for well-maintained backups has a bunch of ideas (including actually testing restores).

If you’re determined to test-to-the-near-max automatically, you could probably do a database recreate following every backup. Probably do it to a test database instead of your real, to avoid failure damage.
I happen to run heavy instrumentation because I want data to debug a bug if my Duplicati ever breaks.
I’m doing both a DB recreate and a test with all and --full-remote-verification, Rather heavy. Because the backup is on B2 cloud storage, I don’t really want to download it all (slow and costly), so I rclone sync it down (fast and cheap) to the local drive, where Duplicati can also read it nice and fast…

Your backup is much bigger. I don’t know where it is or how fast it can download (or if download costs). Keeping it on (for example) a local USB drive is pretty fast, but it’s subject to local loss just like system.
If you’re able to get a default 50 MB dblock file (and more) in a few seconds, maybe you have this peril, however you won’t need to download the backup to local disk for several checks, as it’s already on one.

As a side note on recreate (and other operation) speed (or not), a 1 TB backup for good speeds needs blocksize of 1 MB, increased from default 100 KB. Without this, tracking 10 million blocks can get slow. Unfortunately blocksize can’t be changed once a backup is going. This needs to be chosen in advance.

It’s not clear if this is a disaster recovery, but those are when database rebuild can give nasty surprises previously not seen because you were doing ordinary restores that use the pre-existing local database. Hyper-focus on the sampled file-corruption test will miss possible logical issues that recovery might hit.

1 Like