Question about TEST command vs. Backup with --full-remote-verification and backup-test-samples

Support
1

In a different thread I found a command how to test / verify a backup

"c:\Program Files\Duplicati 2\Duplicati.CommandLine.exe"
test
target
all  
--full-remote-verification=true
--dbpath=... 
--passphrase=xxxxxxx
--disable-module=console-password-input
--console-log-level=profiling
--console-log-filter=-*.Database.*

My question is: Would there be a difference to make a backup command with the additional arguments

--full-remote-verification=true
--backup-test-samples="999999"

?

As I understand the backup would be done and after that the full backup would be downloaded (assuming that it has less than 999.999 files) and tested against the local database. Is that correct?

Is there any difference to the verification done with the TEST-Command?

Background: I use a different method for scheduling my duplicati backups and I want to make a FULL TEST once a month. The easiest way would be just to add the backup-test-samples with a very high number the first time a job runs in a month. That would be easier as making a new job for testing…

1 Like
2

No. The test performed at the end of each backup is actually calling the exact same code as the TEST command, but limited in scope to avoid large overheads.

The test command is just intended for running verifications outside of the normal backup flow.

1 Like
3

I tried it and get different results?

A backup on google drive. Added to backup-Command to test 999.999 files and full-remote-verification and got NO ERROR.

I run the command line with same backup and the TEST command and got different result. Backup says OK and TEST shows Errors?

How could that be if it is the same?

Backup:

20181126 17:39:02.330 DUPLICATI   Starting job google-01
20181126 17:39:02.598 DUPLICATI   C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe
20181126 17:39:02.798 DUPLICATI   backup "googledrive://duplicati?authid=xxxxx" "\\xxxx\xxxx\xxx\xxxx\\" --upload-verification-file="true" --backup-test-samples="999999" --backup-name="google-01" --dbpath="C:\jobdbs\google__rescue.sqlite" --encryption-module="aes" --compression-module="zip" --dblock-size="25MB" --passphrase="xxxx!" --allow-missing-source="true" --full-remote-verification="true" --disable-module="console-password-input"
20181126 17:39:04.888 DUPLICATI   Backup started at 26.11.2018 17:39:04
20181126 17:39:08.725 DUPLICATI   Checking remote backup ...
20181126 17:39:08.743 DUPLICATI     Listing remote folder ...
20181126 17:39:34.496 DUPLICATI   Scanning local files ...
20181126 17:39:34.523 DUPLICATI     938 files need to be examined (217,28 MB)
20181126 17:39:39.538 DUPLICATI     623 files need to be examined (171,23 MB)
20181126 17:39:44.553 DUPLICATI     436 files need to be examined (110,68 MB)
20181126 17:39:49.753 DUPLICATI     364 files need to be examined (86,26 MB)
20181126 17:39:54.774 DUPLICATI     268 files need to be examined (71,47 MB)
20181126 17:39:59.789 DUPLICATI     136 files need to be examined (27,48 MB)
20181126 17:40:04.789 DUPLICATI     26 files need to be examined (10,25 MB)
20181126 17:40:05.876 DUPLICATI     0 files need to be examined (0 Bytes)
20181126 17:40:06.226 DUPLICATI     Uploading file (65,53 KB) ...
20181126 17:40:08.916 DUPLICATI     Uploading file (5,58 KB) ...
20181126 17:40:10.444 DUPLICATI     Uploading file (103,31 KB) ...
20181126 17:40:13.964 DUPLICATI   Compacting remote backup ...
20181126 17:40:22.119 DUPLICATI   Compacting not required
20181126 17:40:22.149 DUPLICATI     Uploading file (245,39 KB) ...
20181126 17:41:06.580 DUPLICATI   Checking remote backup ...
20181126 17:41:06.592 DUPLICATI     Listing remote folder ...
20181126 17:41:32.880 DUPLICATI   Verifying remote backup ...
20181126 17:41:32.948 DUPLICATI   Remote backup verification completed
20181126 17:41:32.963 DUPLICATI     Downloading file (103,31 KB) ...
20181126 17:42:03.318 DUPLICATI     Downloading file (55,75 KB) ...
20181126 17:42:04.366 DUPLICATI     Downloading file (55,34 KB) ...

REMOVED OVER 1000 LINES WITH Downloads

20181126 18:19:52.620 DUPLICATI     Downloading file (25,06 MB) ...
20181126 18:20:13.731 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:20:31.091 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:20:48.106 DUPLICATI     Downloading file (25,00 MB) ...
20181126 18:21:04.791 DUPLICATI     Downloading file (25,00 MB) ...
20181126 18:21:19.222 DUPLICATI     Downloading file (25,08 MB) ...
20181126 18:21:34.084 DUPLICATI     Downloading file (25,07 MB) ...
20181126 18:21:49.247 DUPLICATI     Downloading file (25,08 MB) ...
20181126 18:22:18.235 DUPLICATI     Downloading file (25,08 MB) ...
20181126 18:22:43.430 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:23:01.651 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:23:18.496 DUPLICATI     Downloading file (25,02 MB) ...
20181126 18:23:35.791 DUPLICATI     Downloading file (25,04 MB) ...
20181126 18:24:01.702 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:24:25.237 DUPLICATI     Downloading file (402,70 KB) ...
20181126 18:24:28.205 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:24:48.702 DUPLICATI     Downloading file (25,04 MB) ...
20181126 18:25:07.008 DUPLICATI     Downloading file (25,03 MB) ...
20181126 18:25:25.493 DUPLICATI     Downloading file (25,04 MB) ...
20181126 18:25:54.141 DUPLICATI     Downloading file (123,25 KB) ...
20181126 18:25:55.178 DUPLICATI     Downloading file (25,05 MB) ...
20181126 18:26:19.349 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:26:37.044 DUPLICATI     Downloading file (25,02 MB) ...
20181126 18:26:54.142 DUPLICATI     Downloading file (25,00 MB) ...
20181126 18:27:09.864 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:27:24.420 DUPLICATI     Downloading file (25,07 MB) ...
20181126 18:27:41.577 DUPLICATI     Downloading file (1,18 MB) ...
20181126 18:27:42.992 DUPLICATI     Downloading file (25,04 MB) ...
20181126 18:27:58.750 DUPLICATI     Downloading file (25,05 MB) ...
20181126 18:28:15.023 DUPLICATI     Downloading file (25,05 MB) ...
20181126 18:28:41.996 DUPLICATI     Downloading file (25,01 MB) ...
20181126 18:28:57.696 DUPLICATI     Downloading file (25,04 MB) ...
20181126 18:29:14.421 DUPLICATI     Downloading file (89,08 KB) ...
20181126 18:30:06.947 DUPLICATI     Duration of backup: 00:50:55
20181126 18:30:06.957 DUPLICATI     Remote files: 1375
20181126 18:30:06.970 DUPLICATI     Remote size: 972,07 MB
20181126 18:30:06.982 DUPLICATI     Files added: 0
20181126 18:30:06.992 DUPLICATI     Files deleted: 0
20181126 18:30:07.002 DUPLICATI     Files changed: 2
20181126 18:30:07.012 DUPLICATI     Data uploaded: 419,81 KB
20181126 18:30:07.025 DUPLICATI     Data downloaded: 972,07 MB
20181126 18:30:07.035 DUPLICATI   Backup completed successfully!

Test with

“c:\Program Files\Duplicati 2\Duplicati.CommandLine.exe” test googledrive://duplicati?authid=xxx all --dbpath=C:\jobdbs\google__rescue.sqlite --passphrase=xxxxx --disable-module=console-password-input --console-log-level=profiling --console-log-filter=-.Database. --full-remote-verification=true

Die angegebene Option --console-log-level ist nicht unterstĂĽtzt und wird daher ignoriert
Die angegebene Option --console-log-filter ist nicht unterstĂĽtzt und wird daher ignoriert
Listing remote folder ...
  Downloading file (55,65 KB) ...
  Downloading file (55,64 KB) ...
  Downloading file (55,64 KB) ...
  Downloading file (55,86 KB) ...
  Downloading file (55,81 KB) ...
  Downloading file (55,90 KB) ...
  Downloading file (68,50 KB) ...
  Downloading file (55,64 KB) ...
  Downloading file (55,34 KB) ...
REMOVED OVER 1000 LINES OF DOWNLOADS

  Downloading file (25,08 MB) ...
  Downloading file (123,25 KB) ...
  Downloading file (402,70 KB) ...
  Downloading file (25,05 MB) ...
  Downloading file (25,03 MB) ...
  Downloading file (25,01 MB) ...
  Downloading file (25,02 MB) ...
  Downloading file (25,01 MB) ...
  Downloading file (1,18 MB) ...
  Downloading file (25,02 MB) ...
  Downloading file (25,03 MB) ...
  Downloading file (25,03 MB) ...
  Downloading file (25,08 MB) ...
  Downloading file (25,04 MB) ...
  Downloading file (25,04 MB) ...
  Downloading file (25,01 MB) ...
  Downloading file (25,05 MB) ...
  Downloading file (25,01 MB) ...
  Downloading file (25,04 MB) ...
  Downloading file (25,01 MB) ...
  Downloading file (89,08 KB) ...
  Downloading file (25,04 MB) ...
  Downloading file (65,53 KB) ...
  Downloading file (25,01 MB) ...
duplicati-i8aed06a421c2422c9810086e48588608.dindex.zip.aes: 62 errors
        Extra: +cksZ5KZGPj3/e5PoOPqcDuo96UP/zXIJoHSFwaipPk=
        Extra: +kL+CjMumO5cUd+tBWBgTy+EucOc+FyLE3mLzX62Ijw=
        Extra: +tS4Bge1FJaJR8MckGa8BjlpDWiDuaDVP5qrYWpanOY=
        Extra: /BMt0yPRpQkxUBfibbVCoPSA6/pytWgAm7yTzuNW7FU=
        Extra: /ffnIUXgEGJfhntnp7k1mrRFc5psClucAEvkYKeSND0=
        Extra: /mg+x7/S4Pkbyrzp2ElDwr10BOubibxLHUTztVTh9LM=
        Extra: 1aARUn+HMRyJE62XIQEacfD1VfH2XUygVOmrpPXR/4E=
        Extra: 2f8yZgKAwzAA+9rCRUBnON8JQdk8/s4NrGWHEgfdOUc=
        Extra: 3iGtjDaIYHrVu/zTHKaFvb66hS2uEA8nnasC0bxeaVc=
        Extra: 4a46Q4vLRV+o5YVwLpdk3fa3/3sv/pYvtMZYgVLqqtc=
        ... and 52 more

duplicati-bf6d0576479c941b0ae08302b26d4b71e.dblock.zip.aes: 62 errors
        Extra: +cksZ5KZGPj3/e5PoOPqcDuo96UP/zXIJoHSFwaipPk=
        Extra: +kL+CjMumO5cUd+tBWBgTy+EucOc+FyLE3mLzX62Ijw=
        Extra: +tS4Bge1FJaJR8MckGa8BjlpDWiDuaDVP5qrYWpanOY=
        Extra: /BMt0yPRpQkxUBfibbVCoPSA6/pytWgAm7yTzuNW7FU=
        Extra: /ffnIUXgEGJfhntnp7k1mrRFc5psClucAEvkYKeSND0=
        Extra: /mg+x7/S4Pkbyrzp2ElDwr10BOubibxLHUTztVTh9LM=
        Extra: 1aARUn+HMRyJE62XIQEacfD1VfH2XUygVOmrpPXR/4E=
        Extra: 2f8yZgKAwzAA+9rCRUBnON8JQdk8/s4NrGWHEgfdOUc=
        Extra: 3iGtjDaIYHrVu/zTHKaFvb66hS2uEA8nnasC0bxeaVc=
        Extra: 4a46Q4vLRV+o5YVwLpdk3fa3/3sv/pYvtMZYgVLqqtc=
        ... and 52 more
4

Good question! The flag --full-remote-verification triggers that all files are downloaded and the contents are checked.

I can see from the output that all files are downloaded, so the flag is activated correctly.

The error message indicates Extra contents, which means that it has more data than it is supposed to. This means that the local database and the remote files are out of sync. Most likely cause is that Duplicati has flagged the contents of the files to be deleted, thus it finds no contents for the files, but the files are not empty.

I can only guess that this deletion has happened after the backup, but before the test ?

5

There were some posts recently on what --full-remote-verification actually does, and the documentation says it looks deeper, opening up the files instead of just checking hashes. It’s not a replacement for high test samples. Samples, by the way, also accepts all or everything, if you don’t want to make up your own very-high value.

Discussion is in several posts around Backup issues following NAS crash - Hash mismatch and I just tested too, using samples of 1 along with --full-remote-verification. I got only one sample-set downloaded, watching the log.

6

No. Because now (just 1 backup later) the backup also does not show this Extras-Error?

The backup-command also downloaded everything to test it but it does not recognize this extras and the test does?

...
20181126 23:05:05.494 DUPLICATI     Downloading file (25,08 MB) ...
20181126 23:05:20.984 DUPLICATI     Downloading file (89,08 KB) ...
20181126 23:05:21.242 DUPLICATI     Downloading file (25,01 MB) ...
20181126 23:05:36.949 DUPLICATI     Downloading file (65,53 KB) ...
20181126 23:05:37.449 DUPLICATI     Downloading file (25,04 MB) ...
20181126 23:06:40.610 DUPLICATI     Duration of backup: 00:36:08
20181126 23:06:40.630 DUPLICATI     Remote files: 1378
20181126 23:06:40.648 DUPLICATI     Remote size: 972,31 MB
20181126 23:06:40.665 DUPLICATI     Files added: 0
20181126 23:06:40.680 DUPLICATI     Files deleted: 0
20181126 23:06:40.698 DUPLICATI     Files changed: 4
20181126 23:06:40.728 DUPLICATI     Data uploaded: 486,93 KB
20181126 23:06:40.743 DUPLICATI     Data downloaded: 972,31 MB
20181126 23:06:40.750 DUPLICATI   Backup completed successfully!
7

But the initial question was: Is there a difference between backup with --full-remote-verification and --backup-test-samples=high value (if everything or all or 999999 is not important for this case but thank you for the hint with all!) versus the test command.

Answer was “No. There is no difference” but know the backup states success an no error and the test shows errors and the next backup show a success again.

So I am a little bit confused.

8

Again some problems with a backup were I get sometimes errors and I am not sure if the problem is box.com or not. So I downloaded the complete backup and now I want to make a FULL test.

Now my question: How can I make a really full consistent test with everything Duplicati can test!

Which tests can duplicati make?
How to make all this possible tests on all files.

I tried the “TEST” command with the switches --full-remote-verification=“true” and --backup-test-samples=999999 but always only 3 files are tested?

Full commad
test “file://\path…” --backup-test-samples=“999999” --full-remote-verification=“true” --backup-name=“the name” --dbpath=“C:\Users\xxxx\AppData\Local\Duplicati\jobdbs\xxxxx.sqlite” --encryption-module=“aes” --compression-module=“zip” --dblock-size=“100MB” --passphrase=“abcdef” --disable-module=“console-password-input”

9

When run standalone (as opposed to as part of a backup), the test command syntax (see prior link) is:

Duplicati.CommandLine.exe test <storage-URL> <samples> [<options>]

but you left off the samples value (–backup-test-samples is for backup) so it defaulted to 1 set of three.

See if removing --backup-test-samples= and just leaving 999999 helps. If so, all should also work.