welcome to the forum
Hi @rafficer, welcome to the forum!
I’m glad to hear you like Duplicati, but sorry to hear you’re frustrated be the command-line interaction.
There is a --thread-priority parameter you can use to have Duplicati be more or less polite with the CPU. And the --throttle-download and --throttle-upload parameters can limit bandwidth use.
However there are not currently any disk usage limiters so temp file processing and local disk / USB access will always be max available. Note that temp file disk access can be mitigated a bit by using a RAM drive.
As far as command-line progress notification, the problem you’re running into is that Duplicati.CommandLine.exe passes actual execution off to whatever Duplicati server (service or Tray Icon) you’re running.
So you’d need to pull the server for status info, which is what the web GUI does. Unfortunately. I don’t know how to do that from the command line. 
How do you perform a full verification? By doing a restore of all files to a temporary location? Or by using the test command and supplying the number of .dblock files to --backup-test-samples? Can you provide an obfuscated version of the command you use for verification?
The commandline tools do not return detailed information when running. You can try to use the --verbose option to get more output.
That’s not exactly how it works. The commandline tools are fully independent from the server/service/trayicon. You even don’t have to install/start the server at all to use the commandline tools.
The server part (that’s also included in the Tray Icon tool) is nothing more or less than a webserver and scheduler to control, schedule and start backup tasks.
Information about backend files is stored in an SQLite database. If the commandline backup points to the same SQLite database as the task in the Web UI, you can restore backups from the Web UI that are made with the commandline and vice versa.
Whoops! Thanks for setting me straight on that! 
So the server (either type) just runs the web interface and schedules / kicks off command line calls as appropriate?
In that case I assume the web UI periodically queries the server about “what’s happening” and the server directly (or via command line call) pulls entries from one of the .sqlite DB logs to be fed back for the GUI to display?
Yes, that’s about how it works. As far as I know, the commandline tools itself are not called by the server, but the same code is executed when a backup runs.
If you change the command to test from a to commandline exported backup configuration, you’re not doing a full verification.
Basically, you supply all options used to make a backup to the test command. Some of them are required (URL, credentials, passphrase), but most of the options used for the backup are ignored by the test command.
The test command does exactly the same as verifying the backup after each backup operation is completed: it downloads a number of DBLOCK files (default 1) and checks the integrity and contents of that DBLOCK file. If the file looks as expected, the verification completes successfully.
So instead of a full verification of your backup, you probably let Duplicati download just one random DBLOCK file and let it check if the contents are what Duplicati expects.
If you want to test and be 100% sure that all files can be restored, the only option is doing a full restore. Another option is to supply the --backup-test-samples option to the test command and give it a value that is the same as the current number of DBLOCK files at the backend. This will force Duplicati to download all DBLOCK files and check the integrity of these files.
There is no real progress reporting for the check part. You can add --verbose to the command to see a bunch of internal messages that also reveal what it is doing.
You may also want to use --full-remote-verification as that will look inside the files and check the content, as opposed to just checking the external file hash (default).
If you can run code on the storage node, you can also use the option --upload-verification-file which will put a small .json file in the destination folder. There is a Python script that reads this file and does the (external) verification on the remote server.
2 Likes
Hashes are not guaranteed unique - the more bits in the hash (and the less in the item being hashed) combined with the type of hash, the less likely there will be a collision, but it can happen.
For example, if I had a hashing technique that replaced all duplicate letters with a single instance of that letter (e.g. “aaaab” → “ab”) then there’d be a collision if I was hashing both “aaaab” and “aaaaaaaaaaaab”.
Of course with real hashing algorithms, this is not very likely. In fact I believe in the last year or so Google proved that two different documents could produce the same SHA-1 hash - and it only took them 9,223,372,036,854,775,808 computations over 6,610 years of processor time to do it. 
If you’re interested in learning more, a web search of “hash collision” should get you started.
2 Likes
I can happen if the file is broken when being written (disk glitch, broken zip library, etc).
Yeah, and they deliberately tried to get it. And they got it by appending a lot of junk to the PDF file. And Duplicati uses SHA-256 by default.
It calls /api/v1/progressstate, and you can do the same from the Python script:
1 Like
Oh, absolutely - it just shows that it’s POSSIBLE, even if EXTREMELY unlikely to happen in normal use. That’s part of what makes SHA-256 so much of an improvement over SHA-1.
If somebody were trying to use it for security purposes, it could be dangerous to assume that matching SHA-1 hashes means matching content.
Assuming the downloads are the most expensive part (with might be incorrect) it would be cool if we could have the restore selector window include the number (or size) of archives needing to be downloaded for the selected files to be restored.
You should get some kind of error message along with the 400 code.
(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)
That must be a default Discourse setting to avoid spamming.