Test handler errors

Test handler has started erroring on almost ALL of my backup jobs that until 3 days ago showed no errors. As far as I know, nothing has changed on the systems in question (if a windows update had caused this, I’m guessing there would be a lot more people here complaining about it. This makes for a very large concern on my part that this would suddenly popup. I’m using nextcloud and webdav.

A typical error is this:
2021-01-15 15:47:15 -08 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b479240afc12c4eb081455b4ad8109f9e.dblock.zip.aes
Duplicati.Library.Main.BackendManager+HashMismatchException: Hash mismatch on file “C:\WINDOWS\TEMP\dup-17ddaaa7-a7eb-456b-90c3-98b50f4d108e”, recorded hash: 5S2NvT7Z4HDMiCJP1/n2mxYam7mAqvb1XId8nsQyobI=, actual hash efY9TCU2YJ7NLzOLSZoHEc69kclVB2JrBbwy9E/yeU0=
at Duplicati.Library.Main.BackendManager.GetForTesting(String remotename, Int64 size, String hash)
at Duplicati.Library.Main.Operation.TestHandler.DoRun(Int64 samples, LocalTestDatabase db, BackendManager backend)

Please tell me how to gather the additional info you might need to get this working again.
Thanks

update: I also get a test handler error with ‘file length invalid’. I would be willing to blame my nextcloud server, but maddeningly, some backup jobs work at the exact same time others are failing.
On most of my backup jobs I have log set to error instead of info and discovered by accident that my number of retries and delay between them was what was hanging my test cases up.
I will be adjusting those parameters as time permits, but I need to figure out where the error is occurring and how to address it. In every case I’ve looked at the backup appears to have worked. Is there an easy way to test backup integrity? In other words, can I manually run the test handler / file verify test to see if the issue happens from the command line or if it is just from the main program?

update: looks like my nextcloud data directory is seriously corrupted. So until I can either get sck to work to completion, or build a new raid array and copy the data back in from my clone, I want to suspend this question, as I believe the issue is all on my end, rather than duplicati. I’ll post another update if warranted. Thanks

final update: corruption was the issue on nextcloud and data is currently being copied back in to the raid array after wipe, relog bad blocks and mkfs. I guess the suddenness of the problem should have pointed me sooner to file system issues, rather than duplicati issues, so hopefully, there went many brain cycles wasted on my issue. Thanks for being there! Rick

I’m glad you found the cause. To reply to an earlier question that could only show integrity (or lack thereof):

The TEST command run from Commandline (after some screen setup) is an easy way to do a large test.

For a test that might be runnable at the remote (if it has Python and file access) try upload-verification-file, then DuplicatiVerify.py from the utility-scripts directory in a Duplicati installation (might be /usr/lib/duplicati).

backup-test-samples or backup-test-percentage Advanced option can adjust the post-backup verification.