Operation.TestHandler-FailedToProcessFile

Support
1

Duplicati - 2.0.6.3_beta_2021-06-17 (since January 2022)
Ubuntu 20.04

We run several backups to several BackBlaze buckets. Starting a few days ago they started to fail on the verification portion of the backup.

  • 2022-09-05 02:09:37 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-20220905T060000Z.dlist.zip.aes
  • 2022-09-05 02:10:53 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-i5d947e5b1e7d4334b03cdfceee5c1b3c.dindex.zip.aes
  • 2022-09-05 02:12:10 -04 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b60f6bb514d5d496aa34f6154a2138ca2.dblock.zip.aes

Running a Verfify on the backup, immediately after the fail, will fail again with the same errors. Running a Verify 15-20 minutes later will succeed.

Testing has shown that the files are in the BackBlaze bucket and can be downloaded, decrypted (with AESCrypt) and unzipped, even while the Verify is failing.

Duplicati is making five attempts on each file. A sample error in a verbose log is

Sep 5, 2022 3:19 AM: Operation Get with file duplicati-20220905T065751Z.dlist.zip.aes attempt 1 of 5 failed with message: Raw message:
{“ClassName”:“System.Exception”,“Message”:"Raw message: “,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:” at Duplicati.Library.Backend.Backblaze.B2AuthHelper.AttemptParseAndThrowException (System.Exception ex) [0x000ad] in :0 \n at Duplicati.Library.Backend.Backblaze.B2.Get (System.String remotename, System.IO.Stream stream) [0x000db] in :0 \n at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x00259] in :0 \n at Duplicati.Library.Main.BackendManager.DoGet (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x002ff] in :0 \n at Duplicati.Library.Main.BackendManager.ThreadRun () [0x000ff] in :0 ",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:null,“HResult”:-2146233088,“Source”:“Duplicati.Library.Backend.Backblaze”}

I’ve restarted the duplicati daemon. No change.

Is there any way to log the actual BackBlaze API calls?

Thanks!

2

Maybe related

3

Thank you @gpatel-fr
I believe the suggested link identified the cause as DNS related.
We are using a BackBlazeB2 backend. I was able to test with b2 cli commands and with Duplicati.CommandLine.BackendTool.exe.
Even though the errors logged during backups didn’t point to SSL issues, I was able to provoke a failure with with a get operation with the BackendTool. The error included a reference to a certificate error.
(Sorry no examples. The error’s rolled out of my terminal history and the error has been hard to reproduce)

It appears our comcast internet (not managed by us) has a feature called SecurityEdge enabled. That product hijacks DNS requests and uses comcast’s own servers. It appears it was returning out of date or incorrect DNS information (for example looking up f004.backblaze2.com).

For now I think this is solved (I’ve been able to route DNS around comcast for now while trying to get SecurityEdge turned off). In any event It doesn’t appear to be Duplicati issue.