A backup (small test set that runs each day and uses Backblaze as the backend) has started to throw up errors (out of nowhere).
* 2022-07-14 19:35:44 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-20220518T072431Z.dlist.zip.aes
* 2022-07-14 19:36:38 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-i8f9f758241bd45f48538c13450d40340.dindex.zip.aes
* 2022-07-14 19:37:33 +02 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b606f45efd42148c281af22df4bc89669.dblock.zip.aes
The phrase Failed to process doesn’t tell me much. I checked B2 and the file is there. What should I do (and in what order)?
I am using 2.0.6.3 (2.0.6.3_beta_2021-06-17). I am running a number of small test sets on different machines to see how stable they are in the long run. It is possible that a backup aborts (e.g. when the laptop in runs on goes to sleep or changes from a standard network connection to a VPN). This one has been running without problems for a while. It is currently used in a situation where VPN is more often temporarily used.
Your messages look like the ones in this post: Errors on successful backup
can you confirm that there is the same tls error messages in your log ?
If yes it would confirm that there is something that changed in your storage provider. Maybe in this case they could provide help ?
Yes but my issue is now certainly network related and not a storage provider issue. I created a new bucket and key for Backblaze and ran a test backup at this site, same 3 errors. So I actually took the PC across the street to our other building plugged it in and the backup ran without any errors. I move it back to the original building and network and I get the 3 errors again. Every backup run in this office gets the errors.
Does anyone have any ideas? Or has anyone seen this before?
Well, I can actually corroborate this, sort of. I am currently in another location with my laptop. So, what I did was that I connected via VPN to my normal network, then I ran the back-up again. Result: this time the backup went fine.
I switched the VPN off again, and the backup ran fine as well after.
One possible difference: my network in the location that I got the error on has both IPv6 and IPv4.
Another possible clue: the first one that failed from this location had Deletions. It now doesn’t have Deletions anymore and it succeeds, both with and without VPN.
Thanks for the feedback.
I can carry the PC back and forth between our two offices in separate buildings, with separate Comcast Modems, one location no errors, original location it errors, I can go back and forth with the same results so it has no deletions. It’s the same backup to the same location, same key and bucket. Only thing changing is the network location / modem.
As for the VPN we have a site to site VPN through the firewall. So that should have no real impact as the traffic is going straight from the firewall out to Backblaze. The same was it does at both locations. The locations are identical in the way they are setup. That’s what makes me think Comcast is the guilty party.
I don’t have time today but Monday I am taking this PC to the IT room and will bypass our Router / firewall and connect directly through the Comcast Modem. And see if I get the errors. If I do it’s Comcast, If not something is happening in my firewall.
I will post an update on that Monday. Thanks again for your thoughts on this.
Are all these Windows systems? Linux can have other issues, but usually it’s a certificate problem.
Narrowing it down more might make a better case. There are OS-level tools that could also be run.
Wireshark would be informative but it’s kind of complicated to look at the problem at a packet level.
I’m not sure what an ISP would do to break things. I’d have thought they just do transport and DNS
(which you could probably change to some third-party DNS if you think ISP DNS might be a factor).
Well I tried the DNS change, no effect.
Yes they are all windows systems.
The backups are running and successful, just getting these three error messages.
So why I think it’s Comcast, I can setup a backup to run from my laptop to a new bucket, with a new key. It runs fine and completes without error at 2 of our buildings. I connect the laptop to the network across the street, backup runs, says complete but same 3 errors. I carry it back to the other 2 offices and it runs without error. So something in that building is causing it.
I still have to bring the laptop to the modem and try to bypass the firewall to see if it runs with or without errors.
Well, you’all say that it can’t be Backblaze yet I’m not convinced. When providers use CDN, distinction between network problems and provider problems can become hazy. It may be significant that both of you have problem with the verification stage, even if the actual messages are not the same. It may be a common underlying problem of CDN caching and updating, manifesting itself in different ways.
We have two people reporting similar problems but with slightly different error messages.
suggests something pretty low-level (whatever it is). I used to use Internet Explorer to test,
however Microsoft has been killing it off. Can you test with Duplicati CLI tools I mentioned?
An even easier but less effective test is to use Test connection on Destination screen.
That should do a list operation and 1 set (typically 3 files) of get, as end of backup does.
Other harder methods include two traditionally not-Windows programs that Windows ships: openssl is sometimes good for poking at SSL/TLS issues. curl is aiming a little bit higher.
Grasping at straws, I suppose you could also try allowed-ssl-versions, e.g. hardcode Tls12
Note: I can no longer test this. It happened with a laptop while I was on vacation in Switzerland and I’m now back.
My guess is that something is wrong with Backblaze’s CDN, to be precise: with their certificates and their CDN. Depending on the location you connect from you end up in a part of the Backblaze CDN that has a certificate issue.
Another option might be (but I don’t know enough to ascertain this is a real option) is that depending on where you come from, there is something with the route back. For instance a mix of IPv6 and IPv4 which I noticed I had from my situation in Switzerland. Maybe, Backblaze tries to ascertain the validity of the route back and cannot, e.g. because in the protocol tells the other side what it’s address is and that address is not the same as where the traffic actually comes from (which might be a way to detect spoofing).