The meat of the backup process proceeds as normal (uploading new data, deleting expired backups)
The verification stage fails to verify even a single file, timing out when connecting to the backend
However, in this case, I already have the option --read-write-timeout=0 configured, which fixed the problem last time. Also in contrast to last time, Duplicati now fails out the entire backup job after exhausting its allotted retries, which is certainly preferable to continuing to retry files until the cows come home.
Here’s a log, for the curious.
(The file it’s failing on does exist, and I was able to download it manually via Backblaze’s web UI.)
Are B2 backend calls still subject to some sort of timeout weirdness in 2.2.0.0? If so, do we have a new workaround?
You can try for yourself, but Google search for your error in past year found none.
--read-write-timeout (Timespan): Read/write operation timeout
The timeout in seconds for read and write operations. If no activity is
detected in this interval, a timeout error is raised
* default value: 30s
Have you tried taking the 0 off. It’s undefined by the help, so makes me nervous.
2.1.0.4 help says differently, but 2.2.0.0 did a lot of reworking in the timeout area:
--read-write-timeout (Timespan): Set the read/write timeout for the
connection
The read/write timeout is the maximum amount of time to wait for any
activity during a transfer. If no activity is detected for this
period, the connection is considered broken and the transfer is
aborted. Set to 0s to disabled
* default value: 30s
If you don’t want to test with backup, Verify files could test some downloads.
Unfortunately, removing the --read-write-timeout option from the backup doesn’t seem to have made a difference (log).
I then tried running a test operation, which also failed in the same manner (log).
However, then my Linux server ran its weekly backup to B2 last night, and it also failed to verify any files, despite still being on Duplicati 2.0.8.1_beta (log).
This, along with @andrius’s comment above, led me to believe that this might not be an issue with Duplicati after all. To test, I installed Cloudflare’s WARP client on my laptop and ran my B2 backup again. Sure enough, it completed with no issues whatsoever (log).
Additional testing with Backblaze’s b2 command-line tool produced similar results: I can download files from B2 while connected to a VPN, but running the same command without VPN just hangs indefinitely.
At this point, I highly suspect this to be an issue with my ISP, so I’ll give them a call and let y’all know if I get any useful information out of them.
Testing in their own tool was a good idea – keeps problem more in their space.
Possibly they have an idea, but networking weirdness can be tough to analyze.
So, after much back and forth with my ISP (including a factory reset and replacement of my modem) and after subsequently opening a ticket with Backblaze, it seems that someone, somewhere, has incorrectly flagged some address(es) at their East Coast datacenter as malicious, which has resulted in them being blocked:
We have been made aware of a subset of customers receiving connection issues with our East Coast datacenter. We believe that internet monitoring tools inaccurately marked our subdomains f005.backblazeb2.com as malicious websites. Our compliance team is working on correcting this false reputation.
We do recommend reaching out to your ISP as well to report the false positive. We’ve found that these reports tend to be taken more seriously when it comes from direct customers.
At this point, I have contacted my ISP as Backblaze recommended, and I suppose it will be a matter of waiting until the complaint trickles through their bureaucracy to reach the person who can actually do something about it.