New install today on W7 … 3G of data, backing up to Backblaze cloud. I have heaps of other customers using this exact setup, but this one gets stuck on Verifying at the end:
backupname: Verifying …
Also there seems to be no way to force stop the backup, even tried restarting the service - forgot to mention this is a multi-user setup with Duplicati running as a service.
Have tried repairing the database and this ran very quickly and said there was nothing to do.
Extract from the results log …
[ Key: Error Value: Received an unexpected EOF or 0 bytes from the transport stream. ]
… Title should read “Stuck on Verifying” … anyone had something similar to this?
I am suspicious that the ESET antivirus is interfering with the communication to Backblaze … we tried to uninstall the Eset but the PC wouldn’t boot after that … so after reverting back with System Restore we now find that the Duplicati backup is running every hour but only completing error free about once per day.
The results log message sounds to me like the SSL/TLS encrypted connection to B2 didn’t return anything.
Secured websites do not load with ESET installed might lead you to some way to have ESET leave it alone, although if it turns out to help, ESET support might have more to say on the failure (or that uninstall wreck).
Possibly there’s something else killing the network. Is this system the same, and same network, as the rest?
There are several verifications after the backup. You could try dialing one down with –backup-test-samples.
EDIT: This is a test to isolate the problem. If you can’t verify, your actual restores might also fail. Please test.
Debugging at the network level is possible but awkward due to the encryption. More easily, you can see the operations being attempted by using MENU --> About --> Show log --> Live --> Information in a different tab:
Sep 27, 2018 10:21 AM: The operation Backup has completed
Sep 27, 2018 10:21 AM: Backend event: Get - Completed: duplicati-bd8be1743cca24a4385281d6cecd0971d.dblock.zip (700 bytes)
Sep 27, 2018 10:21 AM: Backend event: Get - Started: duplicati-bd8be1743cca24a4385281d6cecd0971d.dblock.zip (700 bytes)
Sep 27, 2018 10:21 AM: Backend event: Get - Completed: duplicati-i1beed8592ce541adb85338f4150b1f0a.dindex.zip (609 bytes)
Sep 27, 2018 10:21 AM: Backend event: Get - Started: duplicati-i1beed8592ce541adb85338f4150b1f0a.dindex.zip (609 bytes)
Sep 27, 2018 10:21 AM: Backend event: Get - Completed: duplicati-20180915T124156Z.dlist.zip (538 bytes)
Sep 27, 2018 10:21 AM: Backend event: Get - Started: duplicati-20180915T124156Z.dlist.zip (538 bytes)
Sep 27, 2018 10:21 AM: Backend event: List - Completed: (12 bytes)
Sep 27, 2018 10:21 AM: Backend event: List - Started: ()
Sep 27, 2018 10:21 AM: Backend event: List - Completed: (12 bytes)
Sep 27, 2018 10:21 AM: Backend event: List - Started: ()
Sep 27, 2018 10:21 AM: The operation Backup has started
for example. Lines are reverse-chronological, so yours might show an upper Get Started but not Completed.
I have the same thing happening on a Macbook Pro now … Duplicati backing up to Backblaze, no antivirus installed, have restarted the Macbook and deleted and repaired the database all to no avail.
With the Macbook Pro it appears to complete the backup, and indeed on the web interface it reports completed, but then shows Verifying and does not progress beyond that point. There is an email on completion in the configuration but this task is never completed.
This is more experiment than workaround. I edited to clarify that point. Verification first tries to download data from the destination, which is similar (but not identical) to restore usage. Make sure your restore really works. Reports of success or failure there would help further understand what’s going on in the mysterious situation. There’s also The TEST command to try directly or within the Graphical User Interface, editing to fit command.
Yes restore seems to work ok … a test file was backed up, modified, backed up again, then the original version of the file was restored and renamed to avoid overwriting the latest version.
I wonder if that test file backup would also survive the default --backup-test-samples? The original Windows report sounded like something intermittent, only completing about once per day. Such problems might vary depending on what’s going on. I’m glad you got restore to work, but I’d feel better knowing what hung verify. The original problem sounded like it was at a customer site, so possibly there are some other differences…
If you can get one hung again, looking at “Started/Completed” might at least show if hang was in the middle, however after that life may get very difficult from a network debug point of view. Logging detail also runs out.
So this is seen on two different systems, one Windows 7, one Macbook Pro. Any similarities, e.g. is Internet access similar? You ruled out interference from a virus scanner, but boxes in the network can also interfere. “Received an unexpected EOF or 0 bytes from the transport stream” does have some Google results which might lead you to something useful. The first one I checked changed SSL/TLS settings, which Duplicati can configure via –allowed-ssl-versions. That doesn’t seem to explain the original post, where most setups work. The other puzzling thing is that the SSL/TLS should be pretty much the same for backup, verify, restore, etc. EDIT: however that doesn’t mean there’s not some other clues you could get from a search on the error text.
Correct - two different devices, Windows 7 and Macbook totally unrelated, both failing in a similar way. Nothing in common or similar with their internet connections but they are both simple connections through a router and there is no firewall or other filtering.
W7 instance fails again if I set the default value 0 for --backup-test-samples.
Yes the restore works, even though the Verify fails.
Intermittent problems are hard to deal with. Are any failures solid (one good run per day might be solid enough), and do all of them show “Received an unexpected EOF or 0 bytes from the transport stream.” which has been a focus effort, because there’s so little else to go by? The Live log test mentioned earlier might help to confirm the location of the hang is where I’m guessing (and which you removed with --backup-test-samples=0). Was there a typo in the prior reply? I’ll assume “default value 0” meant “default value 1”, or else things got very inconsistent.
If somehow the Live log doesn’t help (because it looks kind of like it says the outcome before the steps towards it), an alternative would be to add to the job the –log-file and –log-level options. Level Information might suffice. Possible step after that would be to examine the network activity directly, if you can install some tools for that…
Here’s log output from 2.0.3.3 of a small backup first showing one without a sampled verify and then one with:
With --backup-test-samples=0, the "Get" sample of one trio of files is not done:
2018-10-15 13:10:01Z - Information: Backend event: List - Started: ()
2018-10-15 13:10:01Z - Information: Backend event: List - Completed: (12 bytes)
2018-10-15 13:10:01Z - Information: Backend event: List - Started: ()
2018-10-15 13:10:01Z - Information: Backend event: List - Completed: (12 bytes)
2018-10-15 13:10:02Z - Information: removing file listed as Temporary: duplicati-b12f0ee42fa814f7690e343901ce2e263.dblock.zip
2018-10-15 13:10:02Z - Information: removing file listed as Temporary: duplicati-i72a3306dde584c49ba6c137e209283bc.dindex.zip
2018-10-15 13:13:28Z - Information: Backend event: List - Started: ()
2018-10-15 13:13:28Z - Information: Backend event: List - Completed: (12 bytes)
2018-10-15 13:13:28Z - Information: Backend event: Put - Started: duplicati-b5f282807110f4bc5a5f2d85b203a8e69.dblock.zip (510 bytes)
2018-10-15 13:13:28Z - Information: Backend event: Put - Completed: duplicati-b5f282807110f4bc5a5f2d85b203a8e69.dblock.zip (510 bytes)
2018-10-15 13:13:28Z - Information: Backend event: Put - Started: duplicati-i5d57f261623a4324b25071c98e4aec6a.dindex.zip (567 bytes)
2018-10-15 13:13:28Z - Information: Backend event: Put - Completed: duplicati-i5d57f261623a4324b25071c98e4aec6a.dindex.zip (567 bytes)
2018-10-15 13:13:28Z - Information: Backend event: Put - Started: duplicati-20181015T131328Z.dlist.zip (2.67 KB)
2018-10-15 13:13:28Z - Information: Backend event: Put - Completed: duplicati-20181015T131328Z.dlist.zip (2.67 KB)
2018-10-15 13:13:28Z - Information: Compacting not required
2018-10-15 13:13:28Z - Information: Backend event: List - Started: ()
2018-10-15 13:13:28Z - Information: Backend event: List - Completed: (15 bytes)
unlike here, so I wonder how far yours got before hang?
2018-10-15 13:14:26Z - Information: Backend event: List - Started: ()
2018-10-15 13:14:26Z - Information: Backend event: List - Completed: (15 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Put - Started: duplicati-b2289b3459d614cdca286e11d6a93beca.dblock.zip (510 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Put - Completed: duplicati-b2289b3459d614cdca286e11d6a93beca.dblock.zip (510 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Put - Started: duplicati-i5b8a7fac1b894b38a40be6740d7a9259.dindex.zip (567 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Put - Completed: duplicati-i5b8a7fac1b894b38a40be6740d7a9259.dindex.zip (567 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Put - Started: duplicati-20181015T131426Z.dlist.zip (2.67 KB)
2018-10-15 13:14:27Z - Information: Backend event: Put - Completed: duplicati-20181015T131426Z.dlist.zip (2.67 KB)
2018-10-15 13:14:27Z - Information: Compacting not required
2018-10-15 13:14:27Z - Information: Backend event: List - Started: ()
2018-10-15 13:14:27Z - Information: Backend event: List - Completed: (18 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Get - Started: duplicati-20181015T131426Z.dlist.zip (2.67 KB)
2018-10-15 13:14:27Z - Information: Backend event: Get - Completed: duplicati-20181015T131426Z.dlist.zip (2.67 KB)
2018-10-15 13:14:27Z - Information: Backend event: Get - Started: duplicati-i5b8a7fac1b894b38a40be6740d7a9259.dindex.zip (567 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Get - Completed: duplicati-i5b8a7fac1b894b38a40be6740d7a9259.dindex.zip (567 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Get - Started: duplicati-b2289b3459d614cdca286e11d6a93beca.dblock.zip (510 bytes)
2018-10-15 13:14:27Z - Information: Backend event: Get - Completed: duplicati-b2289b3459d614cdca286e11d6a93beca.dblock.zip (510 bytes)
You could try changing –number-of-retries to see if hang changes, but that seems inconsistent with a hang…
I’m wondering if it might be a data limit on the Backblaze side - the one working backup of the day uses up all the allowed Backblaze data for the day then the rest fail…maybe?
Actually what you are suggesting makes a lot of sense … but I just checked the account for the W7 setup and there is no data cap for any of the storage or download.
This idea is still worth exploring as for this setup there is only 3.7G of data which is under the free tier, and the account allows 1G of daily download for free … but there are some small charges on the account for past month … hmmm.
Sorry if this sounds like I am hijacking this thread…
I am having a stuck on verifying problem too.
I am backing up to a remote server over SSH. It hangs during backup every time, and I haven’t been able to get a single backup to complete over several weeks.
I have tried creating small backup sets of only a handful of files. These either hang during backup, or hang during the “verifying” phase. The last one I tried completed the backup in 7 seconds, but hung for a day after than (when I then killed it).
Maybe as a workaround, it would make sense to add a ‘watchdog’ function to detect these? One related problem is that when it hangs, it stops all jobs until user intervention. A watchdog could allow other jobs to start, and/or give the broken job another shot at completing before I notice it has died.
Can you supply some Backend event log information to see where it got stuck? See examples above. Additionally, some SFTP servers have logs, and it would be interesting to see the view from that end…
How does the Test connection button on Job → Edit → Destination do? Any solid/random failures?
Although Duplicati is not great at concurrent backend usage (thus limiting performance), there might be enough concurrency that it could do badly to an SSH/SFTP server with an extremely low limit such as 1. Network problems are also possible, but debugging with netstat or Wireshark is a more advanced thing.