To start with, I really like Duplicati! It’s really nice to have it in the background on my Debian server, and the backups just work… well, some of them…
The cloud service with the most space is my OneDrive for Business (about 1 TB), but it just does not work for me.
I have read all the tips about how you “sign in”, testing the connection works, Duplicati manages to write files, but the verification just does not work:
Cut out from result log:
MainOperation: Test Verifications: [ Key: duplicati-20171121T204420Z.dlist.zip.aes Value: [ Key: Error Value: The request timed out ]
**
Errors: [ Failed to process file duplicati-20171121T204420Z.dlist.zip.aes => The request timed out,
All my other services work well: OneDrive Personal, G-Drive, BackBlaze, Mega (extreemely slow, though).
Hi @rickard - thanks for using (and liking!) Duplicati!
It sounds to me like you’ve got a slow connection and it’s Duplicati gives up waiting for OD4B to respond (hence the “timed out” error).
You didn’t mention what version of Duplicati you’re using, but if it’s one of the newer Canary versions you should have access to the --http-operation-timeout parameter which lets you extend how long Duplicati will wait for a response from your cloud provider.
Note that I believe the default timeout is 10 minutes, so if you’re finding that the error comes up 10 minutes after the last step then that’s a good confirmation we’re on the right track.
(BTW - I edited your post by putting “~~~” before and after the error message to make it easier to read.)
The edit of my post, did that create the textbox with scrollbars? (Nice!)
I am running the lastest stable version for Debian “duplicati_2.0.2.1-1_all.deb”.
Well… It’s not a slow connection, I have 100 Mbps and it works great (and fast) with OneDrive Personal, where I store 80 GB, compared to OD4B where my test files are a total of 95 MB. On the other hand, the symptoms are like timeouts, which also the error messages say.
Under “Advanced options” I find a parameter for OD4B “web-timeout”, could that be what you are thinking about?
Yep - I like Discourse (this forum tool) quite a bit.
To be honest, I’ve never used OD4B so I’ve never used it before, but from the description in the web GUI it sounds like it will likely do (just for OD4B) what the http-operation timeout does in general.
I’d say give that (--web-timeout in OD4B options) a try and see if that helps at all.
The backup operations works really fast, and then the verification phase starts, when Duplicati should fetch a file, and that’s when everything stalls…
I changed the web-timeout to 300 secs (5 mins), which sound way to much… I have verifications with 500 MB files that are a lot less than 5 mins. Never the less, it was tried.
The logs below are from different stages in the process.
System.Net.WebException: The request timed out
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00052] in <bd46d4d4f7964dfa9beea098499ab597>:0
at System.Net.HttpWebRequest.GetResponse () [0x0000e] in <bd46d4d4f7964dfa9beea098499ab597>:0
at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute () [0x00010] in <ae42712e06c349faba29591f4f24e683>:0
at Microsoft.SharePoint.Client.File.OpenBinaryDirect (Microsoft.SharePoint.Client.ClientContext context, System.String serverRelativeUrl) [0x00078] in <8e3eb3065b524a909640c2bd1341366d>:0
at Duplicati.Library.Backend.SharePointBackend.doGet (System.String remotename, System.IO.Stream stream, System.Boolean useNewContext) [0x000cd] in <1e96623231d94ecd98631cef66e60b62>:0
at Duplicati.Library.Backend.SharePointBackend.Get (System.String remotename, System.IO.Stream stream) [0x00000] in <1e96623231d94ecd98631cef66e60b62>:0
at Duplicati.Library.Main.BackendManager.coreDoGetPiping (Duplicati.Library.Main.BackendManager+FileEntryItem item, Duplicati.Library.Interface.IEncryption useDecrypter, System.Int64& retDownloadSize, System.String& retHashcode) [0x001a2] in <118ad25945a24a3991f7b65e7a45ea1e>:0
Operation Get with file duplicati-i0ec51ce95e624d979a68a8ae2511a889.dindex.zip.aes attempt 5 of 5 failed with message: The request timed out
System.Net.WebException: The request timed out
at Duplicati.Library.Main.BackendManager.GetForTesting (System.String remotename, System.Int64 size, System.String hash) [0x00054] in <118ad25945a24a3991f7b65e7a45ea1e>:0
at Duplicati.Library.Main.Operation.TestHandler.DoRun (System.Int64 samples, Duplicati.Library.Main.Database.LocalTestDatabase db, Duplicati.Library.Main.BackendManager backend) [0x003f7] in <118ad25945a24a3991f7b65e7a45ea1e>:0
Just to confirm the error is where we think it is, does the job run without errors if you set --backup-test-samples=0 or --no-backend-verification=true (checked)?
Not that you necessarily want to run that way, but it does confirm where the problem lies.
I set the “no backend verification”, and if nothing was changed it was lightning fast. When I added source files, it was slower than the other backups, but a lot faster than “normal” for OD4B. I did, however, get two warnings:
Delete operation failed for duplicati-i6224f3daf48642e596372eb096c7accf.dindex.zip.aes with FileNotFound, listing contents
Duplicati.Library.Interface.FileMissingException: Element with path '/personal/rickard************/Documents/Rickard/Backup/duplicati-i6224f3daf48642e596372eb096c7accf.dindex.zip.aes' not found on host 'https://*****-my.sharepoint.com/personal/rickard***************'.
at Duplicati.Library.Backend.SharePointBackend.wrappedExecuteQueryOnConext (Microsoft.SharePoint.Client.ClientContext ctx, System.String serverRelPathInfo, System.Boolean isFolder) [0x00064] in <1e96623231d94ecd98631cef66e60b62>:0
at Duplicati.Library.Backend.SharePointBackend.doDelete (System.String remotename, System.Boolean useNewContext) [0x00034] in <1e96623231d94ecd98631cef66e60b62>:0
Do others get OD4B to work just like the other services, or is it really tricky to get it to work?
Well, as of yesterday 1,046 backups were saved to OneDrive for Business so I’m guessing there’s something specific to your setup / connection that is causing the issue.
Regarding the new FileMissingException error, I may have to defer to somebody more familiar with that side of things…perhaps @kenkendk has some thoughts.
On my Win10 workstation, I have tested “duplicati-2.0.2.1_beta_2017-08-01-x64”, and here the OD4B seems to work! Seemingly a little slower than the other cloud services, but no warnings and errors!
So, I take it we have verified that my network is OK, and that there is a difference in the two duplicati versions, right?
@rickard, unfortunately that issue (I added the link to your post, BTW - hope you don’t mind) is beyond my expertise so I’m going to put a shout out to @kenkendk for some input.
i was also trying to switch to onedrive for business, but can’t seem to be able to connect (on synology). However, I need to try a bit more.
I see OneDrive has OAUTH authentication in Duplicati. Would that be possible for OD4B too? We are using a SSO login, which probably doesn’t work in the normal login procedures.
I did get OD4B to work with rclone using OAUTH, so it should be possible to do that for Duplicati as well.
2017-12-15 21:14:36Z - Warning: Operation List with file attempt 1 of 5 failed with message: No SharePoint web could be logged in to at path 'https://XXXXX-my.sharepoint.com/personal/XXX_XXXXX_XX/Documents/Duplicati/Test'. Maybe wrong credentials. Or try using '//' in path to separate web from folder path.
System.Net.WebException: No SharePoint web could be logged in to at path 'https://XXXXX-my.sharepoint.com/personal/XXX_XXXXX_XX/Documents/Duplicati/Test'. Maybe wrong credentials. Or try using '//' in path to separate web from folder path.
at Duplicati.Library.Backend.SharePointBackend.getSpClientContext (System.Boolean forceNewContext) [0x00070] in <7071b8cb3c53472e8633e3e37f5e18d5>:0
at Duplicati.Library.Backend.SharePointBackend+<doList>d__28.MoveNext () [0x00032] in <7071b8cb3c53472e8633e3e37f5e18d5>:0
at System.Collections.Generic.List`1[T]..ctor (System.Collections.Generic.IEnumerable`1[T] collection) [0x0008b] in <8f2c484307284b51944a1a13a14c0266>:0
at System.Linq.Enumerable.ToList[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00011] in <63992662b765477a898ef49cdcc99ee2>:0
at Duplicati.Library.Main.BackendManager.DoList (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x0001b] in <df9891e588c04bd199c53e6d7539922f>:0
at Duplicati.Library.Main.BackendManager.ThreadRun () [0x000ee] in <df9891e588c04bd199c53e6d7539922f>:0
When browsing to the mentioned URL, I can see the files that have been uploaded.
@rickard: If it fails on Linux, but works on Windows, my guess is that is some issue with Mono. Make sure you are using the Xamarin provided version (5.x) and not the old Debian provided version (3.x).
If possible, you can also try the latest build (2.0.2.15_experimental, should be available right now), and see if some of the general updates fixes the problem.
I have a bit of trouble debugging the issue, so I hope you can try various things to find out when it breaks. I have a test account for OD4B, but I can’t start throwing multiple GB at it to see if it breaks.
Hello @kenkendk!
Good call with the Mono library, I was running with the built in Debian default (4.6.2.7) and changed to Xamarin (5.4.1.6). However, this did not fix the issue…
I also installed the experimental Duplicati (2.0.2.15), and that did not change anything either. The behaviour is still the same, with timeouts during verification, and sometimes complete authentication errors.
It’s really nice to have it in the background on my Debian server, and the backups just work… well, some of them…
