I am using duplicati to make a daily backup using Onedrive for business as a storage location. This has been working perfectly for 5 weeks or so until this week I got the following error:
Failed: Found 5259 files that are missing from the remote storage, please run repair Details: Duplicati.Library.Interface.UserInformationException: Found 5259 files that are missing from the remote storage, please run repair at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile) at Duplicati.Library.Main.Operation.BackupHandler.PostBackupVerification() at Duplicati.Library.Main.Operation.BackupHandler.Run(String sources, IFilter filter) at Duplicati.Library.Main.Controller.<>c__DisplayClass17_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
I have tried the repair option but that fails as well, since that gives this error:
Duplicati.Library.Interface.UserInformationException: No files were found at the remote location, perhaps the target url is incorrect?
I have checked the url, it is correct and if I go to onedrive I can also still see the files are there. In order to make sure that backing up to my onedrive account still works at all I made another small backup task in duplicati using the same credentials but a different path and that works, so it is not a problem with the connection to onedrive or so. I also tried to see what happens if I change the pathname in onedrive to something incorrect and then I get a different error when trying to do a repair:
Duplicati.Library.Interface.FolderMissingException: Element with path ‘/personal/_onmicrosoft_com/Documents/mybackup/’ not found on host ‘https://.sharepoint.com:443/personal/_onmicrosoft_com’.
which also means that the correct path is still working.
Is 5259 roughly all the files in that folder? Please check the job log or emails for history of “KnownFileCount”. There’s a much-discussed 5000 file limit OneDrive for Business got from SharePoint that might be doing this.
Right, so it seems the problem is indeed with the listing which just seems to return an empty list. It looks like the solution would be to use onedrive v2 as storage type instead of onedrive for business, the only issue I see there is that this type of storage is not available in duplicati version 2.0.3.3_beta_2018-04-02, which is the latest stable version according to the website. I’m not sure if it is a good idea to switch to one of the newer builds, since they are mentioned as being less stable…?
2.0.3.11 is pretty good, but canary in general is more of an unknown compared to beta or even experimental. The work list for 2.0.3.9 made lots of progress in 2.0.3.10, then performance tuning and other fixes went into 2.0.3,11, so canary might be pretty good until the next big rewrite arrives as it did for 2.0.3.6 - note “Beware”.
Returning to Microsoft’s work, I couldn’t find any claim that the limit was removed in v2 (Microsoft Graph) but there’s always the possibility. I read of workarounds for the 5000 limit via indexes. Not sure if we can do them.
If you actually found some evidence that OneDrive v2 (a.k.a. Microsoft Graph) helps with limits, please detail. OnedriveV2 not available as provider choice has some information on what Duplicati and Microsoft are up to.
Maybe it’s time to ask if @tygill has any thoughts on the (painfully low) list view limit, and possible solutions? Subfoldering the destination has not been a popular idea, and increasing dblock sizes can also create issues.
I can’t seem to find any official documentation saying that the Graph API removes the per-folder file count limits, though there is a user in List large folder in Onedrive for business (od4b) · Issue #2580 · duplicati/duplicati · GitHub who reported that the new backend worked around the error for them. Additionally, I was just able to create a folder with 20000 items, and the Graph API started paging through them without problems (I’ve been manually paging through, so I haven’t reached 5000 yet, but it seems to be working).
As for fixing this for the old backend, I don’t think it’s possible, as it was a limit built in to the old OneDrive SDK / API, which is being disabled in November anyway.
I updated to the latest canary build today and switched to OneDrive v2, that did seem to resolve the issue. At least I was now able to repair the local database and the backup seems to be doing something again, as I see that new files are being uploaded. The UI doesn’t really seem to be representing what it is doing exactly, as the progress bar has already been stuck at basically 0% for a few hours now, but looking at the live log data from the server I can see that it is doing all kinds of things. I guess I will see in a couple of hours if the backup is now working correctly again.
The GitHub report seems a bit inconclusive, because a new backup would have fewer files (at least at first).
You’d have to watch KnownFileCount (and I assume @jeroen did so, and this started failing right at 5000).
Manually paging must take true dedication, so thanks . I wonder if @jeroen could try 2.0.3.5 canary on current overly-full destination using OneDrive v2. 2.0.3.5 is sometimes advised for people not wanting later. There should also (I expect) be a new Beta sometime, but if MS sticks with Nov 30 cutoff, timing seems tight.
Well, it seems the backup is working fine again with canary 2.0.3.11 and I have not experienced any other issues so far with this version, so I’ll stick to this one until a next stable/beta release is available. Thanks for your help!
. I wonder if