I am using duplicati to make a daily backup using Onedrive for business as a storage location. This has been working perfectly for 5 weeks or so until this week I got the following error:
Failed: Found 5259 files that are missing from the remote storage, please run repair Details: Duplicati.Library.Interface.UserInformationException: Found 5259 files that are missing from the remote storage, please run repair at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile) at Duplicati.Library.Main.Operation.BackupHandler.PostBackupVerification() at Duplicati.Library.Main.Operation.BackupHandler.Run(String sources, IFilter filter) at Duplicati.Library.Main.Controller.<>c__DisplayClass17_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
I have tried the repair option but that fails as well, since that gives this error:
Duplicati.Library.Interface.UserInformationException: No files were found at the remote location, perhaps the target url is incorrect?
I have checked the url, it is correct and if I go to onedrive I can also still see the files are there. In order to make sure that backing up to my onedrive account still works at all I made another small backup task in duplicati using the same credentials but a different path and that works, so it is not a problem with the connection to onedrive or so. I also tried to see what happens if I change the pathname in onedrive to something incorrect and then I get a different error when trying to do a repair:
Duplicati.Library.Interface.FolderMissingException: Element with path ‘/personal/_onmicrosoft_com/Documents/mybackup/’ not found on host ‘https://.sharepoint.com:443/personal/_onmicrosoft_com’.
which also means that the correct path is still working.
So how do I fix this?
Is 5259 roughly all the files in that folder? Please check the job log or emails for history of “KnownFileCount”. There’s a much-discussed 5000 file limit OneDrive for Business got from SharePoint that might be doing this.
Manage large lists and libraries in SharePoint
You possibly can also see in your job log how well the initial listing is going, by looking for something like this:
2018-09-07 12:28:04 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2018-09-07 12:28:05 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2018-09-07 12:28:05 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (36 bytes),
to see if your listing got very long, really short, or failed. You might need to set --log-level=Information for this.
The initial listing check can be suppressed, but I’m not sure how advisable it is to just try to live without listing.
–dry-run might make tests safe enough (to see if things get further), but lack of lists might break other things.
List large folder in Onedrive for business (od4b) #2580 is a similar problem where an empty list was returned.
Right, so it seems the problem is indeed with the listing which just seems to return an empty list. It looks like the solution would be to use onedrive v2 as storage type instead of onedrive for business, the only issue I see there is that this type of storage is not available in duplicati version 184.108.40.206_beta_2018-04-02, which is the latest stable version according to the website. I’m not sure if it is a good idea to switch to one of the newer builds, since they are mentioned as being less stable…?
220.127.116.11 is pretty good, but canary in general is more of an unknown compared to beta or even experimental. The work list for 18.104.22.168 made lots of progress in 22.214.171.124, then performance tuning and other fixes went into 2.0.3,11, so canary might be pretty good until the next big rewrite arrives as it did for 126.96.36.199 - note “Beware”.
Returning to Microsoft’s work, I couldn’t find any claim that the limit was removed in v2 (Microsoft Graph) but there’s always the possibility. I read of workarounds for the 5000 limit via indexes. Not sure if we can do them.
If you actually found some evidence that OneDrive v2 (a.k.a. Microsoft Graph) helps with limits, please detail. OnedriveV2 not available as provider choice has some information on what Duplicati and Microsoft are up to.
Maybe it’s time to ask if @tygill has any thoughts on the (painfully low) list view limit, and possible solutions? Subfoldering the destination has not been a popular idea, and increasing dblock sizes can also create issues.
I can’t seem to find any official documentation saying that the Graph API removes the per-folder file count limits, though there is a user in List large folder in Onedrive for business (od4b) · Issue #2580 · duplicati/duplicati · GitHub who reported that the new backend worked around the error for them. Additionally, I was just able to create a folder with 20000 items, and the Graph API started paging through them without problems (I’ve been manually paging through, so I haven’t reached 5000 yet, but it seems to be working).
As for fixing this for the old backend, I don’t think it’s possible, as it was a limit built in to the old OneDrive SDK / API, which is being disabled in November anyway.
I updated to the latest canary build today and switched to OneDrive v2, that did seem to resolve the issue. At least I was now able to repair the local database and the backup seems to be doing something again, as I see that new files are being uploaded. The UI doesn’t really seem to be representing what it is doing exactly, as the progress bar has already been stuck at basically 0% for a few hours now, but looking at the live log data from the server I can see that it is doing all kinds of things. I guess I will see in a couple of hours if the backup is now working correctly again.
The GitHub report seems a bit inconclusive, because a new backup would have fewer files (at least at first).
You’d have to watch KnownFileCount (and I assume @jeroen did so, and this started failing right at 5000).
Manually paging must take true dedication, so thanks . I wonder if @jeroen could try 188.8.131.52 canary on current overly-full destination using OneDrive v2. 184.108.40.206 is sometimes advised for people not wanting later. There should also (I expect) be a new Beta sometime, but if MS sticks with Nov 30 cutoff, timing seems tight.
Well, it seems the backup is working fine again with canary 220.127.116.11 and I have not experienced any other issues so far with this version, so I’ll stick to this one until a next stable/beta release is available. Thanks for your help!