Backup failing with key already exists exception on OneDrive

I have two backups configured on my desktop Win 10 PC - one to a network share on another PC, the other to OneDrive, same set of source files for both backups. All of a sudden today the backup to OneDrive has started failing with the following message


Failed: An item with the same key has already been added.
Details: System.ArgumentException: An item with the same key has already been added.
   at Duplicati.Library.Main.BackendManager.List()
   at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
   at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
   at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass16_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

Iā€™ve tried verifying files, deleting and repairing the database, all without any success. There is plenty of free space on the destination (~400GB). I uploaded a bugreport.zip file. Any ideas as to anything else I could try.

Thanks,
Pat.

bugreport.zip (2.6 KB)

This sounds to me like somehow youā€™re getting duplicate files being reported from your destination.

Iā€™m not sure how to debug that (itā€™s too bad the error didnā€™t include they offending ā€œkeyā€). I know it sounds trite, but have you rebooted (or restarted the server - either Tray Icon or service) since the error?

Yes, I had rebooted, should have said so. I should also mention that I have two other PCs that Iā€™m backing up to OneDrive that havenā€™t shown any problems. The PC with the problem had been backing up to OneDrive for about 3 months without a problem until yesterday.

Is there any logging capability beyond that in the web UI that might give a clue?

Do any of these get you want you want?

Iā€™m not seeing a lot more in the log when I add


--log-file=c:\temp\duplicati.log
--log-level=Profiling

If the problem is still happening over the weekend Iā€™ll try cloning the repo and see if I can get it to build and take a shot at debugging it.

Thanks for the help.
Pat.

Youā€™re welcome, sorry I couldnā€™t find anything for you. Good luck with the debugging!

Itā€™s probably not related but I noticed OneDrive mentioned. Could your issue be at all related to this one?

I donā€™t think itā€™s related to that issue - that one is about including the OneDrive sync folder in the source set, seems redundant to be honest.

I was able to do some debugging and your suspicion was correct. It would appear Microsoftā€™s API is badly broken. The exception is being thrown in the OneDrive.List method. I added some logging and was able to see that the total number of records returned was correct, 16887, i.e it matches the number of files reported by the OneDrive Android app, but each time I ran it I only got around 11K or 12K unique file names. For each record I logged both the file name and the record number and when I sort the log I see a lot of lines like

duplicati-b0008a01560704fb981eb59312b784b63.dblock.zip.aes,10024
duplicati-b0008a01560704fb981eb59312b784b63.dblock.zip.aes,12556

duplicati-b0008d8a943814010b6e57440425175ce.dblock.zip.aes,12202
duplicati-b0008d8a943814010b6e57440425175ce.dblock.zip.aes,16822

In the Android app there are of course files visible that are not in the log file.

I apologize if youā€™ve already included this, but I couldnā€™t find what version of Duplicati youā€™re using.

If itā€™s the beta version, how would you feel able trying a portable "install"of a canary version to see if it reports the same issue?

Iā€™m just guessing random stufftnuff here but you actual fine count of 16,887 is just a bit over a nice round 16,384 (nowhere near tnyint or into, but still oddā€¦) so perhaps weā€™ve got a variable thatā€™s just not big enough to handle all your files.

If I have time in the next day our two Iā€™ll try to set up OneDrive backup with a small block size replicate your remote file count.

I think we have the same behaviour, but my backup have 23321 files, so the variable overflow I donā€™t think it is the problem (at least in the duplicate side).

If Iā€™m not wrong, the problem appears this tuesday, right? Maybe MS is making some change to the service or have problems with some servers.

I have another backup to onedrive with similar amount of data. I will wait some days to see if the problem disappears or appears in the other backup.

Glad itā€™s not just me :slight_smile: - The first backup that failed for me was also Tuesday. Iā€™m pretty certain that there was no significant increase in the volume of data being backed up since the previous weekend (Oct 14th) , itā€™s unlikely that a significant increase in the number of files occurred on the backend anytime in the week before the failure, so I have to suspect Microsoft broke something.

Looking at the OneDrive.List method the count and offset variables are both declared as int and as the build target is ā€˜Any CPUā€™ and Iā€™m running Windows 10 the variables should be 64 bit integers so I think we can rule out an overflow issue, at least on the Duplicati end.

I havenā€™t been able to find reports online of this issue with the API. One question that does come to mind is that as the Android app is working is it using the same API?

I am using the current beta release - I tried the latest canary build and see the same issue.

As a side note I should mention I was able to clone and build this project with no issues whatsoever - very nice!

If I find some time Iā€™ll try to set up a test backup to OneDrive and see if I can replicate the issue.

Actually ā€œintā€ in .Net is 32 bit regardless of the machine (and ā€œlongā€ is 64 bit, regardless of the machine).

I think the exception happens here:

Duplicati current calls something like https://apis.live.net/v5/<folder>/files to get the list of files. According to this document, there are a few new ways to call it:

It is possible that they somehow broke the old API. In that case we would have to re-write the OneDrive backend.

For the duplicate files, is there any sense to it? It appears that it just shows random sizes. One very ugly hack, would be to only pick the largest file. This should work, as Duplicati does not overwrite/append/modify files, so the largest file would be the latest.

That idea would hide any flaws that the OneDrive API has, which could lead to more headaches later on, or just work perfectly :slight_smile:

Would it be enough to log that the instance happened so if users starting having more odd OneDrive experiences they could check their log to see how often they were getting that issue?

Did you mean to say something like ā€œthe largest file would be the latestā€? :wink:

Iā€™m still a little (OK, a whole bunch) unclear as to what causes this. Does their API work fine until a certain number of files or requests happen or is it always broken and people just havenā€™t ā€œnoticedā€?

Yes :smile: I have edited the post.

That would be better than hiding it, but often people donā€™t check this. Also, reporting it all the time would be annoying.

From the reports here, it appears to have happened after an update around the 24th. I donā€™t think it is caused by a particular number of files, but maybe @patman can elaborate.

Patman and me have the same problem the same dayā€¦ I think something have changed/broken in one or some servers (or they are deploying something that brokes old api calls). I have another backup similar to this and, for now, is working ok.

As a curiosity: Two months ago I tried to setup another backup to onedrive and I canā€™t make it run. It begans right but at some point the server thrown an 502 error (I think remember). The solution: I used another account and problem solved.

Thanks for the confirmation - should we consider updating the topic title to something more OneDrive specific?

This might help other OneDrive users I expect will be bumping into this issueā€¦

I agree with the OneDrive title and to close my thread making reference to this.