I have two backups configured on my desktop Win 10 PC - one to a network share on another PC, the other to OneDrive, same set of source files for both backups. All of a sudden today the backup to OneDrive has started failing with the following message
Failed: An item with the same key has already been added.
Details: System.ArgumentException: An item with the same key has already been added.
at Duplicati.Library.Main.BackendManager.List()
at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)
at Duplicati.Library.Main.Controller.<>c__DisplayClass16_0.<Backup>b__0(BackupResults result)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
Iāve tried verifying files, deleting and repairing the database, all without any success. There is plenty of free space on the destination (~400GB). I uploaded a bugreport.zip file. Any ideas as to anything else I could try.
This sounds to me like somehow youāre getting duplicate files being reported from your destination.
Iām not sure how to debug that (itās too bad the error didnāt include they offending ākeyā). I know it sounds trite, but have you rebooted (or restarted the server - either Tray Icon or service) since the error?
Yes, I had rebooted, should have said so. I should also mention that I have two other PCs that Iām backing up to OneDrive that havenāt shown any problems. The PC with the problem had been backing up to OneDrive for about 3 months without a problem until yesterday.
Is there any logging capability beyond that in the web UI that might give a clue?
I donāt think itās related to that issue - that one is about including the OneDrive sync folder in the source set, seems redundant to be honest.
I was able to do some debugging and your suspicion was correct. It would appear Microsoftās API is badly broken. The exception is being thrown in the OneDrive.List method. I added some logging and was able to see that the total number of records returned was correct, 16887, i.e it matches the number of files reported by the OneDrive Android app, but each time I ran it I only got around 11K or 12K unique file names. For each record I logged both the file name and the record number and when I sort the log I see a lot of lines like
I apologize if youāve already included this, but I couldnāt find what version of Duplicati youāre using.
If itās the beta version, how would you feel able trying a portable "install"of a canary version to see if it reports the same issue?
Iām just guessing random stufftnuff here but you actual fine count of 16,887 is just a bit over a nice round 16,384 (nowhere near tnyint or into, but still oddā¦) so perhaps weāve got a variable thatās just not big enough to handle all your files.
If I have time in the next day our two Iāll try to set up OneDrive backup with a small block size replicate your remote file count.
I think we have the same behaviour, but my backup have 23321 files, so the variable overflow I donāt think it is the problem (at least in the duplicate side).
If Iām not wrong, the problem appears this tuesday, right? Maybe MS is making some change to the service or have problems with some servers.
I have another backup to onedrive with similar amount of data. I will wait some days to see if the problem disappears or appears in the other backup.
Glad itās not just me - The first backup that failed for me was also Tuesday. Iām pretty certain that there was no significant increase in the volume of data being backed up since the previous weekend (Oct 14th) , itās unlikely that a significant increase in the number of files occurred on the backend anytime in the week before the failure, so I have to suspect Microsoft broke something.
Looking at the OneDrive.List method the count and offset variables are both declared as int and as the build target is āAny CPUā and Iām running Windows 10 the variables should be 64 bit integers so I think we can rule out an overflow issue, at least on the Duplicati end.
I havenāt been able to find reports online of this issue with the API. One question that does come to mind is that as the Android app is working is it using the same API?
I am using the current beta release - I tried the latest canary build and see the same issue.
As a side note I should mention I was able to clone and build this project with no issues whatsoever - very nice!
Duplicati current calls something like https://apis.live.net/v5/<folder>/files to get the list of files. According to this document, there are a few new ways to call it:
It is possible that they somehow broke the old API. In that case we would have to re-write the OneDrive backend.
For the duplicate files, is there any sense to it? It appears that it just shows random sizes. One very ugly hack, would be to only pick the largest file. This should work, as Duplicati does not overwrite/append/modify files, so the largest file would be the latest.
That idea would hide any flaws that the OneDrive API has, which could lead to more headaches later on, or just work perfectly
Would it be enough to log that the instance happened so if users starting having more odd OneDrive experiences they could check their log to see how often they were getting that issue?
Did you mean to say something like āthe largest file would be the latestā?
Iām still a little (OK, a whole bunch) unclear as to what causes this. Does their API work fine until a certain number of files or requests happen or is it always broken and people just havenāt ānoticedā?
That would be better than hiding it, but often people donāt check this. Also, reporting it all the time would be annoying.
From the reports here, it appears to have happened after an update around the 24th. I donāt think it is caused by a particular number of files, but maybe @patman can elaborate.
Patman and me have the same problem the same dayā¦ I think something have changed/broken in one or some servers (or they are deploying something that brokes old api calls). I have another backup similar to this and, for now, is working ok.
As a curiosity: Two months ago I tried to setup another backup to onedrive and I canāt make it run. It begans right but at some point the server thrown an 502 error (I think remember). The solution: I used another account and problem solved.
- The first backup that failed for me was also Tuesday. Iām pretty certain that there was no significant increase in the volume of data being backed up since the previous weekend (Oct 14th) , itās unlikely that a significant increase in the number of files occurred on the backend anytime in the week before the failure, so I have to suspect Microsoft broke something.
I have edited the post.