Another "An item with the same key has already been added."

I’ve seen other topics with this error, related to OneDrive recently update, but I think my case is different.

I have a Ubuntu server that makes everyday backup of to sets of files to two onedrive accounts. Both of them have been working ok for the last 2 months, but one of them are stuck with this error since tuesday.

This is the error reported:

Fatal error
System.ArgumentException: An item with the same key has already been added.
at Duplicati.Library.Main.BackendManager.List () <0x40c86ba0 + 0x0016b> in :0
at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, IBackendWriter log, System.String protectedfile) <0x40c83870 + 0x0015f> in :0
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, IBackendWriter log, System.String protectedfile) <0x40c82e20 + 0x000cb> in :0
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) <0x40c828f0 + 0x001af> in :0

I don’t know how to recover the situation and what is the problem that causes it. Other problem is that if I want to recover some file from this backup, the error repeats and none is restored (although the UI reports that the restore was succesful)

What version of Duplicati are you using?

A failed restore but with a “success” message is definitely something @kenkendk would probably want to look into, regardless of the cause.

My version is 2.0.2.1_beta_2017-08-01

Some more data: I’ve tried to repair database, verify files, and recreate database, but in all cases the error appears.

At this point, I can’t do more restore test, because database is deleted, so I can’t get the list of files to restore, and can’t recreate database.

I guess the only solution is to delete all the uploaded and corrupted backup and rebuild and reupload the 400GiB backup, and give up the backup history.

I think the error is really “the filename appears twice”.
Can you look inside the folder on OneDrive and see if any files are found twice?

In Windows client, it shows no duplicate files.

I tried to remove the last files created (first in the last day the backup works, and then in the last week) and the same error appears.

Maybe it shows in the duplicati API call?

Yes, that is possible. You can run the list call with something like:

Duplicati.CommandLine.BackendTool.exe LIST onedrive://path?authid=xyz`

This will call the OneDrive API with the exact same code that is normally used inside Duplicati and will show the output. But for this problem, the error happens before the LIST operation fails, so the output will probably not reveal the raw list of files.

Yes, the error appears before. This is the output of the command:

Invalid type Microsoft.WindowsAzure.Storage.Blob.BlobEncryptionPolicy for instance field Microsoft.WindowsAzure.Storage.Blob.BlobRequestOptions:k__BackingField
Invalid type Microsoft.WindowsAzure.Storage.Queue.QueueEncryptionPolicy for instance field Microsoft.WindowsAzure.Storage.Queue.QueueRequestOptions:k__BackingField
Invalid type Microsoft.WindowsAzure.Storage.Table.TableEncryptionPolicy for instance field Microsoft.WindowsAzure.Storage.Table.TableRequestOptions:k__BackingField
Name Dir/File LastChange Size
Command failed: An item with the same key has already been added.
System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException (ExceptionResource resource) <0x4103b7c0 + 0x0002b> in :0
at System.Collections.Generic.Dictionary2[TKey,TValue].Insert (System.Collections.Generic.TKey key, System.Collections.Generic.TValue value, Boolean add) <0x40e341d0 + 0x0018b> in <filename unknown>:0 at System.Collections.Generic.Dictionary2[TKey,TValue].Add (System.Collections.Generic.TKey key, System.Collections.Generic.TValue value) <0x40e34190 + 0x00023> in :0
at Duplicati.Library.Backend.OneDrive.List () <0x40f701b0 + 0x002d1> in :0
at Duplicati.CommandLine.BackendTool.Program.RealMain (System.String[] _args) <0x40f54c50 + 0x00854> in :0

With my other OneDrive backup the command runs ok and the list is printed well.