Issue repairing database

Iid88 · June 18, 2020, 4:04pm

Hi, trying to repair a database that went corrupt but getting an error.

  Listing remote folder ...

Mono.Data.Sqlite.SqliteException (0x80004005): Abort due to constraint violation
UNIQUE constraint failed: Remotevolume.Name, Remotevolume.State
  at Mono.Data.Sqlite.SQLite3.Reset (Mono.Data.Sqlite.SqliteStatement stmt) [0x00096] in <03f5dc00126848489ffe244e568a5e02>:0 
  at Mono.Data.Sqlite.SQLite3.Step (Mono.Data.Sqlite.SqliteStatement stmt) [0x00046] in <03f5dc00126848489ffe244e568a5e02>:0 
  at Mono.Data.Sqlite.SqliteDataReader.NextResult () [0x00129] in <03f5dc00126848489ffe244e568a5e02>:0 
  at Mono.Data.Sqlite.SqliteDataReader..ctor (Mono.Data.Sqlite.SqliteCommand cmd, System.Data.CommandBehavior behave) [0x00051] in <03f5dc00126848489ffe244e568a5e02>:0 
  at (wrapper remoting-invoke-with-check) Mono.Data.Sqlite.SqliteDataReader..ctor(Mono.Data.Sqlite.SqliteCommand,System.Data.CommandBehavior)
  at Mono.Data.Sqlite.SqliteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x00006] in <03f5dc00126848489ffe244e568a5e02>:0 
  at Mono.Data.Sqlite.SqliteCommand.ExecuteDbDataReader (System.Data.CommandBehavior behavior) [0x00000] in <03f5dc00126848489ffe244e568a5e02>:0 
  at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader () [0x00000] in <fb434c75f80444cb80ba3a9212e0956f>:0 
  at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64 (System.Data.IDbCommand self, System.Boolean writeLog, System.String cmd, System.Int64 defaultvalue, System.Object[] values) [0x00061] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64 (System.Data.IDbCommand self, System.Boolean writeLog, System.String cmd, System.Int64 defaultvalue) [0x00000] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64 (System.Data.IDbCommand self, System.Int64 defaultvalue) [0x00000] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Database.LocalDatabase.RegisterRemoteVolume (System.String name, Duplicati.Library.Main.RemoteVolumeType type, Duplicati.Library.Main.RemoteVolumeState state, System.Int64 size, System.TimeSpan deleteGraceTime, System.Data.IDbTransaction transaction) [0x000c5] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Database.LocalDatabase.RegisterRemoteVolume (System.String name, Duplicati.Library.Main.RemoteVolumeType type, System.Int64 size, Duplicati.Library.Main.RemoteVolumeState state) [0x0000d] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x0030e] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run (System.String path, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00037] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal (Duplicati.Library.Utility.IFilter filter) [0x000ba] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00012] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass18_0.<Repair>b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00007] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.Library.Main.Controller.Repair (Duplicati.Library.Utility.IFilter filter) [0x0001a] in <80402d19122441e2bd57fbfcc5267c40>:0 
  at Duplicati.CommandLine.Commands.Repair (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x00030] in <c47f7ad6ab9944308264c2287272a0ec>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00342] in <c47f7ad6ab9944308264c2287272a0ec>:0 
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <c47f7ad6ab9944308264c2287272a0ec>:0

I found some very old references to a similar problem and tried to delete my sqlite backup file, but nothing works. I also upgraded from the latest beta to v2.0.5.107-2.0.5.107_canary, but that didn’t make a difference. Not sure how to fix this. Do I have to delete my old backups?

Iid88 · June 19, 2020, 2:31pm

Bump? I really don’t want to delete my old backups

ts678 · June 19, 2020, 7:21pm

Welcome to the forum @Iid88

What Storage Type is this on your Destination screen 2, and how easily can you get to it?
If this is fixable at all (prior cases didn’t go that well) it might require some hands-on file work.

What version was it before? I don’t think anything has changed in the repair code in awhile…
In addition to knowing what version it broke on, any idea what might have led to the problem?

What do you have now? Apparently remote backup. Any databases or backup of databases?
You can sort by date in the path seen on the Database screen to see if anything’s still around.

Iid88 · June 19, 2020, 7:42pm

What Storage Type is this on your Destination screen 2, and how easily can you get to it?

Google Drive, easy for me to get to.

What version was it before? I don’t think anything has changed in the repair code in awhile…

v2.0.5.1-2.0.5.1_beta_2020-01-18

In addition to knowing what version it broke on, any idea what might have led to the problem?

There was an error about duplicate files in the destination. The exact error seems to have rolled off of the stored logs.

What do you have now? Apparently remote backup. Any databases or backup of databases?

Nothing for this backup as far as sqlite databases. I have other backups that are intact and working fine against the same google drive, different path.

ts678 · June 19, 2020, 8:18pm

That’s exactly what worries me. The final error is talking about seeing duplicate file names.
Google Drive is one of the few destinations that can achieve that. The name is an attribute.

I was able to set up this error by making duplicate files using drive.google.com, however I could not break Duplicati unless I set up a full-access login. With a limited login, Duplicati only sees the files that it makes.

Your situation might be different, because I don’t know what made the duplicates. The web UI can tell you (look for Created … with Duplicati), if you can find the file. Here’s a way that worked for me to find my first error. I hope you don’t have a whole series of duplicates. That would take a lot of looking through the files.

In another tab, you can run About → Show log → Live → Profiling while doing the database Delete and Repair. If it keeps scrolling (sometimes it stops – maybe there’s a limit) click on the final error (which will look familiar to you, as the one you hit), then look on lines below to see if you see an INSERT of same file.

EDIT:

Here are Details from the right hand side of the screen (after you turn that display on) of original and copy:

The copy was by right-click Make a copy and right-click Rename. You can see the difference in Created.

Iid88 · June 19, 2020, 8:46pm

The file was created three times, at the exact same time, exact same byte size. They appear to be completely duplicate, though I would have to download to find out. They were also all created by duplicati according to the UI.

Edit: I should mention that the network I’m on, I’ve been working on diagnosing a packet loss issue to the ISP that recently disappeared. Maybe related?

ts678 · June 19, 2020, 8:57pm

It’s too bad the timestamp only goes to minutes. There might be a better one by other tools. I’m not sure…

Duplicati “should” not upload any duplicate. If a file upload gives an error, file content gets a new file name.

If you still had a database (or emails or other reports), there would be signs like “RetryAttempts” in full log.

Regardless, next step is probably to look at the files, convince yourself they’re the same, and get rid of the duplicates. If you want to be careful and allow reversal (not sure why), just rename by adding a new prefix.

If you need to compare content carefully, fc /b on Windows or cmp on Linux should do. Or compare hash.

Iid88 · June 19, 2020, 9:03pm

Yeah I did a Get-FileHash with powershell and they’re all equal… Weird! Looks like I did have the original error in a report:

Failed: Found remote files reported as duplicates, either the backend module is broken or you need to manually remove the extra copies.
The following files were found multiple times: duplicati-i2f14913211474006ac0857af213acd62.dindex.zip.aes
Details: Duplicati.Library.Interface.UserInformationException: Found remote files reported as duplicates, either the backend module is broken or you need to manually remove the extra copies.
The following files were found multiple times: duplicati-i2f14913211474006ac0857af213acd62.dindex.zip.aes
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.String protectedfile) [0x00179] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.PostBackupVerification () [0x00062] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <8f1de655bd1240739a78684d845cecc8>:0
at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011c] in <8f1de655bd1240739a78684d845cecc8>:0

Log data:
2020-06-18 10:42:36 -04 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-DuplicateRemoteFiles]: Found remote files reported as duplicates, either the backend module is broken or you need to manually remove the extra copies.
The following files were found multiple times: duplicati-i2f14913211474006ac0857af213acd62.dindex.zip.aes
2020-06-18 10:42:36 -04 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Found remote files reported as duplicates, either the backend module is broken or you need to manually remove the extra copies.
The following files were found multiple times: duplicati-i2f14913211474006ac0857af213acd62.dindex.zip.aes
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.String protectedfile) [0x00179] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.PostBackupVerification () [0x00062] in <8f1de655bd1240739a78684d845cecc8>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00eee] in <8f1de655bd1240739a78684d845cecc8>:0

Nothing in previous reports though I only have errors. I’ll delete the dupes and repair.

Iid88 · June 20, 2020, 1:11am

The backup went through. Hoping everything is well, seems to be. Thank you much for your help!