Errors While Trying to Fix Broken/Missing Files

Support
1

I’m running Duplicati - 2.0.5.1_beta_2020-01-18 on Windows 10 Pro v1903, build 18362.959 and trying to deal with errors related to missing files. I’ve tried assorted recovery approaches, including deleting the local database and attempting to repair it (repeatedly) at the suggestion of the UI. I’ve also tried listing and purging broken files. I’m currently stuck on this last effort and keep getting errors in the process, for which I have a log file (Start, Checkpoint, and End lines are included to indicate durations of steps; also, various missing parameters like usernames and passwords are actually specified in the commands, they’re just removed here for security):

C:\Program Files\Duplicati 2>cd “C:\Program Files\Duplicati 2”
Start: 01/14/2021:15:30:42.17

C:\Program Files\Duplicati 2>duplicati.commandline list-broken-files
“aftp://unraid//mnt/?auth-username=uname&auth-password=redacted”
–full-result
–passphrase=“redacted”
--send-mail-from="redacted@gmail.com"
–send-mail-any-operation=true
--send-mail-to="redacted2@gmail.com"
--send-mail-username="redacted@gmail.com"
–send-mail-password=<password>
–send-mail-url=“smtps://smtp.gmail.com/?starttls=always”
–backup-name=“<file backup name>”
–dbpath=“\Duplicati\OEUWLIUHEC.sqlite”
6 : 12/30/2018 6:31:54 PM (435 match(es))
<list of 206 files>

Failed to send message: System.NotImplementedException: The method or operation is not implemented.
=> The method or operation is not implemented.
Checkpoint: 01/14/2021:15:32:29.88

C:\Program Files\Duplicati 2>duplicati.commandline purge-broken-files
“aftp://unraid//mnt/?auth-username=uname&auth-password=redacted”
–dry-run
–passphrase=“redacted”
–backup-name=“<file backup name>”
–dbpath=“\Duplicati\OEUWLIUHEC.sqlite”

End: 01/14/2021:15:33:43.05
code = Unknown (-1), message = System.Data.SQLite.SQLiteException (0x80004005): unknown error
No transaction is active on this connection
at System.Data.SQLite.SQLiteTransactionBase.IsValid(Boolean throwError)
at System.Data.SQLite.SQLiteTransactionBase.Rollback()
at Duplicati.Library.Main.Operation.PurgeBrokenFilesHandler.Run(IFilter filter)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action1 method) at Duplicati.Library.Main.Controller.PurgeBrokenFiles(IFilter filter) at Duplicati.CommandLine.Commands.PurgeBrokenFiles(TextWriter outwriter, Action1 setup, List1 args, Dictionary2 options, IFilter filter)
at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action1 setup, Boolean& verboseErrors, String[] args) at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action1 setup, String args)

End: 01/14/2021:15:33:43.05

Any suggestions for what to try next? Should I just setup a new backup with a new destination and then delete the old one, assuming that the new one completes successfully?

2

was more informative than the other two times this quite unusual list-broken-files message was seen.
Not seeing in advance what will happen is a little uncomfortable, but you can possibly find out later by
The COMPARE command if things are intact enough to run. Start-over is also good, but loses history.

I’m not sure how to debug your error message unless you can get in deep, or it’s reproducible. Maybe Creating a bug report and posting a link to it would help an expert, but I think the database format isn’t directly testable with that command, because it has to look at your backup files which I won’t ask for…

One way to possibly get some information detail is to add a –log-file=<path>, and run for another error.

3

I guess I should also ask what this did. And sorry for not seeing your purge-broken-files had also failed.
EDIT: but you used --dry-run and there are some other reports of it giving that error under that option…
I’d ordinarily say be careful with the backup files, but if you can’t Recreate from them it may be too late.
Fix error encountered during dry-run of purge-broken-files #4380 got in 2.0.5.112 (canary) 2021-01-20:

  • Fixed error that occured when using the dry-run option of purge-broken-files, thanks @warwickmm

EDIT to fix the release with the fix.

4

I ran with a --log-file specified. It’s empty.

Re: “What did deleting the local database and attempting to repair it do?”: That gave either:

Recreated database has missing blocks and 1 broken filelists. Consider using “list-broken-files” and “purge-broken-files” to purge broken data from the remote store and the database.

or:

The database was attempted repaired, but the repair did not complete. This database may be incomplete and the backup process cannot continue. You may delete the local database and attempt to repair it again.

5

I decided that I had nothing to lose by running --purge-broken-files (my backup is already screwed) and did that. This appears to have been successful

Uploading file (1.38 MB) …
Deleting file duplicati-20181230T233154Z.dlist.zip.aes …

I’ll verify files next.

Unexpected difference in fileset version 5: 1/14/2019 1:30:01 AM (database id: 2), found 17128 entries, but expected 17304

Repair:

MainOperation: Repair
ParsedResult: Success
Version: 2.0.5.1 (2.0.5.1_beta_2020-01-18)
EndTime: 1/26/2021 6:05:24 PM (1611702324)
BeginTime: 1/26/2021 6:05:09 PM (1611702309)
Duration: 00:00:14.3191364
MessagesActualLength: 4
WarningsActualLength: 0
ErrorsActualLength: 0
LimitedMessages: [
2021-01-26 18:05:10 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started,
2021-01-26 18:05:16 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2021-01-26 18:05:23 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (2.59 KB),
2021-01-26 18:05:23 -05 - [Information-Duplicati.Library.Main.Operation.RepairHandler-DatabaseIsSynchronized]: Destination and database are synchronized, not making any changes
]
LimitedWarnings:
LimitedErrors:

Backup:

Failed: Unexpected difference in fileset version 5: 1/14/2019 1:30:01 AM (database id: 2), found 17128 entries, but expected 17304
Details: Duplicati.Library.Interface.UserInformationException: Unexpected difference in fileset version 5: 1/14/2019 1:30:01 AM (database id: 2), found 17128 entries, but expected 17304
at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass34_0.b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action1 method)

Log data:
2021-01-26 18:07:15 -05 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Unexpected difference in fileset version 5: 1/14/2019 1:30:01 AM (database id: 2), found 17128 entries, but expected 17304
at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass34_0.b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.d__2`1.MoveNext()
— End of stack trace from previous location where exception was thrown —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Duplicati.Library.Main.Operation.BackupHandler.d__20.MoveNext()

Seems like I’m back to “create a new backup.” I’m not that concerned about losing file history.

Is there some other version that I should be running?

6

Only if you’re willing to run Canary, which generally gets better, but occasionally gets broken.
v2.0.5.111-2.0.5.111_canary_2020-09-26 ran well for me for over three months but 2.0.5.112
is so new that it hasn’t had much time to prove itself. Such proof only lasts until next release.

Right now 2.0.5.1 is over a year old, and it’s annoying to have fixes that aren’t in any Beta yet.