First bit more background information, I did check the logs. Which initiated the need to run the repair process:
This was the initial error:
Error Log:
System.IO.InvalidDataException: Found 3 file(s) with missing blocklist hashes
at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
at Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass34_0.b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass3_0.b__0()
at Duplicati.Library.Main.Operation.Common.SingleRunner.d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action1 method)
at Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
at Duplicati.CommandLine.Commands.Backup(TextWriter outwriter, Action1 setup, List1 args, Dictionary2 options, IFilter filter) at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action1 setup, Boolean& verboseErrors, String args)
at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String args)
After this repair gave the error I mentioned earlier.
And after rebuild, it’s all good again. Backus run and repair won’t complain. I didn’t expect this good results, based on earlier bad experiences. Yet, as mentioned it shouldn’t have required rebuild at all.
Rebuild was quite slow, but with fast ssd and high speed networking, it took less than 24 hours. And for sure it did download the dblock files. Yet the time spent downloading is really small compared to the time of updating the database. - I still think it’s done in some quite inefficient way. My personal guess is that the system runs way too many small transactions calling commit in between.
Finding right balance between simple logic, high performance and reliability is hard balancing problem. I do deal with that daily.
@ts678
I’m very familiar with SQLite, no issues there.
Yeah, finding root cause. Oh joy. I often raise the point that if system is designed and implemented correctly, this kind of problems shouldn’t ever happen. If everything didn’t to right, well then stop, don’t corrupt things. -> This is my personal motto when dealing with production systems. Yes, it annoys people when things are “unreliable” in a way, that those stop working. - Sure, that’s right. But the worse option is that someone figures out after two years that all the data they got is corrupted, invalid, misleading or unusable in some way. -> Yet this does happen, when someone wants something very cheaply and quickly. Simple logic -> stuff out, without any validation.
Slide database back in, I’ve been using the CLI version all the time. Gotta check out how I’ll stick the database to the UI instance on another system. Yup, seemed to be working beautifully, I created new backup set with manually configure DB path, and replaced the DB with the “corrupted” one.
Why I did the corrupted in quotes? Well, pragma integrity_check says, it’s all good.
Report generation went well. Actually the filenames in the backup were so general that basically even if those would leak, it wouldn’t matter at all. Anyone could guess the filenames based on my user account name and making some very quick Googling.
I’m sorry if this isn’t according the protocol. But the bug generation didn’t provide direct submit option. ( I guess you don’t want to store all the potentially sent databases, hah). But I’ll send the link to the bugreport.zip link to both of you guys in PM. - Let’s hope it’s useful.
Thank you for being so helpful. I wouldn’t go through this, if I wouldn’t believe Duplicati is worth of the effort.