I’m running a small (~25gb) Sia test backup - intentionally making (or allowing myself to make) standard stupid-user mistakes or ‘fallible equipment’ mistakes - the norm.
I was about 18GB through the backup just now and I decided I wanted to run something else real quick, so I clicked the “stop” button, and for once i chose “stop now”.
When I returned to the backup - now it gave me an error along the lines of
Found inconsistency in the following files while validating database: (one large zip file in my backup set), actual size 133452342, dbsize 0, blocksetid: 73 . Run repair to fix it.
I ran repair - repair seems to conclude normally and gives an output of “Destination and database are synchronized, not making any changes”
I attempted to run the backup job again - same error. Attempted to run repair again - same result.
Then I noticed in my Sia browser that the last dblock file only got to 0.9x (basically incomplete upload) - so i went ahead and deleted it.
Now whenever I attempt to run repair, it fails instantly and I get an error message saying “Got 1 error(s)” - i click “Show” and I get taken to the ‘General’ tab of the log data for that backup set, and there’s no new log entry, no error message or explanation. -_-
This is making me really nervous about what would happen if my system goes down i.e. due to power failure when backing up to my big ‘official’ B2 backup set - I can’t afford to lose a terabyte of backups because some file didn’t complete and the GUI repair mechanisms never actually work…
That sounds like a genuine problem. I have the database downloaded and will see if I can fix it. It does not have anything to do with the partially uploaded dblock directly, but it could be that the partial upload of the file, combined with the abort, is causing the problem.
That sounds strange. It should add a new entry for each time you attempt to run it.
In any case, it is most likely saying that something broke because you are missing a file.
You cannot continue the backup if the remote store is broken; that would make it impossible to guarantee the integrity of the backup. If you want to recover from it, you can run purge-broken-files which will clean the backup sets from all files that depends on the missing dblock file and then making the backup consistent again.
This is why Duplicati is still in beta: sometime you need to do some manual intervention. You can still restore from a broken backup (you get whatever is there, and since the previous backup succeeded, you can always get that).
In a complete fail, there is Duplicati.CommandLine.RecoveryTool.exe bundled, which will do the restore directly on the files without any database required.
I apparently need help in doing this. When I go into the “commandline” section for the backup job in question, and select the Purge Broken Files option (or the List Broken Files option) at the top, then click the ‘Run…’ button at the bottom, I get what seems to be an error message along the lines of
Found 2 commands but expected 1, commands:
“sia://localhost/DuplicatiBackup2?sia-password=9999&sia-targetpath-…”
"D:\TestBackupDirectory"
Return code: 200
Understood. I hope I’m not coming across as pestersome - just doing my best to help troubleshoot so we can get the product maturity up to the point where I don’t have to sweat a little bit when recommending it to family & friends who aren’t as tech savvy as me
Yes. The commandline part is not the most user-friendly part, but it was faster than actually building the UI for list/purge broken files.
When the UI shows up, it is pre-filled to do a backup. When you choose one of the other commands, you need to adjust the contents to fit.
For the list/purge broken files, you simply need to remove the source folder (D:\TestBackupDirectory in your case).
Thanks for the positive approach!
That is exactly what I want: to fix the problems so it can be used by the average user. And to get there, I need the feedback from you and others that explain where it does not work so well.
That seems to have worked.
When I run the ‘purge…’ command, this is the result:
If I try to run the backup job on the same set, though, I get the same result that I described previously (error message with a “show” button which doesn’t actually show anything in the log).
EDIT: I just figured out the discrepancy with the “show” button not showing anything. When I press “Show” on the error pop-up, it takes me to the log for that backup set. But the error message is actually in the global log section. The error message it shows matches the pop-up, but of course still just says the same stuff.
Yeah. Though I’ve already nuked that particular backup job and started over, so I don’t have terribly much more detail for you there - I’ve been experimenting a bit more with other Sia backup jobs and I’m getting more of a feel for what is going on. It mostly has to do with aborting uploads, which is necessary occasionally as the Sia daemon craps out constantly and refuses to upload.
Basically after aborting an upload when there’s a dblock stuck at 0%, it takes several steps to recover - repair won’t work by itself, because it doesn’t notice the uploaded dblock has no data. Purge broken files needs to be run, and there’s some weird case where like you need to remove the broken file first, purge broken files, or something very specific - next time it happens I’ll write down the steps I took, it’s been pretty fumbling so far.
I’m having a very similar problem with purge-broken-files, with SFTP as my storage (and Duplicati beta, running on Windows 7).
I run a “repair” from the GUI and get:
Fatal error
Duplicati.Library.Interface.UserInformationException: Found 4 files that are missing from the remote storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
at Duplicati.Library.Main.Operation.BackupHandler.Run(String sources, IFilter filter)
Then I run a commandline (at the Windows command prompt) to purge files and I get:
No broken filesets found in database, checking for missing remote files
Listing remote folder …
Marked 4 remote files for deletion
Found no broken filesets, but 0 missing remote files
But it doesn’t seem to have deleted the 4 files that it marked. I can run the command as many times as I want, and it does the same thing each time.
In retrospect I’m pretty sure the cause of the issue I originally reported in this thread was the fact that the Sia client would randomly timeout during uploads and/or downloads, and fail to report this to Duplicati, causing all sorts of random sync issues.
