Lost internet in the middle of an auto-Compact. Is there anything special that I need to do to recover? From this post, it seems like the Compact process is fairly failsafe. Is there any manual command I should run to verify the integrity of the local database or remote fileset?
Also, the auto-Compact was part of a backup job, which completed before the Compact began (as verified in the log), so why is there no log in the GUI for the completed backup job?
The below log snippets just show that the internet failed as it was trying to download a DBLOCK to compact. Which should be a good place to fail, I believe.
so if you fall into that, you’ll be able to see the actual Delete and so can just run the Repair.
I suppose a slightly more conservative way to get a preview of things is “Verify files” button.
Unless you have something unusual configured, just running Backup does a similar check.
Whoa. I read the bug report but didn’t realize that. I am not sure what to do. Please tell me exactly what needs to be done to restore integrity (if it is even possible) before the next backup run. I really don’t want to restart my entire backup and lose all the history. Thanks.
Edit: Uh-oh. Ran ‘Verify Files’ and got this: ‘Found 3 files that are missing from the remote storage, please run repair’. Will clicking ‘Repair’ completely fix the problem, or do I need to manually edit the database?
Edit 2: It appears that to completely repair the damage, the following SQL commands need to be run (from ts678’s post in the bug report):
SELECT “Name” FROM “Remotevolume” WHERE Type == “Index” AND “State” == “Verified” AND “ID” NOT IN (SELECT “IndexVolumeID” FROM “IndexBlockLink”)
and then
UPDATE “RemoteVolume” SET “State” = “Deleting” WHERE Type == “Index” AND “State” == “Verified” AND “ID” NOT IN (SELECT “IndexVolumeID” FROM “IndexBlockLink”)
The Deleting dindex patch should be committed immediately. Manually editing a SQL database is really not user-friendly.
Edit 3: Should the SQL commands be run before, after or instead of the Repair?
2022-10-10 11:26:26 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started
2022-10-10 11:26:26 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation List has started
2022-10-10 17:18:49 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started
2022-10-10 17:19:07 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()
2022-10-10 17:19:08 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (443 bytes)
2022-10-10 17:19:09 -04 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Deleting: duplicati-b7bd678b7c2a64f91b2c3f4e7342d60e6.dblock.zip.aes
2022-10-10 17:19:09 -04 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Deleting: duplicati-bd0f90a535cf64895840dfaac1d17cea1.dblock.zip.aes
2022-10-10 17:19:09 -04 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Deleting: duplicati-bbd0666fce32e4eda8ecf382df9543ce8.dblock.zip.aes
2022-10-10 17:19:09 -04 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Deleting: duplicati-20221010T043856Z.dlist.zip.aes
2022-10-10 17:19:12 -04 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-b315a2efb3ec84d52bdf8850669851e7a.dblock.zip.aes
2022-10-10 17:19:12 -04 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-i5474994074284f02a744c1a82d0fd339.dindex.zip.aes
2022-10-10 17:19:12 -04 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i1ddf470ccbd343c0aae9bd42b735858b.dindex.zip.aes
2022-10-10 17:19:12 -04 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-iac3e0c3730da44bfbed3a7510e822102.dindex.zip.aes
2022-10-10 17:19:12 -04 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingFile]: Missing file: duplicati-i3cb62aba375a45b6ae36fa4e0f3dbd10.dindex.zip.aes
2022-10-10 17:19:12 -04 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteFiles]: Found 3 files that are missing from the remote storage, please run repair
All three of your files seen as missing were actually deleted, according to log comparison below:
The Delete are the end of one compact cycle. The Get start the next one, so is a good time to fail,
except for the forgetting that Delete had happened (forgetting is due to unfinished DB transaction).
Let’s also look at the non-dindex messages to see if there seems to be a similar cleanup problem:
This doesn’t have a match in posted log, but it might be before the posted part. Search log if you wish.
I’ve been looking at the lengthy old issue to see if a good way to fix this is known. Not clear one exists.
Just running Repair pushed up some dindex files, but they were then extra, with no associated dblock.
This bothered DB recreate. I was wondering if going hands-on in DB was needed, and it looks like you
already are in there (in later edits). The basic idea would be to get rid of the Remotevolume table rows thinking they know those dindex files, but first make sure that ID isn’t in IndexBlockLink IndexVolumeID.
Make a copy of the database before trying such experiments. I’ll look at your SQL, but mine’s not great.
You seem to have a similar idea, but details matter, so rather than setting State I call for manual delete.
Having a tested procedure that works on the issue test case would be nice. Maybe yours would be it…
Even better, if you do C# and GitHub, would be to test the proposed patch and submit the pull request.
There is such a shortage of developer volunteers that it might sit, but at least it could be readied to go.
Eventually the original author might show up if nobody else does it first, and would be a good reviewer.
Is it safe to run the second command and set the state to Deleting or do you have an alternate command to simply delete the records? BTW, I have not yet run ‘Repair’. Should I do that, or did ‘Verify Files’ already do what ‘Repair’ would have.
Yes, the other dindex files you couldn’t find in the log were successfully deleted:
That explains why the context for the first statement was missing. It was not copied.
can find extra dindex files, then if the user wishes, they can be set up for remote deletion as described previously:
That part seems to have worked, naming your three confused files.
Your log output shows outputs of code that handles Deleting State.
That’s in RemoteListAnalysis, which is run by VerifyRemoteList which I assume Verify files runs.
You’ve also pushed the button and seen it in action. Well, I guess you can run the second SQL, then
push the button again to see what it does, then push it again to see if things now seem to be happy…
I’m not going to post SQL without testing, but SQL DELETE Statement suggests how to write the syntax.
An advantage of setting State Deleting is that the code looks more careful about files’ presence/absence.
You can see if the next Verify files comes up clean. If so, the similar test done at backup start should.
Based on my experience, even if the backups are running and everything seems to be fine. The backup can be corrupted. Only way to be sure is to run full restore test. I do that regularly and unfortunately find every now and then corrupted backups.
The missing files issue can be remedied by repair, it works quite well. And I earlier said in some discussion that the issue is probably fixed, but it’s not. I got it again just yesterday. Similar experience to the opening post.
Aside from getting backups going again, does Repair break database recreate for you as it did for me?
This method basically causes the problem where things seem fine until you find they’re not quite fine…
That’s why I’m recommending against it, although running SQL in a DB browser is a very ugly cleanup.
EDIT: On the other hand, post below seems to think Repair’s extra dindex files were OK. Who knows?
Maybe the results vary depending on the exact nature of the failure? This could use more investigation.
… as evidenced not only by that, but by the issue about it still being open. Feel free to read through that.
If everything else tracks well, you’ve hit the open issue that nobody’s volunteered to fix in several years.
Issue has been hugely studied and even got steps to reproduce, but developers have been unavailable.
Duplicati very much needs developers. Even non-expert could play with the test case and proposed fix.
As a community effort, Duplicati relies on volunteers in all areas. Right now, many areas are super low.