Restore failing and dIndex files failing in rebuild

Support
1

I have an external hard drive that crashed and am trying to restore the data from a cloud backup that is about 3 months old - S3 Compatible storage.

Apparently Duplicati stopped running after one of my OS upgrades. I found out a way to get it started up in the newer versions of MacOS thanks to a post on a forum. I am running 2.0.6.3_beta on MacOS Ventura currently.

I (think) I have an intact database, but when I tried to just test restoring a single file, it said successful in the UI, but the file did not restore. There is an error logged “Found 3462 remote files that are not recorded in local storage, please run repair” I started a repair but it was taking forever. I tried to run another backup to see if I could get the data insync, but it failed because the drive (that failed) was missing. I set the options to allow missing source, but then the backup still failed because of the same error about remote files not recorded in local storage.

I thought I would try just doing a restore directly, so I connected to the storage, it ran for a little bit, but alowed me to select the folders I wanted to restore, which is basically everything, and now it is running a “Recreate database”. It is probably 35% done on the progress bar, but it has been about 5 hours. I wouldn’t normally be concerned, but there are a ton of errors logged “Operation Get with file duplicati-xxxx.dIndex.zip.aes attempt 1 of 5 failed with message: The specified key does not exist.” The expanded error is “Amazon.SE.AmazonSeException”,“Message”:“The specified key does not exist.”,“Data”:null,“InnerException” blah blah “The remote server returned an error: (404) Not Found” After the 5 tries I end up with Failed to process Index file X

I would normally think I didn’t set something correct on the Bucket, but every so often I see one showing up as Operation Get - Completed:

I searched for one of the retry files in the bucket, and it is present. Not sure what to do at this point. Will the rebuild complete even with these errors? It seems to keep moving on to the next files and saying processing index list volume x of x

I have considered maybe it’s taking longer because the files are in cloud? Should I download my whole bucket files locally and try to run it then? At this point I don’t care about the backup, I just want to get my files back

2

To get your files back, I think that the best method is to start from the copy of the backup you did before starting your experiments: using a more recent database with an old backup, starting a repair but killing it in the middle. All this has a sever risk of corrupting your backup, so it’s better to start again with the saved version, and build a new fresh database from it.

3

I’m not sure I know what you mean by starting from the copy of the backup before I did experiments. As far as I know, the actual files in the destination (cloud repository) have not been touched - they still have older modification dates. My most recent successful backup was about 3 months ago. Are you saying I should try to restore an earlier version than most recent? Not sure what you mean by starting again with the saved version.

4

Eventually it might come down to saying which one, but let’s attempt generic approaches awhile.

Although pieces to this puzzle don’t quite fit, is it cold storage where retrieval needs special help?

Meaning it’s visible? Is it also downloadable with something? A GUI tool is an easy first test. Best:
Duplicati.CommandLine.BackendTool.exe using Export As Command-line URL if you can get one.

Do you have an ability to sort by date to see how well they line up? My first worry was a mismatch.
Running a Repair with a mismatch can make things match by force, e.g. extra files can be deleted.

Does this still exist as a copy? It has useful data, but just running Recreate on it means it’s gone…

5

Yes, download locally would be more secure since you would have a copy of your backup - it may also work around the strange behaviour of your backend.

6

Local is also a little easier to look over, for example to see if the dlist file series ended 3 months ago.
Direct restore from backup files can get a better look, including seeing the files available in the backup.
This experiment could have been done remotely too, but experiments are faster and safer on a copy…

Direct restore should run a bit faster because database is partial and temporary for specific restore.
Using local files also sets up for Duplicati.CommandLine.RecoveryTool.exe, if that becomes necessary.
Alternatively, maybe having answers to previous questions will be enough to understand current oddity.

7

Thanks. Give me a minute to supply some more feedback. To clarify I am currently running a “direct restore from backup” and the database that is rebuilding right now is “temporary” according to what the menu item it says. I did successfully see my files in a list that I selected, but once I hit the restore button then it switched to building a database, which is still currently running.

8

Live log at About → Show log → Live → Verbose is a reasonable level to watch at. It’s more than Retry level shows without being absurdly large like Profiling would be. You can change levels as you like, but please leave at least Warning up because Warning and Error messages may come in handy if needed.

70% to 100% is where it downloads dblock files looking for blocks that it needs. 90%+ is especially bad, where an exhaustive search is done. That can be lots of downloading. Earlier is more process intensive.

9

I’ll try to answer the questions that have been asked:

Not aware of any cold storage retrieval needs. Yes, one of the dindex files I was looking at that was retrying was visible in the explorer I was using to browse, and I was able to download it just fine. It appears the dates line up. I sorted by last modify date and it was 8/29 just like my last successful backup. Yes, the local database is still available. Actually, I can’t remember now if I actually tried hitting the repair or not. I thought I did, but I think really what I had first tried was just a direct restore which I ended up canceling. I know I tried running another backup that failed thinking that would get things to sync up, but it just failed.

As you mentioned before about doing direct restore, that is what I am actively doing. It has been running for over 24 hours. It did show my files list and I select the most recent backup and selected the majority of the folders and started. It is “building partial temporary database” and looks like it’s a smidge over 50%. I looked at the logs again and I am still getting those retry errors, but it keeps chugging along processing index list volume 13311 of 20454 even though it appears 95+% are failing with the specified key does not exist error. Will it download the dblocks to where the restore location is? I’m not sure I have enough storage available for it to hold dblock AND the unpacked files. If you think I should just download the bucket locally and go from there and try again, I can do that. Also I think I still have the database to work with. Maybe I can copy that and then actually try a repair?

10

The original error on the restore using database when I tried to just fetch one file was “Found 3462 remote files that are not recorded in local storage, please run repair” I was nervous when I saw that thinking that if I repaired that it would somehow make files I deleted not available or something, which had me trying the direct restore option. Just a little more background. So yes, I don’t know that I actually tried hitting the repair button.

11

I have no idea why your destination is doing that, so this is speculative…

I think it downloads to tmpdir (default or explicit) and distributes its blocks.
Recovered files are rebuilt from blocks, and files can have the same ones.
How the restore process works, but that won’t start until the DB is created.
Success seems unlikely if few of the dindex files are actually downloading.

Avoid repair for the reasons stated, until there’s some clarity on destination.

If you still have the original database, you can copy that to the current Local database path on the Database screen if you’re certain there’s nothing worth keeping there, or move temporarily to copy of original database by filling in its path and using the Save button (after noting where you moved from).

Turn on live log at Warning level, then click Verify files which is similar to the test at backup start.
This will probably get a bunch of names which you can sample-verify, and also look at for any pattern.

You will also get your old log files back, so you can see what your last good backup thought it had for KnownFileCount under BackendStatistics under Complete log. Any relationship to current view?

If you’re technically ambitious, you can look in copy of original database with DB Browser for SQLite in Remotevolume table. That’s what Duplicati is expecting, and if it’s Verified it was actually seen once.

Do you know the current Destination file count, courtesy of destination Web UI or some other S3 tool?
Possibly you wound up creating an empty Duplicati database, which would be expecting no files at all.
It would also be kind of small (likely well under a MB), not have any logs, and look odd in other ways…

EDIT:

Verify files, just like a backup, will do a list operation to see whether expected files are at remote. Clicking on the list will expand it, in case you need another reference to track down big discrepancy…
Possibly you have to do this in <job> → Show log → Remote (or maybe live log will do it – you can try).

12

Okay, I killed the job. Copied the local database to another location, then ran “verify”. It just finished with a whole page or so of “Extra unknown file” errors in the warning. They are all dlist files. The final error message was “Found 3071 remote files that are not recorded in local storage, please run repair”

13

What database was that run on? The suggestion was the old one, but it’s not mentioned in post.

Does this still seem to be the old database still sitting where it is, complete with its old logs, etc.?
If so, making a copy would be good, before any accident happens that loses its previous history.
You can also proceeed with the other tests mentioned.

That’s better than the 3462 originally reported. Do you know what dlist files Destination holds?
You could look at Versions on home page for an opinion (but trust it less, as it’s a different DB).
You could also see how many an initial Restore dropdown has if you think you’re on the old DB.
Comparing dates will be possible, if you know destination names (but times on names are UTC).

Extra files are sometimes a symptom of an old database, e.g. when restored from image backup.
Unless you have upload-unchanged-backups in Advanced options (do you?) you’d also expect additional files such as dblock and dindex which are where newer backups would backup data.

If on the other hand the database is not the old one, there’s no telling what mismatches to expect.

14

The one I killed that was running 24+ hours was the temporary rebuild as a result of running the restore directly option. When I tried to do a restore from my local DB I got the “remote files that are not recorded in local storage” error.

There are 17 dlist files in my bucket, which correspond correctly with it showing 17 backup versions in the GUI.

I do not. The only advanced settings I have are tls 1.2 and 2 S3 extended settings.

I did start downloading the bucket directly while I was waiting to figure this out to see if that helps any from trying to restore. I did make a copy of the db, so is it worth actually trying a repair? That doesn’t mess with any of the remote files?

15

Somewhat doubtful to me. You verified on regular database, and I don’t think the temporary alters that.

Good to hear at least something matches. Which GUI location? Home screen or a Restore for the job?
Home screen is less reliable because it does not necessarily report what’s in the current job database.

It can. That’s what everybody has been warning you about repeatedly.

Creating a bug report might be the thing to do now, so somebody else can look at state of the database, basically using a database browser, Remotevolume table as mentioned, and many other possible spots.

If you go this way, the best plan is to put the database in some cloud service and put a link to it in forum.

EDIT:

This would be the other option if you don’t want to look more into the question of what’s in the database, because then you can worry less about losing the backup (no dlist means no backup), and maybe avoid whatever’s going wrong with the downloads.

16

I was just saying I killed the long running temporary build in order to run the verify on the regular db, and if you were saying the dblocks would download to the temp folder, that wouldn’t have worked out in the long run anyways because I don’t have much storage on my primary Mac HD. I didn’t assume I could run multiple tasks, so that’s why I stopped it to try running the verify. Thanks for the feedback. I did confirm that if I go to restore from the backup job, it shows all 17 revisions and also shows 17 revisions on the home screen. I’m going to wait for the bucket to finish downloading before I try anything else.

17

Also I mistyped earlier. The “extra files” on the verify in the logs were all dIndex files (not dlist)

18

I don’t think it downloads them all before it starts restoring blocks. I think it restores as it progresses.
Someone who reads C# better than I do can look at RestoreHandler.cs and related code if they like:

I’m trying to figure out what the regular db has in it, e.g. if you actually see the old logs I mentioned.
If you do the bug report, I can’t get log details (paths are redacted), but at least I can count the logs.
Previous operations and dates would also be visible. Both would prove that it’s truly a historical log.

Knowing Restore has 17 versions says at least that part’s right, but why is it claiming 17 extra dlist?
Clues to that are in the mentioned Remotevolume table. Until somebody looks, that’s still a mystery.

BUT

OK, that changes everything. Extra dindex files are sometimes redundant and completely harmless.
The challenge is to get confident that’s the case. This is pretty sure to see from browsing database, however specific names would be needed. One can also sort of visually sanity-check DB for oddity.

Accidentally deleting all dlist files kills all hope of restore. Missing dindex files hurt much less but require the long dblock download that I mentioned, so it’s still not a great thing to have an accident.

The question is whether the dindex have no value or are useful, in spite of a claim that they’re extra.
We can continue tomorrow. Maybe the downloads will be done, so we can experiment more freely…

19

Based on the rate it’s downloading, it could be a 24hr yet. Also I noticed I had files dating back to 12-2021, which is when I started fresh, but my oldest restore point is June or July, but maybe that makes sense since I was doing 7 days, 4 weeklys, and 3 monthlys if it runs in a forever incremental method.

Are you suggesting there is a way to fix the db so I could perform a restore with it without running repair?

I am running the bug report, but it seems to be stuck on creating report for some time now.

20

Just noticed that nobody in this thread cited the recovery tool:

May be relevant.