Please let me know if I can attach some logs to make helping easier! I haven’t seen any recent topics on this so wanted to make sure I wasn’t seeing a new issue.
I’m running Duplicati on an Unraid machine and my cache drive died last week. I’m dealing with corrupt home assisstant and plex databases – luckily I have my appdata folder (where docker data is held) backed up.
I installed the Duplicati container and from what I understand I need to recreate the local database in order to restore the files back onto my array somewhere. However, the recreate has been taking FOREVER. Seems like it went from 0-70% in just an hour or so, then it went from 70-72.5% overnight last night. I’m worried I’ll be waiting weeks to get my missing data back.
that’s a bit surprising you did not see a recent topic since the issue is cropping about one time a week. This happens when not only the computer has crashed, but the data on the backend is badly corrupted. In this particular case, the database recreation is very slow.
In my opinion, you have better to follow the disaster recovery option. This bypasses the problematic part of database recreation to get only the data (the backup can’t be used afterward, it’s only a way to restore the data, or sometimes even only a part of it if the backend is really damaged).
Here is the chapter about the disaster recovery tool:
You know, I do have to apologize – I was using Google to search the forums and kept seeing articles from 2-3 years ago saying newer versions of Duplicati would be faster. Then I posted this and saw that there was a support sub forum and sure enough I saw all the recent topics. Thank you for offering this though, probably what I need as once I get these files back I plan on changing the way I backup this folder (will just be backing up one zipped file).
I use neither Unraid nor Docker, but I think Docker for Duplicati is supposed to be configured by you so that it keeps its configuration (including the database) on the host. Are you sure you never set that up?
Another advantage of not keeping configuration in the container is that a container upgrade gets easier. What version of Duplicati do you think first did this backup (or around what time)? Older had more bugs.
When running disaster recovery, you might want to watch About → Show log → Live → Warning in case something goes wrong there, but your odds are a little better than when trying a total database recreate.
Of course, if you can find the old database on the Unraid system somewhere, that’s a great solution too.
Appreciate the welcome! So yeah unfortunately where the database for Duplicati is stored I had some files go missing/corrupt when one of my drives failed. So I’m thinking the Duplicati DB suffered an ill fate as it keeps telling me I have to rebuild all the local databases.
I’m not exactly sure how to do the disaster recovery – I’m storing my data on b2. I stopped the other recreate and started to do a “restore from direct files” I was able to pick the two folders I really care about (Home Assisstant and Plex) but it apparently still has to recreate the database to make this happen… I’m guessing I’ll be in the same boat.
I think it builds a partial, temporary database for specific restore, so small might not hit prior issue.
Suggestion of looking at log is because there’s also a chance something might go wrong here too.
Watching it at Verbose level will show more, including what it’s doing now, and what’s left to finish.
Anything past 70% on progress bar is getting worrisome, but 90% to 100% can be a large search, downloading dblock files to look for data that a dindex should have known about. Result unknown.
What is Verbose live log saying it’s doing? It should be downloading dblock files occasionally.
Typically these are 50 MB, but backup could have a higher Remote volume size in Options.
Yeah – it is downloading the files in 50MB chunks, just seems to be taking a long time to make its way to 100%. I only chose two folders but to be fair Plex is by far the biggest folder contained in the backup.
In Verbose log, I think it tells you how far it is in this phase, so you can sort of forecast a total duration.
Possibly I’m mistaken. Haven’t tried looking at the code. Seeing anything like a count and total count?
EDIT:
70% to 80% on progress might say “Pass 1 of 3, processing blocklist volume X of Y”. How long each?
So that’s managed about 5% of pass 1 in not a whole lot of time. One option is to let it continue.
Problem is that if pass 1 doesn’t find what it wants, there are pass 2 and pass 3 to keep trying…
Eventually the entire backup will have downloaded, with no certainty that all blocks are present.
Do you pay for downloads? Do you have enough extra space to hold the entire destination too?
If normal restore winds up not finishing well after searching, alternative needs the big download.
Yeah I forget the exact pricing structure of b2 but I do remember uploading/storing data is super cheap but downloads do come with a cost… not a big deal though. I have 3TB of free space and the backup is only ~160GB so that shouldn’t be an issue.
Update – I found the command line tool in my container so I might go that route… seems like it’d be faster?
Unknown, but you can try. If the destination is missing a dindex it won’t care a bit. It uses only dblock.
I think a missing dblock will make a complaint of some sort about some file that can’t be fully restored.
EDIT:
More accurately put, it looks at dblock files directly, so doesn’t care about the dindex file for a dblock.
If you’re lucky your destination is missing a dindex, so all of your data will be restored, but time will tell.
Same luck would mean DB recreate would have eventually finished, and I don’t know which runs faster.
Normal restore would also have to do the restore after getting enough of DB back, so there’s more time.
Thanks for all your help here – I decided to go ahead and cancel the database recreate and just go with the disaster recovery plan. Right now I’m downloading all the files necessary to restore then I’ll use filters to pick and choose which folders to restore.
It’s only been running for a few minutes and I’ve already downloaded almost 10GB!
Whether or not the end restore gets there faster, this method (I think) at least keeps things moving.
Typical restore with database can jump to restore faster. Without DB, there’s the variable first step.
When all goes well, it’s not that bad. When things go astray, later parts can add an annoying delay.
Figuring out what went astray in your case would be easier from the database, but that’s gone now.
Counting index versus dblock files could be an indicator. Fewer dindex than dblock might be a clue.