I’m performing a DRP test from Google Drive on a simulated total crash scenario: create new VM, install OS, install duplicati, and direct restore from backup files.
It takes forever… – already running for more than 12 hours just to recreate the DB
Quick look at Task manager (see screenshots), I do not see any resource maxed out (be it CPU, disk or network).
It looks like duplicati downloads one dblock file, then processes it, then downloads the next one and so on, and like some parallelism (e.g. → one thread downloads dblock files, maximizing bandwidth usage, while another thread processes them, maximizing disk & CPU usage) would improve things.
I imagine it’s desireable to do so, since whenever you are in this scenario (you lost everything) you’d want to get back online as soon as possible.
The idea is that the database can be recreated by simply downloading the .dindex files.
But, I have recently discovered a bug where the compact feature does not write some required information to the .dindex files.
This was not discovered in testing, because Duplicati will automatically search the .dblock files if anything is missing.
In practice it means that recreate works reliably, but will often need to download .dblock files which is very slow. Once the fix is released, there is still the problem that existing .dindex files is missing the information.
The download process is not massively parallel, but it is doing overlapped download/recreate. So it will download files back-to-back while processing them.
Hi Kenneth,
Thanks for the feedback. That’s a pretty clear explanation, and glad to see this “let’s not just fix the bugs, but also provide a way to correct the existing data” mindset
Kudos for that
Best,
Gaston
