I have to do a restore to rebuild the files that were on a server whose hardware failed. My backup has about 15K index files. The restore process is reading about 120 files per hour, which means I am in for a five days-long ordeal
What resource will speed this up?
Target machine is a VM running Debian 11 with 6G ram and 2 cores. I can give it more of each, at least temporarily. The /tmp partition is 25G so I don’t think it’s running low on space in there
Suggestions?
For performance-limitation question, you probably have to see what resources seem to be the limitation.
For example, the top command seemingly has a 1 key toggle to show the %Cpu0 and %Cpu1 information.
It’s my impression that backup has more parallelism. Recreate might be limited by single-thread SQLite.
Your connection speed to the destination can matter. If not now, dblock file dowloading will be more data.
You can view sample dindex file being processed in a brief visit to About → Show log → Live → Profiling. Although I don’t think anybody has broken out timings closely (and definitely not on your equipment), the belief has been that SQL INSERT of many blocks (default 100 KB – too small for large backups) is slow.
Database work involves I/O, so iotop or sar (and probably other ones) can show how busy the drives are.
If you think a drive might be a limit, you might be able to (at risk of power loss) use ramdisk /tmp and DB filesystems (might need to move the DB on Database screen). If you think this might be of interest, read
Create a RAM disk in Linux
Is it known if the disk with Duplicati database survived? Fastest fix might be to put it in something to grab the database, but depending on how big the restore is, there might also be lots of dblock downloads later.
Anybody who is familiar with Linux performance (and especially Duplicati restore speed), feel free to add.
I am definitely seeing one and only one CPU pegged, so this indicates the single-threaded SQLite process is where my trouble resides.
I might be able to get the old database - what do I do with it?
Meanwhile I am off now to try a ramdisk /tmp and a ramdisk DB
I created a 30GB RAM Disk and mounted it at /var/duplicati_rd
I started a restore based on reading the remote restore, and I gave it the additional parameter, --dbpath=/var/duplicati_rd/RestoreInit.sqlite
That file was created at an initial size of about 44MB
Then the restore failed with:
Sep 4, 2021 6:43 AM: Failed while executing "Restore" with id: a61b985e-f2d2-4b33-b54f-c675016d7391
Duplicati.Library.Interface.UserInformationException: Found 30355 remote files that are not recorded in local storage, please run repair
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00103] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter backendWriter, System.Boolean latestVolumesOnly, System.Data.IDbTransaction transaction) [0x00019] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.RestoreHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.RestoreResults result) [0x00136] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.RestoreHandler.Run (System.String[] paths, Duplicati.Library.Utility.IFilter filter) [0x00062] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass15_0.<Restore>b__0 (Duplicati.Library.Main.RestoreResults result) [0x0001c] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.Restore (System.String[] paths, Duplicati.Library.Utility.IFilter filter) [0x00021] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x0040a] in <156011ea63b34859b4073abdbf0b1573>:0
Sep 4, 2021 6:43 AM: Failed while executing "Repair" with id: a61b985e-f2d2-4b33-b54f-c675016d7391
Duplicati.Library.Interface.UserInformationException: The database was only partially recreated. This database may be incomplete and the repair process is not allowed to alter remote files as that could result in data loss.
at Duplicati.Library.Main.Operation.RepairHandler.RunRepairRemote () [0x000b3] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x0016d] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass18_0.<Repair>b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00007] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Library.Main.Controller.Repair (Duplicati.Library.Utility.IFilter filter) [0x0001a] in <e60bc008dd1b454d861cfacbdd3760b9>:0
at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x003ad] in <156011ea63b34859b4073abdbf0b1573>:0
with that sqlite file at just under 50MB
Duplicati runs as a service so “permissions” are not the real issue here…
On Database page, use the Local database path and the buttons. For an emergency restore you don’t even need the full backup configuration (did you Export and save one somewhere safe?), just a database.
Simply getting a partially set up backup that you won’t actually backup could read the DB to allow restore. Better still might be to install from the Export To File you saved elsewhere, but turn off any scheduling. Right now you’re just trying to restore, I assume, so you don’t want anything messing with the destination.
What’s a “remote restore”?
“was created” by what? Is this the old database pulled from somewhere? Be careful about stale ones because a Repair can “fix” the consistency issue by deleting remote files to try to match old database.
Found 30355 remote files that are not recorded in local storage, please run repair
suggests that whatever database you grabbed doesn’t know about a lot of files seen. DON’T just repair.
If you don’t have a database from failed system, don’t insert a stale one. It likely will fail, and could harm.
How much are you trying to rebuild? If it’s most files, and you have hardware, you could simultaneously try Duplicati.CommandLine.RecoveryTool.exe restore. It definitely won’t build an SQLite database, and rather than rely on the dindex files it will read all the dblock files directly to find out what blocks they have in them.
There’s still an index made, but it’s not in a database. I’m not sure of the exact restore algorithm, and hope it’s not opening the dblock files for individual blocks, because that would be a slow way to rebuild the files.
I don’t know if anyone has ever run a head-to-head speed test. This is usually used for corrupted backups where the important thing is not the speed, but the ability to get something when the usual way errors out.
Recovering by using the Duplicati Recovery tool if you want to throw some fast hardware at this method…
EDIT:
and because this was a second post, be sure to read post above it about the method that you were trying.
Directly reading the backup files.
Duplicti reported success repairing the database but did not restore anything.
Having a thought: I am asking it to restore to original location. But this is a new machine. Is it possible that the /home/david/ on the old machine is not recognized as the match to the /home/david/ awaiting the restore on the new machine?
Should I tell it to restore to a new location and then specify the one I want (even if it’s the same)?
The origin of RestoreInit.sqlite is still unclear. If a stale DB, I hope your destination survived repair.
It probably doesn’t matter. Definitely won’t speed up database rebuild. When standard restore happens (meaning this is not RecoveryTool), Duplicati will try to look for restore blocks from original source files unless you use no-local-blocks option. This takes a bit of time, but it’s relatively minor if on a local drive.
Location matters more when there’s already something there (not the case here), as Duplicati doesn’t need to obtain blocks at all if the block is already there, which might happen on a revert to older version.
It did not exist.
What do you mean “…destination survived repair”? The old backup appears intact, all 30K+ files
I got no answer, so I don’t know the origin of the database and whether it exactly matches the destination.
Or maybe the database didn’t exist. I don’t know if a restore causes a recreate. At least it’s safe (I think) however I’m impressed if the recreate with ramdisk ran that fast compared to previous 120 files per hour.
Restore did indeed create the database - but 5 minutes later crapped out saying it could not repair it.
I am not using that anymore - the ramdisk is only serving as tempdir.
Recreated the database a second try - succeeded after about 30h.
Restore was trivial from that point on.