Unable to restore any files after harddrive crash

mdwyer · January 25, 2018, 8:46pm

I’ll try that shortly. The latest rebuild has finished, I have backed up the database and I’m attempting to restore a file. After that I’ll try updating Duplicati.

mdwyer · January 25, 2018, 8:55pm

The restore failed again. At the end of the db rebuild it gives this error…

Could not find file ‘G:\beast_backups\aes\duplicati-bf11547ab15ad4008aa3a543ec54550f8.dblock.zip.aes’.

The actual restore process sat for minutes eventually doing a verifying remote files stage after which it errors with the message about the missing 85 files and nothing is restored.

Why doesn’t the db rebuild deal with all of that? It seems that rebuilding the database is simply step 1 of many steps.

mdwyer · January 25, 2018, 9:47pm

I installed the latest Canary and…

C:\Windows\System32>"C:\Program Files\Duplicati 2\Duplicati.Server.exe"
A serious error occurred in Duplicati: System.IO.FileNotFoundException: Could not load file or assembly ‘CoCoL, Version=1.5.0.24305, Culture=neutral, PublicKeyToken=0983de3c914beeaa’ or one of its dependencies. The system cannot find the file specified.
File name: 'CoCoL, Version=1.5.0.24305, Culture=neutral, PublicKeyToken=0983de3c914beeaa’
at Duplicati.Library.UsageReporter.Reporter.Initialize()
at Duplicati.Library.UsageReporter.Reporter.SetReportLevel(Nullable`1 maxreportlevel, Boolean disable)
at Duplicati.Server.Program.StartOrStopUsageReporter()
at Duplicati.Server.Program.RealMain(String[] args)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

Crash!
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.IO.FileNotFoundException: Could not load file or assembly ‘CoCoL, Version=1.5.0.24305, Culture=neutral, PublicKeyToken=0983de3c914beeaa’ or one of its dependencies. The system cannot find the file specified.
at Duplicati.Library.UsageReporter.Reporter.ShutDown()
at Duplicati.Server.Program.RealMain(String[] args)
— End of inner exception stack trace —
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Duplicati.Library.AutoUpdater.UpdaterManager.RunMethod(MethodInfo method, String[] args)

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly ‘CoCoL, Version=1.5.0.24305, Culture=neutral, PublicKeyToken=0983de3c914beeaa’ or one of its dependencies. The system cannot find the file specified.
at Duplicati.Library.AutoUpdater.UpdaterManager.RunMethod(MethodInfo method, String[] args)
at Duplicati.Library.AutoUpdater.UpdaterManager.<>c__DisplayClass58_0.b__0()
at Duplicati.Library.AutoUpdater.UpdaterManager.WrapWithUpdater(AutoUpdateStrategy defaultstrategy, Action wrappedFunction)
at Duplicati.Library.AutoUpdater.UpdaterManager.RunFromMostRecentSpawn(MethodInfo method, String[] cmdargs, AutoUpdateStrategy defaultstrategy)
at Duplicati.Server.Program.Main(String[] args)

mdwyer · January 25, 2018, 9:54pm

I got this version to run without crashing… v2.0.2.15-2.0.2.15_experimental_2018-01-03

How do I fix the missing files problem?

mdwyer · January 25, 2018, 10:39pm

At the moment I am stuck in a loop of having 85 missing files please run repair.

mdwyer · January 26, 2018, 2:10am

I tried list broken files (using the web interface’s command line section) and it says “Cannot continue because the database is marked as being under repair, but does not have broken files.
Return code: 100”

Where can I reset that flag so it is not under repair?

mdwyer · January 26, 2018, 2:46am

GAH! This is driving me nuts! How do I find out which 85 files are missing and how do I ‘remove’ them such that Duplicati stops breaking because of them?

[Edit] Ok so inspecting the database I now have the list of 85 missing files. What now? Here is one of them…

Remote file referenced as duplicati-b19e4d6db9ee34496b50f8c4e4984a51f.dblock.zip.aes, but not found in list, registering a missing remote file.

I found this in a table called RemoteVolume.

“3021” “1” “duplicati-b19e4d6db9ee34496b50f8c4e4984a51f.dblock.zip.aes” “Blocks” “52396173” “bbbOLPhpj3yq6rbkZ8iwUwBTYYlwbPbbTuQNoCqIxwY=” “Verified” “0” “0”

The version table has the number 7

mdwyer · January 26, 2018, 3:06am

I am beating up the database. I have found and set false the repair flag and I am picking on 1 single small file in my backup and gathering all the info I can find about where it is located in the backups. I am going to attempt to manually extract it.

mdwyer · January 26, 2018, 7:50am

What happened to this guide? The link doesn’t go to it.

code dot google dot com/p/duplicati/wiki/ManualRestoreHowTo

mdwyer · January 26, 2018, 10:59am

GREAT NEWS! I am successfully restoring my entire backup (latest version). I will document what I did soon. I am busy pouring myself a drink atm and enjoying the moment.

mdwyer · January 26, 2018, 1:09pm

I’ll file a bug report when I get time but I found a bug in Duplicati.CommandLine.RecoveryTool.exe
I told it to restore to a new folder which it did, but it also created every folder in the original locations that are in the backup set. So I have a lot of empty folders in the AppData folder for example.

The recovery tool is making the same folder twice, once in the specified new location and once in the original location.

drakar2007 · January 26, 2018, 3:01pm

This is something I haven’t actually tried to do before - what method and/or tool do you use to open/examine the local DB? I have some SQL experience but it’s all remote server stuff with dedicated web UI’s.

Regardless, thanks for persisting, I hope the developers will implement some fixes that take your findings into account.

I’m curious if the steps you took would help me in my attempts to run a headless restore from my Google Drive backup set - whenever you feel like typing it out, I wouldn’t mind looking over the ‘quick version’, at least.

mdwyer · January 26, 2018, 8:11pm

In the end I used the Duplicati commandline recovery tool to restore my files. I did try using this tool many times without success before but there were a few differences this time. I noticed in the database that my backups used a specific version so I uninstalled Duplicati and installed that version. I also let the tool decrypt the files instead of using a 3rd party tool. I don’t know that that mattered but I hadn’t had any luck with the tool up to this point.

My steps to restore my backup.
Note: I included a database but I don’t think it used it so that part can probably be omitted.

Download or in my case simply copy the backup to a local folder.

Run the recovery tool with the download option to decrypt the backup files
.
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe” download “file://g:\beast_backups\aes” “g:\backups\temp” --dbpath=g:\db6.sqlite --passphrase="******"

Index the decrypted backup files.
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe” index “g:\backups\temp” --no-encryption

Restore the backup files either to their original locations or in my case to a temp folder for manual copying later.
“C:\Program Files\Duplicati 2\Duplicati.CommandLine.RecoveryTool.exe” restore “g:\backups\temp” --dbpath=g:\db6.sqlite --no-encryption --targetpath=“g:\restored” --version=0

The restore step initially didn’t restore to the temp folder though fortunately I caught it before any damage occured. I didn’t use --targetpath I used --restore-target not knowing that the different commands used inconsistent terms.

The main lesson I learnt from all of this was to not even bother trying to repair the database. Complete waste of a lot of time (nearly 2 weeks). The successful restore process was underway in a matter of minutes.

I have marked kenkendk’s reply as the solution but I had previously found that and tried it without success. I think why it failed was because I didn’t find good examples of using it in a real world scenario so I didn’t get some of the steps right previously. Also I was under the impression that it needed the database fixed which it didn’t.

mdwyer · January 26, 2018, 8:22pm

The tool I used to explore the database was SQLiteDatabaseBrowserPortable

mdwyer · January 26, 2018, 8:36pm

I found a bug report from 2016 that seems to match the bug I found. The version of the repair tool I used was from late 2017 so I’m going to assume the issue is not fixed although the post below seems to state that it was.