Restore restores folder structure but no files

Support
1

I’m using duplicati to backup my complete macbook (with some files beeing filterd) and an external hard drive to another external hard drive. The resulting folder with around 17 000 duplicati files is only 423 GB. I’ve now tried to restore this backup on a different machine (an older x86 Windows 7 installation).
Since I didn’t backup the configuration or database I did a direct restore from backup files (both backup as well as restore installation were done using the “2.0.5.107_canary_2020-05-26” version ). I’ve selected the latest backup run and selected to restore all files to a folder on the local hard drive. After waiting for couple of hours (according to the UI most of the time was spend at restoring the database) I came back to the machine and noticed that the backup was completed (at least there was an entry in the log that the “Repair operation was completed” - no files has however been restored, only the folder structure (about 190 000 folders). No entry in the “Stored” “Log data from the server” section only in the “Live” section.
The questions now are

  • What could be the reason that no files but only the folder structure was restored?
  • I already tried various things and the restore of the database always seems to take ages: is there a way to save the database and use it for the next restore attempt? My temp folder contains a 2.2 GB tmp file starting with dup-* which seems to be the database - is there a way to reuse it?

Thank you in advance for your help. I know that it is better to also back up the configuration file and the next thing I’ll try is to do a restore with the configuration file but I thought a restore should also work without it

Kai

2

It looks like it creates the folder structure as a separate and early step in the restore:

Maybe something went wrong after the CreateDirectoryStructure and before the file restores began.

Anything interesting in the live log beyond hearing that Repair completed? You might need to set a higher logging level, e.g. Verbose to see more of the action. A log file can also be made, but it’s a little awkward.

As far as I know, the partial temporary database is created each time. Ordinarily it should be quite fast for typical backups, unless there’s a problem with the backup files that make it read all files (takes awhile…).

The live log at Verbose level is good for watching this. Ideally it would just read the dlist files (one for each backup version) and dindex files (one per dblock file). If you see a progress bar at more than 70%, dblock files are being read to look for missing information. The last 10% (90%-100%) can be especially lengthy…

is the database recreate code showing progress computation, the 3 passes, and the Verbose messages.

Duplicati.CommandLine.RecoveryTool.exe might be worth a try. It works quite differently, and gives sort of what you ask for, in that the index build is persistent and can be reused for multiple restores, however the way of telling it what you want restored is crude, so you might just say you want all files. Is that your wish?

Recovering by using the Duplicati Recovery tool details usage, but I don’t know if it can do cross-platform.

I had assumed you were doing Direct restore from backup files, basically re-entering the configuration for single use (reminding me of the single-use DB problem). Setting up the backup job on another machine simply for restore purposes will let you reuse the DB (because it’s not a partial temporaray database) but risks damaging your backup files if a second machine actually attempts operations that change the files. Basically, don’t try a backup, repair, or such operations, but I’d hesitate to even set up this sort of design.

1 Like
3

Thank you very much for the detailed response!!!
In regards to the issues that only folders, no files were restored:

It looks like there was a indeed an error during the 1st restore. The “Stored” log contains the error

code = Unknown (-1), message = System.Data.SQLite.SQLiteException (0x80004005): unknown error
No transaction is active on this connection
   bei System.Data.SQLite.SQLiteTransactionBase.IsValid(Boolean throwError)
   bei System.Data.SQLite.SQLiteTransaction.Commit()
   bei Duplicati.Library.Main.Database.LocalDatabase.Dispose()
   bei Duplicati.Library.Main.Database.LocalRestoreDatabase.Dispose()
   bei Duplicati.Library.Main.Operation.RestoreHandler.Run(String[] paths, IFilter filter)
   bei Duplicati.Library.Main.Controller.<>c__DisplayClass15_0.<Restore>b__0(RestoreResults result)
   bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   bei Duplicati.Library.Main.Controller.Restore(String[] paths, IFilter filter)
   bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

I think this was caused by the energy saving settings of the pc. I’ve deactivated a hibernation/standby after x hours and retried the backup.

This time I got the following exception. Looks like the partition which contains the windows tmp folder was full as the tmp database from the 1st restore was still taking up several GB. I’ve moved the tmp folder to a different partition and restarted the restore.

code = Full (13), message = System.Data.SQLite.SQLiteException (0x800007FF): database or disk is full
database or disk is full
   bei System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
   bei System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   bei System.Data.SQLite.SQLiteDataReader.NextResult()
   bei System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   bei System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   bei System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
   bei System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
   bei Duplicati.Library.Main.Database.LocalRestoreDatabase.FindMissingBlocks(Boolean skipMetadata)
   bei Duplicati.Library.Main.Operation.RestoreHandler.PrepareBlockAndFileList(LocalRestoreDatabase database, Options options, IFilter filter, RestoreResults result)
   bei Duplicati.Library.Main.Operation.RestoreHandler.DoRun(LocalDatabase dbparent, IFilter filter, RestoreResults result)
   bei Duplicati.Library.Main.Operation.RestoreHandler.Run(String[] paths, IFilter filter)
   bei Duplicati.Library.Main.Controller.<>c__DisplayClass15_0.<Restore>b__0(RestoreResults result)
   bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   bei Duplicati.Library.Main.Controller.Restore(String[] paths, IFilter filter)
   bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

This time I’ve got the following exception in the log. I believed it could be caused by the fact that the target folder still contained all the folders from the first restored. Therefore I’ve completely emptied the target folder.

Duplicati.Library.Interface.UserInformationException: The version(s) being updated to, already exists
   bei Duplicati.Library.Main.Operation.RecreateDatabaseHandler.RunUpdate(IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   bei Duplicati.Library.Main.Controller.<>c__DisplayClass26_0.<UpdateDatabaseWithVersions>b__0(RecreateDatabaseResults result)
   bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   bei Duplicati.Library.Main.Controller.UpdateDatabaseWithVersions(IFilter filter)
   bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

This time the restore seemed to be successful, restoring about 500 GB of data :slight_smile: It took about 3-4 days but at least it prooved my backup is working. The only exception I got in the Stored log is the following

System.ArgumentException: Illegal characters: [] in path.
Parametername: path
   bei Alphaleonis.Win32.Filesystem.Path.CheckInvalidPathChars(String path, Boolean checkAdditional, Boolean allowEmpty)
   bei Alphaleonis.Win32.Filesystem.Path.CheckSupportedPathFormat(String path, Boolean checkInvalidPathChars, Boolean checkAdditional)
   bei Alphaleonis.Win32.Filesystem.File.CreateFileCore(KernelTransaction transaction, Nullable`1 isFolder, String path, ExtendedFileAttributes attributes, FileSecurity fileSecurity, FileMode fileMode, FileSystemRights fileSystemRights, FileShare fileShare, Boolean checkPath, Boolean continueOnException, PathFormat pathFormat)
   bei Alphaleonis.Win32.Filesystem.File.CreateFileStreamCore(KernelTransaction transaction, String path, ExtendedFileAttributes attributes, FileSecurity fileSecurity, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, PathFormat pathFormat)
   bei Alphaleonis.Win32.Filesystem.File.Create(String path)
   bei Duplicati.Library.Common.IO.SystemIOWindows.PathTooLongFuncWrapper[T](Func`2 nativeIOFunc, Func`2 alternativeIOFunc, String path, Boolean prefixWithUnc)
   bei Duplicati.Library.Common.IO.SystemIOWindows.FileCreate(String path)
   bei Duplicati.Library.Main.Operation.RestoreHandler.DoRun(LocalDatabase dbparent, IFilter filter, RestoreResults result)
   bei Duplicati.Library.Main.Operation.RestoreHandler.Run(String[] paths, IFilter filter)
   bei Duplicati.Library.Main.Controller.<>c__DisplayClass15_0.<Restore>b__0(RestoreResults result)
   bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   bei Duplicati.Library.Main.Controller.Restore(String[] paths, IFilter filter)
   bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

But hopefully this is sth. I can ignore in my use case. I’ve choose duplicati for the the following reasons

  • it allows me to do a full backup of my 2011 macbook and an external hard drive to a different external hard drive
  • the backup is encrypted (I’m keeping one of my backup drives in the office so the data should be encrypted)
  • I’m able to restore the backup on windows and on osx (Once my macbook breaks I’ll move back to windows, so I need a backup tool that covers that)

Looks that it actually works if you have enough time for the restore. When the time comes and I actually need the backup I’ll certainly try out your suggestions in regards to the restore tool :slight_smile:

Thanks again for the support and keep up the great work!!

Kai

1 Like