Why restore needs so much time of verifying remote data

gpatel-fr · November 18, 2023, 2:50pm

I have to admit that this has made me laugh, thanks. Yes, that’s a beginning, and then recreate the database, and hope that the stars align (the file and directory rights will not I think). It’s just that’s well, a bit long and complicated, it’s not exactly available out of the box and there is no warranty it will work well enough for all uses, so it’s just better to say that it’s not supported officially.

hubert · November 18, 2023, 3:35pm

i havent decided yet, how to restore. Preferred way was to do all stuff on the server. But if it crashed and i need my data it would be nice to have the possibility to use other systems like windows. And like i understood restore is possible on other systems.

You are welcome ;-).
The idea comes from other backup software, where the absolut pathes to the backup folders was stored inside the config and from there the backup internal stores only the relative pathes. I like this concept to have the possibility to change pathes without changes in the backup (only config) to have it as much flexible as possible.

Sorry for this offtopic. Its not 100% needed for me. But maybe it helps to think of the architecture.

ts678 · November 18, 2023, 4:49pm

I think the way that works is that if Duplicati and its database gets moved and finds an OS mismatch, backup will be refused but restore will be allowed, but obviously you need to tell it new restore folder.

Restoring files if your Duplicati installation is lost

however if you want to be sure Direct restore from backup files works cross-OS, please try it.

Duplicati Destination is portable this way, even between types of storage, as names and storage are separated. I’m not quite sure how one would pull this off on the Source side. Paths could be anywhere. Duplicati does break a Source path into a folder (shared by other paths) and a file name to save space.

EDIT:

For a specific example, I have a backup with 67101 rows in its File SQL view, but underneath that is a 6074 row PathPrefix table with OS-specific prefix to get to the folder, and 67101 row FileLookup table giving the end part. It doesn’t have drive letters, but it’s a little OS-specific because a folder gets slash final character, and direction is OS-specific. This final slash also shows up (but isn’t obvious) in Filters.

The Remotevolume table has simple file names, and the destination folder comes from the server DB.

Jojo-1000 · November 18, 2023, 10:23pm

I created this issue:

github.com/duplicati/duplicati

File list verify is unnecessarily slow on some filesystems

opened 10:19PM - 18 Nov 23 UTC

Jojo-1000

- [x] I have searched open and closed issues for duplicates. - [x] I have searc…hed the [forum](https://forum.duplicati.com) for related topics. ---------------------------------------- ## Environment info - **Duplicati version**: current master - **Operating system**: Windows 10 - **Backend**: File ## Description Listing the backend files for exFAT targets (may also apply to other FAT types) takes much longer than required. It is suspected this is due to unnecessary lookups of files by name. Using `DirectoryInfo` to directly list the folder contents as `FileInfo`, this lookup is avoided and the list operation completes much faster (50 seconds instead of 1 hour for 110000 files). `ISystemIO` should be updated to implement the more efficient method of listing. ## Steps to reproduce 1. Create a folder on exFAT partition with 10000 - 100000 empty files 2. Create new backup with that folder as target 3. Run backup - **Actual result**: File verify takes a long time (minutes to hours). - **Expected result**: File verify should not take longer than a minute. ## Test code This code simulates two different methods of file access. It was discovered that accessing `LastAccessTime` or other metadata is the main reason for the slowdown. ```cs string path = @"F:\test"; int iterations = 1; var timeInfo = TimeSpan.Zero; var timeListNames = TimeSpan.Zero; var timeLookupNames = TimeSpan.Zero; var watch = System.Diagnostics.Stopwatch.StartNew(); for (int i = 0; i < iterations; ++i) { // List files directly (current implementation) watch.Restart(); string[] files = System.IO.Directory.GetFiles(path); watch.Stop(); timeListNames += watch.Elapsed; watch.Start(); var accessTimes = (from fileName in files let fi = new System.IO.FileInfo(fileName) select fi.LastAccessTime).ToList(); watch.Stop(); timeLookupNames += watch.Elapsed; // List by DirectoryInfo watch.Restart(); var accessTimes2 = (from fi in new System.IO.DirectoryInfo(path).GetFiles() select fi.LastAccessTime).ToList(); watch.Stop(); timeInfo += watch.Elapsed; } Console.WriteLine($"List only names: {timeListNames}\nList + lookup by name: {timeLookupNames}\nList DirectoryInfo: {timeInfo}"); ``` ### Output with 17000 files ``` List only names: 00:00:00.0264646 List + lookup by name: 00:00:58.3627105 List DirectoryInfo: 00:00:00.0315012 ```

It should not be too hard to fix, but there is some special handling of extended paths that needs to be taken care of.