Could not find a part of the path, but why?

Support
1

Could not find a part of the path '\\?\d:\restore\...'

Edit from release notes:

Maybe this is a new feature in this specific canary version, which is probably more or less broken?
“Support for Windows paths starting with “?”, thanks @dferreyra”

Repeated several times, it seems that restoring these files failed. Even if I’ve configured separate restore path. Lots of files got restored, but only small subset of the files failed to be restored. My question is why? Why the path contains that prefix? And why only for some files, but not for all? The source is similar for all files. Many directories were created, but without the \?\ prefix, only the failing directories had that prefix, and those are just sub-directories of the properly restored directories.

Version: 2.0.5.111_canary_2020-09-26 - Windows 10 with latest updates, 64 bit, running as normal user, but I’ve got full write access to the specified test restore path.

I did read the discussions about Linux file permissions and those didn’t seem to be relevant with this one. I’m not restoring to the original path, filenames / paths aren’t too long, and so on.

Just as side comment. Also while running these test, I found again the inherent traps of Duplicati. Everything seems to be fine, except restore fails. Now the reason is the classic without the original database, the backup sets can’t be properly restored at all. - This is still a devastating show stopper. Backup software which can’t properly restore is just ultimate trap for people.

2

Support for Windows paths starting with “\\?\”

was meant to address the case where the user provided a source path that started with “\\?\”. If you didn’t explicitly do so, then the issue might have come from

“Improved handling of “problematic” filenames on Windows and increased test coverage”

introduced in 2.0.5.109. I think these changes require .NET 4.6.2 to work properly.

Other than the prefix, is there anything common about the files that couldn’t be restored? Can you provide more of the error message or stack trace? It’s unclear if the problem is due to the \\?\ prefix, or some other part of the path.

I don’t know if @dferreyra is around, but their knowledge may be useful here.

3

Sorry, I didn’t start with all the details, because I expected this to be known issue.

I haven’t ever used \\?\ paths with Duplicati. Always traditional drive references c:\data\etc…

I’ll check the .NET version issue, I haven’t paid any attention to the .NET version, yet I’ve got Windows 10 with all the latest updates and feature updates. But I’ll check the .NET version.

Yes, there is one factor which immediately came to mind, but the release note comment made me suspect it might be irrelevant. The files which have that \\?\ path issue, are all old files. Which means that those were backed up probably using much older Duplicati version. I did think about that for a while while checking what the problem might.

For the files which were successfully restored, there are also always mentions of creating directory for the files because I’m restoring to empty test restore path. But for the files which failed to be restored with the invalid (?) path reference the notification about directory creation was also missing. I’m not familiar with the source. It might be relevant or not. Depending how the code is structured and what’s the actual failure point.

But then I decided that because the backups are run daily, there has been plenty of opportunities to upgrade the path references if changed in time to fix the problems. I’ll re-run the restore-job with trace (later) again, and let’s see if it gives me any hints.

4

There’s no indication why the files were not restored. But in the verification stage it gives erros like that.
I used --debug-output when running the restore.

Verifying restored files ...
Could not find a part of the path '\\?\d:\restore\Memo\2014\filename-redacted'. => System.IO.DirectoryNotFoundException: Could not find a part of the path '\\?\d:\restore\Memo\2014\filename-redacted'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at Duplicati.Library.Common.IO.SystemIOWindows.FileOpenRead(String path)
   at Duplicati.Library.Main.Operation.RestoreHandler.DoRun(LocalDatabase dbparent, IFilter filter, RestoreResults result)

.NET version
Client 4.8.04084
Full 4.8.04084

Yet I also got this during restore… But if I run all the checks repairs and tests, there’s “nothing wrong”.
Failed to patch with remote file: "duplicati-b43c50965b03842b0b63b7d11db7fea73.dblock.7z.aes", message: System.IO.FileNotFoundException:

5

After running more tests and recreating the database, I think the reason for the error above is “generic duplicati integrity and consistency failure issues”.

So this just confirms the fact that even with latest version the serious integrity issues do persist. Most likely the best fix would be deleting the backup set and completely building it again. (The source files are still available).

Kind of getting seriously tired with these issues. But let’s hope that the problems are resolved at some point.

After all the repair operations, building database, repairing, running NEW backup set, and running restore, the files are still missing and un-restorable. -> DANG.

6

Reproducible issues in Canary are always worth looking at. Canary is there to be the first user testing activity and can fail massively, though this is rare. Issues that are fixable are often ones that give clear, direct steps from scratch with as few dependencies as possible, to raise chance developers can see.

Duplicati has a fix checked in for an issue some people saw in the new path handling, however it’s not available in a Canary yet. As an alternative, it would help to try older compatible releases like 2.0.5.108.

Seeing that work would mean there’s definitely a new issue in Canary. Or maybe this is an older issue. Either way, starting with a simple backup and seeing where it breaks between there and yours helps…

Although my restore work on 2.0.5.111 is fine, it’s guaranteed that my specifics are different. What’s the smallest simplest case that fails for you? After knowing it, others can try it. It’s possible there’s an issue.

7

@Sami_Lehtinen, I wanted to throw out some thoughts regarding the problem you’ve encountered.

The Canary you’re running includes some changes I made to allow for files with embedded wildcard characters to be restored. This changed code in the file restore GUI and the internal processing of file filters (#4302) so I am very interested to know if I’ve broken something.

On the subject of the funny prefix, most file paths will be prefixed with “\\?\” internally before being passed on to .NET to support more file name characters under Windows. So low level errors like the one you reported will show paths with that prefix.

Generally I think that if Duplicati had encountered an error while actually performing the restore step, you would have gotten an error about it at that point. But the error you’re seeing is coming from the verification step. That makes me think that maybe the restore step thinks those files don’t need to be restored for some reason, but the verification step thinks they should have been restored and so complains about it. If Duplicati didn’t think the files needed restoring, it wouldn’t create the associated directories either, which would account for the “Could not find a part of the path” error at the verification step which I think is complaining about a missing component of the path leading up to the file in question.

Is there any pattern you can see in the files that aren’t being restored? For example, do their source paths (i.e., the directories and file names where the source files reside) contain any interesting characters, like “?” or “*”, or do they start or end in “.”? Are symbolic links involved?

I’m not sure if this will help, but one thought might be to try using the Duplicati command line tools to list files in the backup and to perform targeted restores, maybe first of a file that you know can be restored to confirm you’ve gotten the command line right, then of one of the files that fails to restore. If you get different results by going through the command line, that might give us a clue. I’m not sure if that’s something you would be comfortable trying, and like I said, I’m not sure if it would help.

Of course, if you were able to succeed with any of @ts678’s suggestions, that would be very helpful.

8

Sorry that I didn’t provide more detailed information earlier.

It just took a better look at this problem.

Luckily, I just did read the other thread about very slow disaster recovery where people were complaining about the classic problem of files missing from restore and having issues with corrupted data blocks.

It reminded me about the broken logic of the program and after manually verifying, I found out that’s exactly the case.

I do get this error during restore:
Failed to patch with remote file: “duplicati-b43c50965b03842b0b63b7d11db7fea73.dblock.7z.aes”, message: System.IO.FileNotFoundException:

And when I run affected command with the filename:
A total of 310 file(s) are affected:
… And for sure … Those files which were missing from restore are in that specific archive …

I think this confirms that the problem isn’t related to paths / permissions at all.

There are a few main issues:

  1. Misleading or confusing error messages - If files can’t be restored due to dblock issues, why it’s not being clearly stated?
  2. Bad error detection logic - This error can be only detected when restoring
  3. Bad repair logic - Repair won’t do a thing about this error
  4. Bad recovery logic - The source files are STILL AVAILABLE, any time, when I run backup, the situation (at least for these specific files) could have been fixed by storing the dblocks again. But that won’t happen because the steps 2 and 3 fail.

Again this discussion wouldn’t even exist if those age old issues would have been fixed.

This is exactly the con side of de-duplication, especially when combined with many other things making it very brittle. Another big con of de-duplication is restore performance when the original database isn’t available.

@dferreyra
I did get an error message. Which I pasted in this thread. But it seems that nobody else thought about that immediately. Because it was kind of obscure, and didn’t clearly indicate what the practical ramifications are. Also the information of known missing files wasn’t passed to the verification step.

I did once again run: test, list-broken-files, purge-broken-files, repair… And then backup… And even after these steps the restore of those files still fails, even if the files are still available from the source. Which just proves the point I made earlier. The situation would be (at least for these files) trivially fixed. Yet of course it still could and would affect files which aren’t available anymore.

Generic question about restore recovery

Based on this discussion and disaster recovery thoughts. What happens in situation where restoring a a large history log file or something like that fails due missing block(s). Does duplicati allow restoring partial files? What if from 10 GiB log file there’s a 100 KiB deduplication block which isn’t available. Does it prevent restoring the rest of the file or is there an option to allow restoring files which are known to be corrupted (or the missing parts filled with zero or so). Just wondering…

I know it’s generally not preferred ot restore known corrupt data. But in some situations, the actually very valuable data can very small part of a big data set. When that happens, allowing the restoration of known corrupted files would still be very helpful.

9

✓ Confirmed - After deleting the “offending file” list-broken-files and purge-broken-files fixed the situation and running new backup set fixed the problem.

Now restore doesn’t report the path error anymore either.

But this just proves the point that the logic is still inherently broken and requires manual tinkering every now and then. - Which is of course something what normal users never do and then they’ll end up with broken and unrestorable backups.