Could not find a part of the path, but why?

Could not find a part of the path '\\?\d:\restore\...'

Edit from release notes:

Maybe this is a new feature in this specific canary version, which is probably more or less broken?
“Support for Windows paths starting with “?”, thanks @dferreyra

Repeated several times, it seems that restoring these files failed. Even if I’ve configured separate restore path. Lots of files got restored, but only small subset of the files failed to be restored. My question is why? Why the path contains that prefix? And why only for some files, but not for all? The source is similar for all files. Many directories were created, but without the \?\ prefix, only the failing directories had that prefix, and those are just sub-directories of the properly restored directories.

Version: 2.0.5.111_canary_2020-09-26 - Windows 10 with latest updates, 64 bit, running as normal user, but I’ve got full write access to the specified test restore path.

I did read the discussions about Linux file permissions and those didn’t seem to be relevant with this one. I’m not restoring to the original path, filenames / paths aren’t too long, and so on.

Just as side comment. Also while running these test, I found again the inherent traps of Duplicati. Everything seems to be fine, except restore fails. Now the reason is the classic without the original database, the backup sets can’t be properly restored at all. - This is still a devastating show stopper. Backup software which can’t properly restore is just ultimate trap for people.

Support for Windows paths starting with “\\?\”

was meant to address the case where the user provided a source path that started with “\\?\”. If you didn’t explicitly do so, then the issue might have come from

“Improved handling of “problematic” filenames on Windows and increased test coverage”

introduced in 2.0.5.109. I think these changes require .NET 4.6.2 to work properly.

Other than the prefix, is there anything common about the files that couldn’t be restored? Can you provide more of the error message or stack trace? It’s unclear if the problem is due to the \\?\ prefix, or some other part of the path.

I don’t know if @dferreyra is around, but their knowledge may be useful here.

Sorry, I didn’t start with all the details, because I expected this to be known issue.

I haven’t ever used \\?\ paths with Duplicati. Always traditional drive references c:\data\etc…

I’ll check the .NET version issue, I haven’t paid any attention to the .NET version, yet I’ve got Windows 10 with all the latest updates and feature updates. But I’ll check the .NET version.

Yes, there is one factor which immediately came to mind, but the release note comment made me suspect it might be irrelevant. The files which have that \\?\ path issue, are all old files. Which means that those were backed up probably using much older Duplicati version. I did think about that for a while while checking what the problem might.

For the files which were successfully restored, there are also always mentions of creating directory for the files because I’m restoring to empty test restore path. But for the files which failed to be restored with the invalid (?) path reference the notification about directory creation was also missing. I’m not familiar with the source. It might be relevant or not. Depending how the code is structured and what’s the actual failure point.

But then I decided that because the backups are run daily, there has been plenty of opportunities to upgrade the path references if changed in time to fix the problems. I’ll re-run the restore-job with trace (later) again, and let’s see if it gives me any hints.

There’s no indication why the files were not restored. But in the verification stage it gives erros like that.
I used --debug-output when running the restore.

Verifying restored files ...
Could not find a part of the path '\\?\d:\restore\Memo\2014\filename-redacted'. => System.IO.DirectoryNotFoundException: Could not find a part of the path '\\?\d:\restore\Memo\2014\filename-redacted'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at Duplicati.Library.Common.IO.SystemIOWindows.FileOpenRead(String path)
   at Duplicati.Library.Main.Operation.RestoreHandler.DoRun(LocalDatabase dbparent, IFilter filter, RestoreResults result)

.NET version
Client 4.8.04084
Full 4.8.04084

Yet I also got this during restore… But if I run all the checks repairs and tests, there’s “nothing wrong”.
Failed to patch with remote file: "duplicati-b43c50965b03842b0b63b7d11db7fea73.dblock.7z.aes", message: System.IO.FileNotFoundException:

After running more tests and recreating the database, I think the reason for the error above is “generic duplicati integrity and consistency failure issues”.

So this just confirms the fact that even with latest version the serious integrity issues do persist. Most likely the best fix would be deleting the backup set and completely building it again. (The source files are still available).

Kind of getting seriously tired with these issues. But let’s hope that the problems are resolved at some point.

After all the repair operations, building database, repairing, running NEW backup set, and running restore, the files are still missing and un-restorable. -> DANG.

Reproducible issues in Canary are always worth looking at. Canary is there to be the first user testing activity and can fail massively, though this is rare. Issues that are fixable are often ones that give clear, direct steps from scratch with as few dependencies as possible, to raise chance developers can see.

Duplicati has a fix checked in for an issue some people saw in the new path handling, however it’s not available in a Canary yet. As an alternative, it would help to try older compatible releases like 2.0.5.108.

Seeing that work would mean there’s definitely a new issue in Canary. Or maybe this is an older issue. Either way, starting with a simple backup and seeing where it breaks between there and yours helps…

Although my restore work on 2.0.5.111 is fine, it’s guaranteed that my specifics are different. What’s the smallest simplest case that fails for you? After knowing it, others can try it. It’s possible there’s an issue.

@Sami_Lehtinen, I wanted to throw out some thoughts regarding the problem you’ve encountered.

The Canary you’re running includes some changes I made to allow for files with embedded wildcard characters to be restored. This changed code in the file restore GUI and the internal processing of file filters (#4302) so I am very interested to know if I’ve broken something.

On the subject of the funny prefix, most file paths will be prefixed with “\\?\” internally before being passed on to .NET to support more file name characters under Windows. So low level errors like the one you reported will show paths with that prefix.

Generally I think that if Duplicati had encountered an error while actually performing the restore step, you would have gotten an error about it at that point. But the error you’re seeing is coming from the verification step. That makes me think that maybe the restore step thinks those files don’t need to be restored for some reason, but the verification step thinks they should have been restored and so complains about it. If Duplicati didn’t think the files needed restoring, it wouldn’t create the associated directories either, which would account for the “Could not find a part of the path” error at the verification step which I think is complaining about a missing component of the path leading up to the file in question.

Is there any pattern you can see in the files that aren’t being restored? For example, do their source paths (i.e., the directories and file names where the source files reside) contain any interesting characters, like “?” or “*”, or do they start or end in “.”? Are symbolic links involved?

I’m not sure if this will help, but one thought might be to try using the Duplicati command line tools to list files in the backup and to perform targeted restores, maybe first of a file that you know can be restored to confirm you’ve gotten the command line right, then of one of the files that fails to restore. If you get different results by going through the command line, that might give us a clue. I’m not sure if that’s something you would be comfortable trying, and like I said, I’m not sure if it would help.

Of course, if you were able to succeed with any of @ts678’s suggestions, that would be very helpful.