Win10/NTFS: symlinks to non-existing targets are not backuped

Features
1

I noticed that Duplicati does not backup symlinks on NTFS file systems (Windows 10) if they point to targets that do not exist in the NTFS filesystem (the target actually exists, but not from the Duplicati process’ context, e.g. it exists in Cygwin or WSL2). Instead Duplicati issues a warning:

[Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: .

Is this issue known? Will this be fixed? I searched the forum but did not find anything.

I use the latest version Duplicati - 2.0.6.3_beta_2021-06-17.

2

Welcome to the forums @markdoll.

If you don’t need to or can’t backup the data at the end of the symlink you could set the symlink policy to ignore but it will then ignore all symlinks in the system.

--symlink-policy=ignore

Hope that helps.

3

Please provide detail lines that follow that one. Maybe watch About → Show log → Live → Warning

Official issues are tracked in GitHub Issues, although lack of forum talk probably means it’s a rare one which isn’t likely to be prioritized. There are hugely more issues than there are volunteers to fix issues.

Still, anyone can open an issue as they like to get it on the record, though I see some sort of like yours.
Without details, I can’t say more, and WSL has been changing over the years. You can browse around.

4

Hi @JimboJones, @ts678,

thanks for your answers. This is what the Live log shows for such a link:

{“ClassName”:“System.IO.IOException”,“Message”:“The file cannot be accessed by the system.\r\n”,“Data”:null,“InnerException”:null,“HelpURL”:null,“StackTraceString”:" at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)\r\n at System.IO.FileStream…ctor(String path, FileMode mode, FileAccess access, FileShare share)\r\n at Duplicati.Library.Common.IO.SystemIOWindows.FileOpenRead(String path)\r\n at Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.<>c__DisplayClass1_0.<b__0>d.MoveNext()",“RemoteStackTraceString”:null,“RemoteStackIndex”:0,“ExceptionMethod”:“8\nWinIOError\nmscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.IO.__Error\nVoid WinIOError(Int32, System.String)”,“HResult”:-2147022976,“Source”:“mscorlib”,“WatsonBuckets”:null}

I investigated a bit more and figured out that Duplicati actually does backup symlinks to non-existing targets, but just not those that were created under Cygwin and WSL2.

Interestingly, if I create a tar archive under Cygwin or WSL2 that includes such symlinks, and extract those with 7-zip (latest version 22.01), Duplicati backups (and restores) those symlinks successfully.

Searching for the reason I found that 7-zip creates symlinks with targets given in UTF-16, while Cygwin & WSL2 use UTF-8 [edit: if the target exists, Cygwin and WSL2 use UTF-16] as usual for today’s Linux (“link” by WSL2, “link2” by 7-zip):

PS C:\Software\test1> fsutil.exe reparsepoint query .\link
Reparse Tag Value : 0xa000001d
Tag value: Microsoft
Tag value: Name Surrogate

Reparse Data Length: 0x7
Reparse Data:
0000:  02 00 00 00 61 73 64                              ....asd
PS C:\Software\test1> fsutil.exe reparsepoint query .\link2
Reparse Tag Value : 0xa000000c
Tag value: Microsoft
Tag value: Name Surrogate
Tag value: Symbolic Link

Reparse Data Length: 0x18
Reparse Data:
0000:  06 00 06 00 00 00 06 00  01 00 00 00 61 00 73 00  ............a.s.
0010:  64 00 61 00 73 00 64 00                           d.a.s.d.
PS C:\Software\test1> Get-Item '.\link' | Format-List -Property Target


Target : {}



PS C:\Software\test1> Get-Item '.\link2' | Format-List -Property Target


Target : {asd}

Windows Explorer shows the 7-zip/UTF-16 version as a symlink and the WSL2/UTF-8 version as a file of 0 byte. Cygwin and WSL2 can correctly interpret both kinds, i.e. both UTF-16 and UTF-8 coding of the target seems both to work just fine [edit: e.g. with ls]:

$ ls -dl link
lrwxrwxrwx 1 li05910 li05910 3 Sep  9 12:59 link -> asd
$ ls -dl link2
lrwxrwxrwx 1 li05910 li05910 3 Sep  9 13:01 link2 -> asd

I have no idea how Cygwin and WSL2 could be forced to create symlinks with targets in UTF-16 (if that is at all possible). Though, I wonder why Duplicati needs to interpret the target data at all instead of just storing it on backup as it is (as opaque byte stream) and restoring it the same way. Maybe the system calls used by Duplicati do not allow it?

If there is no easy/near term fix for Duplicati, I probably then need to create tar archives of Cygwin and WSL2, respectively, and backup those tar archives with Duplicati instead. :frowning:

5

About the only guesses I can find from rummaging in the code are that the link target read might be:

and if it backed up nothing (as opposed to some “wrong” thing), I wonder if it caught an exception?
If you look in your restore and the name of the link is missing, possibly you got the “nothing” result.
If somehow there’s a name there, you might be able to find the link target around, maybe in dblock
which is just a zip file holding blocks which are used for storing all kinds of things that need storing.

File.GetLinkTargetInfo Method (String) doesn’t talk about Unicode, and seems light on its details…

GetFinalPathNameByHandleW function (fileapi.h) is maybe the Win32 underneath, also light docs.

Due to lack of volunteer developers from the community (any around?), sadly this is almost a given.
If you can figure it out well and make a pull request, there’s a moderate chance that those still go in.
If you can just do research and analysis, it might still be worth an issue on that. Might help someday.
As a note on “someday”, I found one person saying .NET 6 has FileSystemInfo.LinkTarget Property