Empty files don't backup and generate warnings

Simon_Ratcliffe · February 4, 2022, 1:33am

Node has a really bad have of creating zero byte files within the node_modules directory. When Duplicati finds one of these it throws a warning like this:

Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: D:\Develop\bom-radar-card\node_modules\tsconfig-paths\node_modules\.bin\json5

Currently I have over 500 warnings (I assume they are all the same, but it is time consuming to check). Is there a way to eliminate these warnings? Ideally Duplicati should actually back them up correctly.

I am running on Windows. The files are on a NTFS fixed disk. Version: Duplicati - 2.0.6.3_beta_2021-06-17

Cheers
Simon

Simon_Ratcliffe · February 4, 2022, 2:12am

Ok, more investigation. These empty files are actually symlinks that the OS thinks are broken.

D:\Develop\bom-radar-card\node_modules\tsconfig-paths\node_modules.bin>dir json5
Volume in drive D is Data
Volume Serial Number is 2868-0E71

Directory of D:\Develop\bom-radar-card\node_modules\tsconfig-paths\node_modules.bin

05/12/2021 10:13 AM json5 […]
1 File(s) 0 bytes
0 Dir(s) 118,555,746,304 bytes free

D:\Develop\bom-radar-card\node_modules\tsconfig-paths\node_modules.bin>attrib json5
The target of the symbolic link D:\Develop\bom-radar-card\node_modules\tsconfig-paths\node_modules.bin\json5 does not exist

But when I open the project in VS Code (using containers) the links are correct.

I have tried using symlink-policy in advanced options, but I still get the warnings not matter what option I choose. Is there any way to remove the warnings?

drwtsn32 · February 4, 2022, 4:50am

Hi, I did a test and created a symlink that pointed to a non-existent file. I only see warnings when --symlink-policy is set to “follow”. If it’s set to “store” (the default behavior) or “ignore” I don’t get any warnings.

Simon_Ratcliffe · February 4, 2022, 5:29am

So if I list the symlink is a regular command prompt I get this.

D:\Develop\bom-radar-card\node_modules\tsconfig-paths\node_modules.bin>dir
Volume in drive D is Data
Volume Serial Number is 2868-0E71

Directory of D:\Develop\bom-radar-card\node_modules\tsconfig-paths\node_modules.bin

05/12/2021 10:13 AM .
05/12/2021 10:13 AM …
05/12/2021 10:13 AM json5 […]
1 File(s) 0 bytes
2 Dir(s) 113,117,831,168 bytes free

Using a git bash shell I get this.

simon@platinum MINGW64 /d/Develop/bom-radar-card/node_modules/tsconfig-paths/node_modules/.bin (master)
$ ls -l
total 0
lrwxrwxrwx 1 simon 197121 19 Dec 5 10:13 json5 → …/json5/lib/cli.js*

That link is correct and exists in the filesystem. I agree that using store which I believe is the default should be just fine. I don’t understand how the native tools have no idea, while a 3rd party shell fully understands what is going on.

drwtsn32 · February 4, 2022, 3:30pm

Intersting… my cmd prompt has no trouble showing me three different types of links (file symlink, dir symlink, and junction point):

Capture

The hard link is the only type that doesn’t really show as such in a cmd dir command.

Here’s how it looks in bash for me:

Simon_Ratcliffe · February 7, 2022, 9:51pm

Ok, so after lots more investigation. I actually think it is a bug in WSL2 and how it creates soft links.
If I open a WSL shell create a text file, create a link to it, in WSL it looks fine.

If I then go to that directory in git bash it also looks fine.

But if I do the same in a command prompt this happens.

This is really weird. The files that I was getting all the Duplicati warnings from where create in a Docker Desktop container that uses WSL under the hood.

I think for now I might just exclude any node_modules directory from the backup since it is something that can easily be redownloaded if needed.

Simon_Ratcliffe · February 7, 2022, 10:11pm

Just found this, but it looks like Microsoft are ignoring it.

github.com/microsoft/WSL

access Linux symlinks from \\wsl$

opened 04:12AM - 21 Apr 20 UTC

trentis

feature file system

**Is your feature request related to a problem? Please describe.** I want to be… able to open a symlink from the **'\\wsl$\Ubuntu'** directory. The reason for this Is I'm writing a program that would read an elf binary and import the required objects to run it inside the directory that the binary is in. This is from a virtualization perspective the kernel we use is linux based and Im purely looking at a way to improve development of apps on Windows. The only thing I want to do with the files is be able to read them and copy to a location without any changes. But most elf objects link to shared libraries that are symlinks but Windows doesn't seem to know what to do with that file. Any sort of open on that file results in system could not find the name specified. I assume this is because the explorer is attempting to open the symlink but its at a path it doesn't understand like **'/lib/64'** instead of **'\\wsl$\Ubuntu\lib\64**'. **Describe the solution you'd like** Maybe on the file open in the windows explorer prefix it with the location of the symlink. **'/lib/64**' turns into opening **'\\wsl$\\Ubuntu\lib\64'** I imagine this would be a big task probably. **Describe alternatives you've considered** I could probably resolve this from the perspective of installing another daemon on my subsystem and just do the calls there. I'm mainly querying about this so I can improve UX of our application and the user doesn't have to install two of the same thing. Is there a use case where we could set the LD_LIBRARY_PATH on our Windows machine to point to one of the sub system shares? **Additional context** Basically want to read the ldd of an elf binary copy the objects into a windows directory. Proceed to do that recursively for everything that binary requires.

drwtsn32 · February 7, 2022, 11:00pm

Good catch… yeah that should not be a junction!