File System Warnings from backup jobs

[Richard_DeVito_Jr]

I think there is a bug in the enumeration routines on this platform. Maybe it needs to be able to look at extended file status when reading network drives? Maybe the timing (timeouts) need to be extended slightly for files that take a second or two to decompress from the network server?
Thoughts? Thanks! radjr

• 2024-07-19 17:44:23 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\JEM Directory Project\CMS070113_1\ProductTypeFrm.aspx DirectoryNotFoundException: Could not find a part of the path ‘\?\Q:\JEM\JEM Directory Project\CMS070113_1\ProductTypeFrm.aspx’.
• 2024-07-19 17:44:23 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: Q:\JEM\Special Issue Mental Health in EM\dreamstime_xxl_197002960-Modified.psd IOException: The request is not supported.
• 2024-07-19 17:44:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: Q:\JEM\Special Issue Mental Health in EM\dreamstime_xxl_197002960-Modified1024.png DirectoryNotFoundException: Could not find a part of the path ‘\?\Q:\JEM\Special Issue Mental Health in EM\dreamstime_xxl_197002960-Modified1024.png’.
• 2024-07-19 17:44:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\JEM Directory Project\CMS070113_1\RegisterUser.aspx DirectoryNotFoundException: Could not find a part of the path ‘\?\Q:\JEM\JEM Directory Project\CMS070113_1\RegisterUser.aspx’.
• 2024-07-19 17:44:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Special Issue Mental Health in EM\dreamstime_xxl_197002960-Modified_2000x1333.png DirectoryNotFoundException: Could not find a part of the path ‘\?\Q:\JEM\Special Issue Mental Health in EM\dreamstime_xxl_197002960-Modified_2000x1333.png’.

[kenkendk]

@Richard_DeVito_Jr

I think there is a bug in the enumeration routines on this platform. Maybe it needs to be able to look at extended file status when reading network drives?

From Duplicati’s perspective, the mounted folders are “local” and it is treated this way, so no extra logic accounts for transient errors, and no timeouts are managed by Duplicati. It just opens the file and starts reading. Any delay or special cases should be handled by the OS.

I guess we could retry files that are not accessible, but do you have any suggestions? Have you seen another way of dealing with such errors in another application?

[ts678]

More background on this:

@Richard_DeVito_Jr

Thanks. Please see my other email reply. .Net does not handle extended file ops from other drivers well. We have seen this before. Let me see the details in procman and go from there.

and I think Process Monitor wasn’t run, but it’s definitely hard to read the goings-on at low level.

If later versions of .NET run any better, .NET 8 is coming. Beta is currently on .NET Framework.

[Richard_DeVito_Jr]

Thanks. When running ProcMan, what are you specifically looking for? Can I trigger on some event state to log the error?

It is some times though to tell if duplicati is running because it seems like nothing is going on…some way to set “verbose” mode?

Thank yoU!

[ts678]

@Richard_DeVito_Jr

what are you specifically looking for?

Like I said, it’s hard to read. Generally something that looks like an error, especially if it sounds like “The request is not supported” probably returned as some sort of a shortened internal code though.

We tend not to do Windows internals here. We tend not to even do Win32. It’s mostly C# and .NET (various), although the Linux side has to do some low-level work that goes around mono to Linux…

[Richard_DeVito_Jr]

Good Day. Still have the issue with file access. These are all files and folders on our server. It really looks like the file enumeration is not working properly. I watched the server as this backup was running and I have no console errors on the fileserver. The next time the job runs, it will not error with these files. I still think you are not properly handling extended file status properties and responses properly. Would appreciate if you can look into it again. What other logs do you need? Many thanks! Rich DeVito.

• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2.pdf DirectoryNotFoundException: Could not find a part of the path ‘\?\Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2.pdf’.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: Q:\ajrt\issue 16\Spring\layout\AJRT_V15_N2_Spring_2016_BC_ATRA.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Aini.pdf DirectoryNotFoundException: Could not find a part of the path ‘\?\Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Aini.pdf’.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2.qxp IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2.zip IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Aini2.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2_150DPI.png IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Aini3.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2_300DPI.png IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2_96DPI.png IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Angeletti2.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Front_cover_v15v2_original.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Angeletti_080022.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Spine_v15v2.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Clerveaux.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Thumbs.db IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Clerveaux2.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Title.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\JEM\Issue08\09_10\proof\JEM-6-5-00-Clerveaux3.pdf IOException: The request is not supported.
• 2024-08-16 13:05:24 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: Q:\ajrt\issue 16\Spring\layout\Profiles\ IOException: The request is not supported.

[ts678]

@Richard_DeVito_Jr

I still think you are not properly handling extended file status properties and responses properly.

Based on what experience, evidence, and expertise? You already got the reply from the author.

If you’re going to keep making this claim (also in original post), can you give technical explainer about what exactly you are talking about? Also talk about how some other programs are failing.

@Richard_DeVito_Jr

.Net does not handle extended file ops from other drivers well. We have seen this before.

Please expand on that, in detail. Who diagnosed this (or is that a personal guess?). Who fixed?
PowerShell is a .NET app. Has anybody ever come up with a reliable repro of issue using that?

@Richard_DeVito_Jr

Would appreciate if you can look into it again.

A developer level look is best done by developers, preferably the author. This would be easier if someone besides you can get it, otherwise you will have to do a lot of the “look” on your system.

The best I can do (as time allows – this is going in the wrong direction IMO – more on that later) would be to try to learn the Duplicati code design a little better in order to guide you in your tests.

We do not write .NET or Windows or your Windows driver. If the problem is there, a workaround may be possible, and priority depends on the frequency of the problem. This one is rare, but see

@ts678

Basically, Duplicati can only back up a working drive. Network drives are more likely to break.

for a similar case and my suggested steps. Don’t you have support (likely paid) for your platform?
Talk to them, see if they have ever heard of the problem scenario you have in mind. Original post here guesses at extended file status (which is what), and timing. This all sounds like guesswork…

The first step would be to see if you can find a small test case that shows the problem, ideally with data (or at least paths) that you would be willing to post (if that is an issue). Small reduces log size.

The other reason for a controlled test is you can determine whether or not it’s purely enumeration, or maybe data reading is also needed. The similar topic is currently waiting for work described as:

@ts678

If Mountain Duck support has some debug ideas beyond you running Process Monitor, try them.

You can try The TEST-FILTERS command in Duplicati to walk a folder without all the other stuff. Knowing whether simplifying things that way helps (or not) might be helpful to debug the drive…

Replace “Mountain Duck” with whatever support spot you have for problems involving your drive.

@Richard_DeVito_Jr

It is some times though to tell if duplicati is running because it seems like nothing is going on…

There should be status all over the place. I see you posted a GUI screenshot. See the status bar. There is detailed information on the specific job being run on the home page, even the files read.

@Richard_DeVito_Jr

some way to set “verbose” mode?

In GUI, you have the choice of About → Show log → Live → Verbose (or whatever level you like). Logging to a file is much better for big logs. Options log-file=<path> log-file-log-level=verbose etc. Verbose will show paths, which is why I mentioned privacy. You’ll also see timestamps to second, which may be helpful if timeouts are a suspect, but the smoothest flow will be if no data changes.

Channel Pipeline shows the internal flow. Your message shows the spot of the complaint. Seeing Verbose log (especially of known paths that you can describe) may help isolate enumeration fails.

Underneath all that is probably a Windows driver that Process Monitor can watch, but that’s way below the ideal level for Duplicati application workarounds – we can only control the application…