Desktop.ini generates a warning log when I backup google drive folders

Support
1

Hi, there.

I’ve started using Duplicati not long ago, and I looked for this specific issue on the forum with no success.
I have a couple of backups set to run on a daily basis for my work files/folders, source is my google drive folder in my desktop and the destination is another team google drive (set up with AuthID and everything, connection is working fine). However, everytime the backup runs it gives 200+ warnings, all of them related to the desktop.ini file that gdrive creates inside every folder.
Imgur

I’ve got 3 filters, two of them seem to work fine, except this one
Imgur

My backups work, so this is not a major issue. So, if anyone can help me with this, I would appreciate!

2

Didn’t look at it but I see that its just saying it has an access error on the desktop.ini files as you know since you put that into the exclude.

If its only that then its not a big deal as long as you can successfully restore the files you need to be backed up. So its only a message and icon annoyance in that case.

But, I have something similar happen with different files under a folder and the exclude of the folder doesn’t always kill the message. Like, just had a backup run and no messages. It will spit out messages again at some point under that folder.

There’s all kinds of things that can cause code to do something that wasn’t wanted so its looking like a bug. I haven’t looked at the file crawl code in relation to this nor exclude so don’t know why its doing it.

Actually, I do have one theory. If the files were previously backed up in that backup task (since the task total file crawl size and count don’t change from exclusion after the fact)… but there’s many possibilities.

3

Thank you for your reply.
In fact this message appears even if I don’t have the filter on.

Maybe this will be patched at some time, for now I’ll just keep dismissing the warnings.

4

Welcome to the forum @rgheno

This makes it especially important that information be gathered in order to see what’s happening now.

There are usually some error details that don’t make it into default run log, which just shows first lines.

Easiest way to begin may be a browser tab at About → Show log → Live → Warning during a backup.

Are Google Drive preferences set to Stream files or to Mirror files? Is Duplicati running as you?

I’m not so sure that gdrive creates these. To me, it looks like Windows. You can do your own research.
Regardless, if Duplicati is running as you (is it?) you can see if you can open one, e.g. using notepad.
They’re typically hidden files, so if you can’t see them, you can cd there to do notepad desktop.ini.

Your desktop.ini exclude should probably be done differently, but let’s look into FileAccessError first.

6

Oh. Just noticed you used exclude file. This should be “exclude files whose name contains” I would think. The exclude file I think requires a path for one file at one location. The following should help you to confirm what’s happening. Just did a test with that and “desktop.ini” entered into the field and it does look like it may exclude them but I didn’t look at it fully as below.

Ah ha. So you can get the actual reason as this is a catch all block by going to settings and enabling advanced settings for log file and the setting that enables that for verbose. After the backup where it happens has finished then open the file in notepad++ and search for the file showing in the warning. If it doesn’t happen then is nicer to delete the log file as it will increase in size and take up a lot of space fast.

I’m getting it from the file being in use from its application and the wrong exclude was somehow set.

For me, despite the exclude:

2022-06-22 21:26:19 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: [redacted]

System.IO.IOException: The process cannot access the file because another process has locked a portion of the file.

It also has

Including path as no filters matched: [redacted]

And exclude %HOME%\the folder the files are found under - exclude folder seems to have helped. I thought it was on that before lol.

Now it says

Excluding path due to filter: [redacted]

You can see how that matches up with what you’re seeing. Technically, the exclude file I would expect should filter it out and so it may be a different issue. Should show in the log like I did.

7

The post above this one ends at:

Original post image was a FileAccessError in FileEnumerationProcess. You cited a different problem, however underneath both is probably some sort of complaint from filesystem that needs to be logged.

Log to file by Advanced options on screen 5 is good for rare errors but, as you note, has drawbacks.

was for the original consistent case. Yours may be different. While using Verbose level gives more info, starting at Warning seemed good to see the Warning, when original post seems to already know name.

Either form of enhanced logging will do, and either way, some fiddling to get the right level may happen.

Filters describes syntax underneath the tries-to-help GUI. To see it, use three-dot-menu Edit as text.

That’s my thought too. One needs a * wildcard to allow the first part of such paths, e.g. *\desktop.ini
For troubleshooting filters, there’s not only the Verbose log, but also the The TEST-FILTERS command.

8

Either way including more may benefit in better diagnosing. Notepad++ opens the large file very quickly so it makes little difference in including more. Just shouldn’t be posted online :slight_smile:

Oh, so the “exclude file” can also be used with wildcard? They should probably make the UI more clear. Think I tried that before with something there to no effect.

Maybe I tried *filename instead of *\filename. To me, either should work.

Oh. Whoops. Too tired apparently. I looked before I posted and was sure it was the same at the time lol.

Yeah, it still contains an ex message for error that isn’t found here so it should work out the same anyway for diagnosing. Need that message.

9

Maybe, unless it doesn’t. Too much information makes relevant information harder to find.
Try to “skim” a Profiling log. For even more fun, add --profile-all-database-queries

  --profile-all-database-queries (Boolean): Activates logging of all database
    queries
    To improve performance of the backups, frequent database queries are not
    logged by default. Enable this option to log all database queries, and
    remember to set either --console-log-level=Profiling or
    --log-file-log-level=Profiling to report the additional log data
    * default value: false

It’s best to not assume users have specific software or to expect installation and learning.
Another that can open huge files is EditPad Lite. Specialty tools like glogg/klogg exist too.
My big logs look like they’re about 20 GB. At some size, you don’t want it all in memory…

I’m not sure. It would be most certain what you get if you type it into Exclude expression.
The GUI tries to help, but as you say it’s not clear exactly how to use it. Manual is weak too.
One way to see what the GUI builds is with Edit as text as mentioned. Exclude file is

-*\default.ini

so (if I tested right), it did what I wanted, and made the same filter as Exclude expression
however I can’t promise it would always do that. Much testing and JavaScript study needed.

“They” is the community. If nobody (including you) will help with UI or docs, change is slow.
Filters are a pretty messy area, easy to get wrong. Arguably, Exclude by attribute is worse.

Regardless, we seem in agreement that we need the message details, obtained somehow.
If you look at these much, does it look to you like logging two lines instead of one will help?
Maybe that could be a GitHub request, although these days a pull request will be far better.

Example:

2019-03-25 18:55:53 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\PortableApps\Notepad++Portable\App\Notepad++\updater\
Alphaleonis.Win32.Filesystem.DeviceNotReadyException: (21) The device is not ready: [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy68\PortableApps\Notepad++Portable\App\Notepad++\updater\]
   at Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
   at Alphaleonis.Win32.Filesystem.File.GetAttributesExCore[T](KernelTransaction transaction, String path, PathFormat pathFormat, Boolean returnErrorOnNotFound)
   at Duplicati.Library.Snapshots.SystemIOWindows.GetFileAttributes(String path)
   at Duplicati.Library.Utility.Utility.<EnumerateFileSystemEntries>d__23.MoveNext()

Logging the whole stack trace, while sometimes helpful (there’s that issue of “how much” again)
might be a bit much. Some can get long. The first line after the logged error often seems helpful.
I don’t know the logging code, but if it now grabs only one line, maybe it can pick up two instead.

10

I’m only commenting on one other thing here and I’m sure its included in the issue in some way.

If you do “exclude file” of eg “**.lock” (one star the formatting is being a pain) and (so “*.ini” should as well) then after hitting next or moving on to a different page and go back then it will have switched it to “file extension”.

That means they are doing changes in the code as to what the user chooses. Which I think is why I’m seeing excludes changing compared to what I originally did and am scratching my head.

When I originally excluded the folder in my first post in here, I was sure it was set right and if Duplicati incorrectly adjusts it then it wouldn’t have worked and I’d be left thinking it was a bug with Duplicati which is what I did.

Either way, the user is specifically entering in selections and values. It probably should not be adjusted that way. Validation to me should be to alert the user as its being entered, not to adjust blindly after the fact. The user should be aware or confusion is chaos.

11

Not quite. User input is turned into filter syntax. When GUI needs it again, that’s reversed.
This tries (not sure how well it succeeds) to explain what user did, using dropdown labels.
You can also look at Commandline or job export to view --exclude and --include lines.
Those match the minus and plus signs in the filter syntax seen in the GUI three dot menu.

12

I’d have to look at the code but its messed up. If the user chooses exclude file then it shouldn’t try to change it to file extension without the user deciding that or otherwise knowing about it. It just blatantly does it. That’s the main point.

While its correct in this case that doesn’t mean all cases are and I’m sure I’ve run into incorrect ones or I’m eating too many mushrooms.

So, if say you do “exclude folder” folderName then next and go back as you want to add a path then its changed to “exclude dir whose names contain” and you enter the path and now its borked. You selected “exclude folder” and have to re-look at it. That was my issue.

Now however, after you change it back to “exclude folder” and next and back and it will no longer change even without a path. So its better at that point.

I’m not sure what you’re typing in the reply but there’s a bug here lol :slight_smile:

13

This is why I wanted to focus on the topic and not get lost in a filter discussion. But here we are.

If I do Exclude file of *.ini (using backticks while typing that, to reduce forum reformatting):

-*.ini

is the filter expression in Edit as text

If I do Exclude extension of ini the filter builder builds the same thing. Two ways to get that.
Duplicati does not remember the way, so on the return trip it picks one that works – the second.

There’s been talk of trying to find a way to remember the GUI dropdown choice, but remember
that whatever is devised needs to fit within the existing option system like Commandline shows.
If you have a proposal at that level, feel free to suggest how it be done. Better yet, go and do it.

If you think the code is doing something wrong, that’s why you’d look. I think some code is here.
The goal is either to do an arguably-correct (which is hard to do with the unclear rules around it)
round-trip from the stored options, or find some tolerable way to remember the actual dropdown.

Was making a reply to an earlier version of your post, which you have edited three times – so far.
Let me see if I can figure out the new addition which gives an actual case (though not very clear).

Exclude folder of folderName makes -*folderName*\ which seems a bit looser than asked.
Using Edit as list calls that Exclude directories whose names contain of folderName.
This seems a correct interpretation of the seemingly extra-loose filter string that the GUI created.
Using Edit as text to remove the extra-loose trailing * (before backslash indicating folder) got
Exclude folder of *folderName. If adding path is important to your case, please describe well.
“borked” is not an adequate description. Step by step, with details please (or maybe my test is it).

1 Like
14

My edited reply has the steps to reproduce. This is with current beta. I’d be surprised if its been fixed in master as of yet.

I feel this is still on topic and its a possibility as to what happened. There’s no way to know if any user with an incorrect filter type ran into this.

But basically:

  1. add new filter

  2. change value to “exclude folder”

  3. enter folder name (any name will do)

  4. next

  5. previous

  6. Notice filter type has changed.

  7. Change back to “exclude folder”

  8. Next

  9. Previous.

  10. Notice its now correctly staying as “exclude folder”

Note: Path doesn’t matter. You don’t need to enter one. Its just changing the type during one browse back while not changing it during the other after second type change. So its having two minds.

I had to edit a bunch of times as its all happening live lol.

I’m trying to find the code for this. Any idea where it is? So far my searches won’t find the specific class.

I’m just about done with it though. Now that I know about it, I don’t feel so crazy. If they fix it or not its up to them. I probably won’t other than make a git issue at most. End of edits lol.

15

Using a wrong filter (with help from little documentation) probably allowed files access.
Beyond that, I don’t see how a filter itself could make an access warning, but we’ll see.

A test could be done without any filters at all, e.g. backing up one specific troubled file.
The easy test (if Duplicati is running as this user, but we don’t know) may be notepad.
For that matter, type would probably do. The default.ini I found are just text inside.
Why the ones in the OP can’t be accessed is unclear. That’s the primary question IMO.

There is no such thing as a filter type internally in C#. That’s an artificial concept created in GUI.

I explained all of this just above. Two ways to get to filter expression. Reverse trip must pick one.

Yes, and I gave you the link:

EDIT:

and I suspect rx helps decide what it is, and splitFilterIntoTypeAndBody later may be part of that.
My JavaScript is not enough to pursue this, but you might be amused that I’m searching in Notepad++.

16

You seem to be misunderstanding me. Eg when I say “filter type”, I’m not referring to c# .

I can reproduce two effects on Duplicati most recent release from git here using a fresh OS that hasn’t had Duplicati before and doing the provided steps. Duplicati does two different things when it should be doing one.

You also misunderstand that the user above has a wrongly inputted value to the exclude file so that it will not exclude all desktop.ini files. The issue I present clearly can cause this to happen and is 100% relevant as it will change the original choice. Oh, the error itself is besides the point if the exclude had worked it wouldn’t be a thing here.

I will deal with it on my own. Thanks anyway :slight_smile:

17

@rgheno give this a try, from my quick testing it seems to work.

Use “Exclude regular expression” with \w\:\\.*\\desktop.ini|\w\:\\desktop.ini

image
image791×282 9.73 KB

Tested against these folders at regexr.com

C:\test\desktop.ini
D:\abc\def\desktop.ini
F:\desktop.ini
C:\users\my user\desktop\new folder\important stuff\desktop.ini

After adding the filter to a test backup job. Now in the GUI when I select any folder that contains a desktop.ini file, it now excludes that file with a red X all on it’s own.
You may be able get away with just using \w\:\\.*\\desktop.ini, the |\w\:\\desktop.ini is only for files directly off the root like “F:\desktop.ini” otherwise if it’s in a subfolder the first part should be enough.

Hope that helps.

18

You refer to GUI, right? C# reference was to say (correctly I hope) that GUI filter type is a step removed.
I think ultimately what matters in terms of the C# code and actual filtering is what Edit as text shows.

No misunderstanding, but a politely stated request that we chase the access warning instead. Notice:

Furthermore:

This is where we diverge. That’s the mystery that I want to solve. Fixing filters “should” be easy.
We both suggested versions that should do the job, but got caught in JavaScript GUI oddities…

I’d note that the original post has an Exclude file of desktop.ini, and that one seems solid.
At least it passes the Next → Previous test, but it was the wrong syntax for the desired result…

I think I see what you’re getting at. You’d like step 10 to show the same as step 6, correct?
Both follow the visual pattern of Exclude folder and a folder name, yet the result varies?
Did you look at Edit as text to see what was set up? That explains this, but worries me.

Step 6 has left actual filter at -*someFolder*\ which is the extra-loose one I’d mentioned.
Step 10 has left actual filter at -someFolder\ which is quite a bit different in wildcard uses.
That might be why it sticks, but it’s probably not what you actually mean to be filtering with.
This gets back to the issue of there not being a clear definition (that I find) of GUI behavior.

I’m still not sure what things you mean, but I see identical GUI view making different filters.
On the return trip back to list view after Previous, dropdown varies, likely from the above.

19

Interesting. I wonder how smart those indicators are? I can’t get non-regex wildcard to redden things.
I actually had to do backup with *\desktop.ini wildcard to test, but a restore did in fact lack the file.

As a syntax nit, the period in desktop.ini for a regular expression technically deserves backslash too.
Do you know of any advantage in this case other than wildcard apparently doesn’t do color change?

Regardless, there’s yet another option to filter out the troublesome files, but I’d still like to know what FileAccessError is being raised. All of this depends on what @rgheno is willing to do to help look.

20

You are correct, the proper syntax should contain desktop\.ini but doesn’t seem to change the result either way for some reason… (I’ll look into it further tomorrow). The auto marking of excluded files was a completely unexpected bonus.

Advantages, Duplicati seems to like working with regex filters and so long as your expression is correct you get the expected results. I know most people probably don’t want to use regex but given it’s an option, I figured I’d mention it.

21

Here is a slightly smaller expression \w\:\\.*\\?desktop\.ini, that catches any “desktop.ini” file regardless of path depth.

FYI: You can change the filename to any filename you want. To exclude a file named “mySecretWords.txt” change the expression to \w\:\\.*\\?mySecretWords\.txt.