Case sensitivity regarding filenames

[runmode]

Is it true, Duplicati isn’t case sensitive regarding file names?
It does not recognize a renamed file as a new version, if only capitals and small letters have been exchanged. Example:

Create a file “dAILY nEWS.txt” and run a backup
→ a new version is created and restore files shows dAILY nEWS.txt

Then correct the spelling to “Daily News.txt” and run a new backup
→ no version is created and restore files shows dAILY nEWS.txt only.

Is there a way to include such changes?

Edit: Duplicati version 2.0.6.3_beta_2021-06-17 / Windows 10

[gpatel-fr]

Hello

I can’t repro on Win10 with Duplicati 105.

[runmode]

My Verison is 2.0.6.3_beta_2021-06-17 (I added to my post)

[gpatel-fr]

Try to do and check your changes in a terminal window, for example
cd yourdirectory
dir “dAILY nEWS.txt”
→ Should display “dAILY nEWS.txt”
do a backup

then

move “dAILY nEWS.txt” “Daily News.txt”
dir “dAILY nEWS.txt”
→ Should display “Daily News.txt”
and then redo a backup.

[ts678]

@gpatel-fr

Try to do and check your changes in a terminal window

The behavior here on Windows 10 and Duplicati 2.0.6.106 is that it notices Explorer rename too –
if the backup gets to the file by having its containing folder configured. Direct config doesn’t notice.

[runmode]

Back home, I run a couple of tests:

1. The move-method did not change anything compared to rename (files with just a capital/minor change are not recognized as new).

2. I can confirm the rename directory test: that files get recognized as new, if the containing folder name has been recognized as new.

3. The folder test worked recursively: such kind of files deeper in the directory tree get recognized as new, even if only a higher level directory has been recognized as new.

[runmode]

I don’t know if this is a relevant hint, but may be worth a look:

Configure case sensitivity with Git:
git config core.ignorecase false

[ts678]

@runmode

if the containing folder name has been recognized as new.

My file in unchanged containing folder had a rename noticed.

[runmode]

@ts678

by having its containing folder configured. Direct config doesn’t notice.

I don’t quite understand what you meant with folder vs. direct “config”?

[ts678]

By config I mean what you checked in the GUI – the file itself, or the folder that contains it.

[runmode]

I have a folder tree in my backup configiration, and the files in question are located deep in sub-sub directories.

This is interesting, because the case insensitivity of NTFS and the system level commands for file manipulation sould be normal Windows behaviour, since ever, and I nearly accepted to live with it. This is at least my conclusion from reading this:

Case Sensitivity

You did’nt apply “fsutil.exe file setCaseSensitiveInfo enable” to your root folder, which is explained in this article, did you? Maybe a long time ago?

Just a guess. What else could explain the different behavour of our installations?

[gpatel-fr]

@runmode

The folder test worked recursively: such kind of files deeper in the directory tree get recognized as new, even if only a higher level directory has been recognized as new.

I have trouble reconciling this assertion with:

@runmode

I have a folder tree in my backup configiration, and the files in question are located deep in sub-sub directories.

Now what has been tried by @ts678 and myself is running a test backup on a given directory and changing a file under this directory. It worked. Then after your first reply I think everyone assumed that the problem was when selecting a file direct from Duplicati (a corner case if there is one, absolutely not worth the effort to fix). And then you seem to imply the reverse, that a file changed under a directory included in Duplicati don’t trigger a backup. I don’t get it. What’s the deal with WSL ? you are not doing some mixing up Linux and Windows by any chance ? If yes, that’s a very significant detail that you forgot to mention in your initial post.

Can you create a test backup with one directory and one file under it, no WSL, and post the results of doing a backup, preferably by exporting the job as command line and run it from a terminal window, add

–dry-run --console-log-level=verbose

to the command and just post the output, not the actual command line if you prefer to keep the connection details to yourself.

[ts678]

@runmode

You did’nt apply “fsutil.exe file setCaseSensitiveInfo enable” to your root folder, which is explained in this article, did you? Maybe a long time ago?

I very much doubt it. Here’s my current test area:

C:\backup source\case>fsutil.exe file queryCaseSensitiveInfo "C:\backup source\case"
Case sensitive attribute on directory C:\backup source\case is disabled.

C:\backup source\case>dir /b
daily news.txt

@runmode

What else could explain the different behavour of our installations?

I tested a tiny test backup, with zero advanced options. Please do so.

[ts678]

One suspect from advanced options that completely changes how changes in files are detected is:

usn-policy

This setting controls the usage of NTFS USN numbers, which allows Duplicati to obtain a list of files and folders much faster.

and for me, using USN results in a failure to notice rename even if a containing folder is configured.
Possibly other configuration differences exist, so either start simple then get complex, or vice versa.

[gpatel-fr]

Well, this should have been documented:

Real-time backup

Multiple volumes are supported, and tracked separately. USN support is only available for NTFS, and the case is handled accordingly for that filesystem (currently not case sensitive). The actual behavior if USN isn’t available for one of the volumes (file system not NTFS, platform Linux, …) depends on the value of -usn-policy: off = USN not used auto = Tries to use USN, silently (info message) use full scan for any volume where USN cannot be used on = Tries to use USN, use full scan for any vo…

[ts678]

@gpatel-fr

Well, this should have been documented

If you mean specifically this part, then it would be worth finding out what it means and if it still holds:

@Daniel_Gehriger

(currently not case sensitive)

Windows is thought of as case-insensitive but case-preserving, so in that spirit a backup should use case-sensitive change detection to preserve current case, even though either one works for access.

[gpatel-fr]

@ts678
Well, I tried USN on a fresh Win11 VM and things are not pretty. Seems horribly bugged. I renamed the file 3 times and when restoring, Duplicati is presenting 3 files

[ts678]

@gpatel-fr

when restoring, Duplicati is presenting 3 files

Do you mean like below, which I got after turning on all-versions – and haven’t tested further than:

however I think most of this was without USN because USN wasn’t picking up the rename properly.
If you can describe your test steps, I can see if I can reproduce a similar problem - for investigation.

[gpatel-fr]

Thanks for the reply. No I never touched all-versions options. BTW, the fact that after picking a version (the first one in your case), Duplicati could display all versions should be considered a bug by itself. But I’m not sure that it is the case.

Anyway, what I did on a vanilla backup without any options (except picking a backend), was to rename a file in different ways with USN on. After my tests and seeing the display problems I have cited, I turned USN off and did a backup again. Here is the result for restoring the version 0:

and preceding version:

I did not change anything between the 2 screenshots, only selecting a different version.
Needless to say, the real situation on disk is in both cases the one displayed in version 0 (created without USN)

[ts678]

Problem is not reproducible with this minimal test case which tries to follow the given description:

select parent folder
USN on
database deleted
destination deleted
create dAILY nEWS.txt
backup (versions: 1)
rename DAILY nEWS.txt
backup (versions: 1)
USN off
rename DAILY NEWS.txt
backup (versions: 2)
check version 0 -- OK DAILY NEWS.txt
check version 1 -- OK dAILY nEWS.txt

Only oddity was that USN ignored the case change. This ideally gets fixed, or at least documented.
See any other differences between my steps and yours?