Explain USN option

Support
1

Hi can anyone help me understand USN options

This is only applicable to NTFS Partitions.
Does setting it to on help faster searching for files for backup.

2

The USN is not currently working (option does nothing).

The idea is that the NTFS disk has sequence numbers for each operation.
By storing the operation number once the backup is clocked, the next backup can just look at whatever has a higher sequence number. This makes it much faster to find files that have changed, as you get a list of potentially changed files, instead of having to read all folders from disk.

I disabled it as we discovered some cases where it would miss an update, and then would not notice that a file was changed.

1 Like
3

Hi,

I am using USN without problems for our company backup with HP Data Protector. Could you be more specific about the missed updates? As far as I know, it tracks file and metadata modifications. USN is also used by
Windows for Backup and Restore feature.

I checked the USNHelper.cs. There are few things that you should check:

  • is the USN enabled for the volume? It is enabled by default only for system volumes.
  • the USN journal has limited (configurable) size, do you check whether the size was exceeded? In that case you cannot use USN, you should warn the user to increase the journal size instead.
  • you should also check the case where NextUsn approaches MaxUsn and also warn the user

Use fsutil usn to query/configure the journal.

Kind regards, Jan

4

I was not implying that USN is faulty, it is more likely that Duplicati’s USN implementation is faulty. We rolled out a test version and a few users reported files missing in backups, so we deactivated it. I have not looked at it since, so I do not have more details.

5

Thanks for clarification Ken and thank you for Duplicati, Jan

6

Sorry for pumping this up, but what is current status of USN? Is it reliable on Windows?

I have backup job with 1,5M files running 25 - 70 hours. So I have some hopes that USN could help : )
Thanks

7

I didn’t know about the “test version” that came and went. The current USN implementation start:

Implemented backup based on changes recorded in NTFS USN journal #3184

v2.0.3.7-2.0.3.7_canary_2018-06-17

Added support for using the USN numbers for faster backups on NTFS drives, thanks @dgehri

v2.0.4.5-2.0.4.5_beta_2018-11-28

USN support on Windows

and doesn’t seem to have any clear open issues besides:

With --usn-policy=On, source deselection retains files and folders from removed items #4071

which (as failure to remove files when config tries to remove them) is probably more tolerable.

Full scan may also occur even with USN because journal has a finite size and might overflow.

So I have some hopes that USN could help : )

Certainly seems worth a try. Despite lack of issues, feel free to watch carefully and report any.

8

I have submitted a pull request that should resolve issue 4071.

9

Wonderful. Thanks! BTW I’m not sure the pushbutton “Exclude” button changes are detected either, however whatever can be done to reduce the amount of unwanted retentions on change is good. :wink:

10

What do you mean by ‘pushbutton Exclude’? From my testing, any changes to either the selection list or include/exclude filtering are detected and forces a full scan. If pushbutton exclude is something different from normal filtering, I can look into it!

11

image

so the question is whether anything picks up changes in the Exclude checkbox settings shown above?
Being able to test will probably be easier now that your other fix is in (having looked at what it’s doing).

12

Ok, thanks… let me see if those checkboxes are included in the emitFilter portion.

Edit: Just looked, changing those checkboxes does not trigger a change in the overall configHash. Will look into it some more…

13

I just submitted PR #4080 - it includes File Attribute filtering and Max File Size filtering into the config hash calculation.

1 Like