In order to make it easier to tackle the APPDATA problem, how about building a repository of common exclude and include filters that people can copy and paste into their backup jobs?
(Just to keep things simple, let’s limit this topic to Windows. Feel free to start a similar topic for another OS)
One starting point might be these posts from crashplan:
Looks like you can’t override an exclude filter with an explicit “+” include entry, so I’m probably going to end up dropping the exclusion for the Windows directory so I can backup the Duplicati-server.sqlite database in C:\Windows\System32\config\systemprofile\AppData\Local\Duplicati.
Yes tried a bit earlier without any success. Tried it at the start of the list as well as the end on the off chance it mattered if the include was read in first or last.
It’s possible I messed up the syntax so curious to hear if anyone else gets it working. I had assumed that an explicit include would trump an exclude as well.
Include some files, exclude others. Now let’s define a filter that does both of the above. First it excludes @eaDir specifying -/@eaDir/. Then it includes only JPG files specifying +.jpg. The problem here is, that Duplicati includes all files and folders per default. This means that e.g. /photos/movie.avi will also be part of the backup. To make the including rule effective an additional rule is required that excludes all files that do not match any of the current rules. The filter must say “exclude this, exclude that, include this but nothing else”. The best rule for “but nothing else” is a regular expression that excludes all files. It is -.[^/] on Linux or Mac, and on Windows the rule is -.[^\]. The rule says “exclude everything that is not a folder”. The final filter then is -/@eaDir/ +.jpg +.jpeg -.[^/]. Duplicati will process all folders but @eaDir/ and it will include JPG and JPEG files but exclude all other files.
Will enabling this (or more generally, adding any new exclusions) result in these files eventually getting purged from the remote dblocks or will the versions prior to the change in filters/exclusions stay there until they would get removed from being too old?
Also my reading is this should be available in the latest canary but I cannot find it anywhere in the GUI?
Duplicati should treat previously included files that are now excluded due to filter (or folder) changes the same way as if they had been deleted, so versions (and eventually the backed up files themselves) will be removed from the dblocks based on whatever “keep” rules are set in the backup.
In other words, filter and source folder changes do NOT cause an immediate purge of dblock contents.
Windows has a default exclusion list already, it is in the registry at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToBackup
I posted the question on the forums, but this might be a good way to generate the list instead of having a manually generated one. Or in addition…
I could see maybe offering to import it on first install into the Duplicati config (or one time importing it silently). After that though, it seems like it would be somewhat confusing if the registry settings were used and files were being excluded from Duplicati backups that you have to go manage completely outside of Duplicati?
I sounds like the GUI might need the “Source data” Exclude section to be updated to include more detail on exclusions from less obvious sources (such as external or hard coded internal lists).
Are you thinking that some users (or applications) will edit the list in the registry?
That would be strange, but I guess anything added to this key is really not meant to be backed up.
Not sure how to handle that, because the change would happen via another program that is not related to backup, and the user would not want the change the other program makes.
I don’t think we should do anything to note changes that something outside of Duplicati makes to those registry keys.
I’d suggest that these default exclusions using this registry key can be implemented in one of two ways:
Those keys are read during each backup run and added as a dynamic exclusion. Similar idea to the “temporary files” exclusion that excludes files with the “temporary attribute set”, in that it is a dynamic operation.
If someone really wanted, add an option to not use the windows keys.
Make it an option “–use-windows-default-exclusions” that has to be activated.
In terms of having an option to not follow Microsoft’s specification, my gut says that is a mistake. I’ve installed enterprise backup software for about 20 years, and have never seen documentation that notes that these registry keys are used to exclude data. At best, I’ve run across it in a knowledgebase, and more often its been an engineering guide. It may be part of being qualified by Microsoft, I don’t know–but nowhere can I remember there being a “ignore windows files/dirs to skip”.
And if someone is advanced enough to modify those registry keys, all the more power to them.
#1 makes the most sense to me, so long as it’s clearly stated somehow/somewhere to the end user that there are exclusions being enforced outside of Duplicati config. A way to bypass it when/if Microsoft does something dumb isn’t ever a bad idea .
Another checkbox for “exclude OS defined files” or something would take care of both.
I added --default-filters=Windows to my existing backup definition.
Still the backup complains about NTUSER.DAT:
Warnings: [
Failed to process path: C:\Users\vegar\NTUSER.DAT => The process cannot access the file ‘C:\Users\vegar\NTUSER.DAT’ because it is being used by another process.,
Failed to process path: C:\Users\vegar\ntuser.dat.LOG1 => The process cannot access the file ‘C:\Users\vegar\ntuser.dat.LOG1’ because it is being used by another process.,
Failed to process path: C:\Users\vegar\ntuser.dat.LOG2 => The process cannot access the file ‘C:\Users\vegar\ntuser.dat.LOG2’ because it is being used by another process.,
…
]
Yes, I would think so…I’m not sure why it’s not working for you.
If you manually add those filters (with back slashes instead of forward) does it work for you? I’m curious if the issue lies somehow in the wildcard vs. the application of the Windows filter set.
.