Exclude list for windows

I don’t think this was posted here yet. Based on this post and prepared for easy copy / paste to duplicati:

-[.*/(?:42|\d{8,}).*/(?:cp|~).*]
-[(?i).*/CrashPlan.*/(?:cache|log|conf|manifest|upgrade)/.*]
-[.*\.part]
-[.*/iPhoto Library/iPod Photo Cache/.*]
-[.*\.cprestoretmp.*]
-[.*/Music/Subscription/.*]
-[(?i).*/Google/Chrome/.*cache.*]
-[(?i).*/Mozilla/Firefox/.*cache.*]
-[.*/Google/Chrome/Safe Browsing.* ]
-[.*/(cookies|permissions).sqllite(-.{3})?]
-[(?i).*\.(class|obj)]
-[(?i).*\.(tmp|temp|cache|part|rbf|chk|dov)]
-[(?i).*\.(bac|bak|bkp|bup|back|dmp|dump|err|log|old|prv)]
-[(?i).*\.(asd|wbk|xlk|bde|bdi|bdm|bmb|bom|crc|bdb|bks|bps|mch|sol|idb|nch)]
-[(?i).*\.(bmk|chw|ftg|fts|gid|hhc|hhk)]
-[(?i).*\.cache.*]
-[:/Config\\.Msi.*]
-[.*/Google/Chrome/.*]
-[.*/Google/(Drive|CrashReports)/.*]
-[(?i).*/(Google Drive|LocalGoogle)/.*]
-[.*/Dropbox/.*]
-[.*/.*SkyDrive.*/.*]
-[.*/(My Cubby|cubby)/.*]
-[(?i).*/Cookies/.*]
-[(?i).*/Downloads/.*]
-[(?i).*/.*cache.*/.*]
-[(?i).*/.*log.*/.*]
-[.*Thumbs\.db]
-[.*/Local Settings/History/]
-[.*/NetHood/]
-[.*/PrintHood/]
-[.*/Cookies/]
-[.*/Recent/]
-[.*/SendTo/]
-[.*/LocalService/]
-[.*/NetworkService/]
-[.*/Start Menu/]
-[(?i).*/NTUSER.*]
-[(?i).:/WINDOWS/]
-[(?i).:/autoexec.bat]
-[.*\$RECYCLE\.BIN/.*]
-[.*/System Volume Information/.*]
-[.*/RECYCLER/.*]
-[.*/I386.*]
-[.*/pagefile.sys]
-[.*/MSOCache.*]
-[.*UsrClass\.dat\.LOG]
-[.*UsrClass\.dat]
-[.*/Temporary Internet Files/.*]
-[(?i).*/ntuser.dat.*]
-[.*/Local Settings/Temp.*]
-[.*/AppData/Local/Temp.*]
-[.*/AppData/Temp.*]
-[.*/Windows/Temp.*]
-[(?i).*/Microsoft.*/Windows/.*\.log]
-[.*/Microsoft.*/Windows/Cookies.*]
-[.*/Microsoft.*/RecoveryStore.*]
-[(?i).:/Config\\.Msi.*]
-[(?i).*\\.rbf]
-[.*/Windows/Installer.*]
-[.*/Application Data/Application Data.*]
-[(?i).:/Config\.Msi.*]
-[(?i).*\.rbf]
-[(?i).*/Microsoft.*/Windows/.*\.edb ]
-[(?i).*/Google/Chrome/User Data/Default/Cookies(-journal)?]
-[(?i).*/Safari/Library/Caches/.*]
-[(?i).*/(Application Data|AppData)/Apple Computer/Mobile Sync/.*]
-[.*\.tmp]
-[.*\.tmp/.*]
-[.*/hiberfil.sys]
-[.*/swapfile.sys]
-[.*/Local Settings/History/.*]
-[(?i).*\$RECYCLE\.BIN/.*]
-[(?i).*/Windows/.*]
-[.*/(Program Files|Program Files \(x86\))/.*]
-[.*/AppData/Local/Packages/.*]
-[.*/AppData/Local/Microsoft/Windows Store/.*]
-[.*/NetHood/.*]
-[.*/PrintHood/.*]
-[.*/Cookies/.*]
-[.*/Recent/.*]
-[.*/SendTo/.*]
-[.*/LocalService/.*]
-[.*/NetworkService/.*]
-[.*/AppData/LocalLow/.*]
-[.*/Internet Explorer/.*]
1 Like

Welcome to the forum, @Phlogi - and thanks for the great list!

I believe some of those are also included in the built in “checkbox” excludes, but there’s no harm in having them in twice.

Personally, I feel some of the expressions are broader than I’d use (I usually limit things to the Users folder) but it’s a really good starting point for people who want to have broad source folders.

I’m being lazy here, but is the first one for excluding some CrashPlan files?

Note that windows comes with it’s own recommended exclusion list and support for this has been added to duplicati recently:

And, yes, similar lists have been discussed before:

That’s because that list comes from your neighbours (i.e. Crashplan) :wink:

I was switching from CrashPlan to duplicati, that’s why I added this - just in case. It might be helpful to others too.

Also I guess the default exclusion from windows is a good idea, however I still got errors for files from user applications that were in use. Not sure how this should be treated. I remember that CrashPlan had some smartness to also backup those files once they became available and not complaining about them.

An example file:
C:\Users\user\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.dat

In Duplicati, the solution is:
Run Duplicati Server with administrative privileges (or run it as a Windows service with an administrative or Local System account) and set --snapshot-policy to on or required.
This will make Duplicati create a VSS snapshot using AlphaVSS, making it possible to backup files that are in use by another process.

1 Like

This list looks great, and I’m attempting to use it. But on a Windows 8 PC, running Duplicati Beta, it is showing “Counting files” for hours instead of under a minute without the filters. This is on an SSD, so disk access is about as fast as it can be.
Is anyone else seeing similar results with big filter lists?

I’ve got a filter list about half your length but with minute regular expression bits (lots of “or”) and noticed no difference between running as a service or the tray icon.

Is the speed the same for you with the test-filters command?

Can you post your list too? Also I noticed that counting files is really slow for me too now. So I have to review that. I guess using less regular expressions should help.

I’ll run the test-filters command as soon as I find time.

Here’s my stuff, but be warned I do a bunch of half-completed testing and haven’t cleaned it up - plus it’s a carry over from CrashPlan rules which I was trying to set up to handle both older (XP) and newer Windows layouts.

Odd - now that I look again, it’s many fewer lines than I recalled. :slight_smile:

-[.:\\(Users|Documents and Settings)\\.*\\([^\\]*Cache[^\\]*|[^\\]*Thumbnails[^\\]*|[^\\]*Temp[^\\]*|NoBackup|_NoBackup)\\]
-[.:\\(Users|Documents and Settings)\\.*\\[^\\]*\.cache]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\(Videos|Downloads|dwhelper)\\]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\LocalLow\\]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\Local\\(ConnectedDevicesPlatform|HP|CrashDumps|Temp|ClassicShell|Microsoft|Packages|ElevatedDiagnostics|Comms|TileDataLayer)\\]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\Local\\Google\\Chrome\\User Data\\Default\\(.*\\LOCK|data_reduction_proxy_leveldb|Extensions\\|Cache\\|Application Cache\\|GPUCache\\|Media cache\\|Current Tabs|Current Session|Lockfile)]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\Local\\Google\\Chrome\\User Data\\(PepperFlash\\|CertificateTransparency\\|lockfile|CrashpadMetrics.pma[^\\]*)]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\Local\\Mozilla\\Firefox\\Profiles\\[^\\]*\.default\\(cache2|safebrowsing|saved-telemetry-pings|datareporting|saved-telemetry-pings)\\]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\(Local|Roaming)\\Thunderbird\\(Mozilla Thunderbird|updates|Crash Reports.*)\\]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\(Local|Roaming)\\Duplicati\\(\.*\.sqlite*|control_dir.*\\)]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\Roaming\\Thunderbird\\Profiles\\[^\\]*\\.default.*\\.*\\[^\\]*\.(msf|lock)]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\NTUSER.DAT]
-[.:\\(Users|Documents and Settings)\\[^\\]*\\AppData\\Roaming\\Mozilla\\.*\\.*\.lock]

The speed was really horrible. I now enabled the exclude system files and temporary files flags and could reduce the list to simply exclude:

-*.lock
-[.*UsrClass.dat]

The initial list is an overkill, especially with the regular expressions. So I don’t recommend to use that extensive list anymore.

Agreed. As I warned, this was mostly inherited from CrashPlan which seemed to only have minor exclusion lists.

Hopefully soon I’ll have time to test the built in lists to see how well they compare to my list - and check for what’s up with the NTUSER issue.

Sorry to dig this one back up but another that I suggest adding to the list for Windows related backups is \System Volume Information\ or set the option for --exclude-files-attributes to Hidden, System, Temporary. I tend to want to backup the Read Only files anyways, but this should cover that list as well.

Yeah, I’d say that’s probably a safe one to add. :slight_smile:

Please be careful with this exclusion list and use it as a reference and not blindly copy/paste it.
I made that mistake. Not only was it excluding stuff I did want to backup, but there are some duplicate filters. Also some of those filters might simply do not apply to you(if you don’t have any apple software on your computer).
Another point is that I observed these exclude filters slow down backup time. After I pruned that list my backup time dropped from 55min to 26min on ~60k files and ~6-7GB of data.

Edit: Also looking at those regex’s, many of them will never match at all because Windows uses \ slashes, not /.

1 Like