Exclude filters not working


#1

I have a duplicati backup job configured with the following filters:

-._*
-.DS_Store
-*.out
-*.o
-*/iPod Photo Cache/

But when I run test-filters command from the Command-line option in the Web-UI, only the last filter seems to work.

Below is a snippet from the output I see.

Excluding path due to filter: /Users/mohak/Pictures/iPhone Sync/iPod Photo Cache/ => (*/IPOD PHOTO CACHE/)
Including file: /Users/mohak/Pictures/iPhone Sync/.DS_Store (20.00 KB)
Including file: /Users/mohak/Pictures/iPhone Sync/a.out (unknown)

The first directory is excluded as expected, but the next 2 files are not.


Exclude option not working
#2

What type of filter fields are those in?


#3

I’m not 100% sure about it, but you could replace -.DS_Store with -*.DS_Store or -*\.DS_Store.
According to the documentation about filters, it’s best practice to supply folder exclusions first. Maybe it helps changing the order of the filter list.


#4

Also, could you post a screenshot of how you supply the filters?

There is a bit of difference in that the format you use works under the “edit filters”, but on the “commandline” page you need to set each with --exclude-filter=.


#5

The filters are supplied via the Web-UI under Edit->Source Data->Filters


#6

Neither of those alternatives seem to work. Strangely though, when I moved -*/iPod Photo Cache/ to the top of the list, even that rule stopped working. I moved it to the bottom again and the rule started working again. :thinking:


#7

mohak, did you use the browse tool to browse to the “/iPod Photo Cache” and check it (giving it a red X) or did you manually type that in?

Also, if you browse through the Source Data folder tree (usually shown above the Filters list you provided) do you see the “offending” iPod Photo Cache file listed with a red X next to it such as seen here next to Downloads?
image


#8
  1. No, I manually typed that in.
  2. No, I see a green checkmark. Is the Source Data folder tree supposed to show a red X next to files excluded via regex filters?

#9

mohak, OK - then I think the issue is that your formatting of the Exclude XXXX value is incorrect.

May I suggest using the folder tree to browse down to and “red x” the folder or file you don’t want backed up and then go down to the Filters section and see what filter was added? That should give you an idea of what formatting the “Exclude folder” or “Exclude file” fields are expecting.

I used that method for some of my initial exclusions as well as the “Edit as text” option (the 3 vertical dots) to manually put Exclude Expressions in and then use the tree view above to verify what I my rule was supposed to exclude was really excluded.


#10

O.K., this is quite strange. First, I removed all existing filters. Then manually excluded a file and a folder from the source tree. This added a “red X” next to them in the source tree and added the following rules in the filters section.

-%HOME%/Pictures/iPhone Sync/iPod Photo Cache/
-%HOME%/Pictures/iPhone Sync/.DS_Store

I then re-ran test-filters command and made the following observations:

  1. I didn’t see any line that said “Excluding” or “Including” either of the above.
  2. But, I did see the sub folders of iPod Photo Cache being included with lines like this: Including folder: /Users/mohak/Pictures/iPhone Sync/iPod Photo Cache/F00/

#11

mohak, yes - the test-filters command including items that the GUI says should be excluded is similar to what I commented on in another topic

However in reviewing my actual backups I’m not finding any of the items that I excluded - so it looks like the filters during backups are working as expected and that it’s likely just a bug with the test-filters command that we’re seeing. Note, however, that this has not yet been confirmed.

If you have the time / ability I’d suggest doing a backup then browse through the files available to be restored specifically looking for items you know you excluded. I found the “Search” button very handy for that purpose. :slight_smile:


Some filters don't work properly
Help with filters
"Test Search Selection" tool as in Duplicati v1
#12

I think I found the reason for the strange behaviour in the previous post and I think it’s because of the %HOME% variable in the source data selection. When I created the backup job, I selected the source directories from User Data->Home->....

Apparently, the filters did not work when I did that. So I removed those directories and instead selected the same folders from Computer->Users->mohak->....
Now when I use either of the following filters individually, the I see the appropriate files being exclude.

-*/.DS_Store
-*/iPod Photo Cache/

Unfortunately, when I use both together, I still get the original behaviour, i.e., whichever one I put at the bottom, is the only one that seems to work. I am now thinking this could be a fault in the way the Web-UI is calling the command line. I shall look into it further.


#13

mohak, good catch there! If you’re thinking the command line might be the issue you can review the actual command line via:

  1. “expand” your backup set
  2. select “Export…”
  3. select “As Command-line” and click Export

The next page will give you the text of the actual runnable command line as specified by the GUI.


#14

I get the impression that excluding files which have ONLY an extension is a bit wonky. For example, I couldn’t get “.plexignore” to be filtered out whether I tried filtering the extension “plexignore” or the whole filename “.plexignore” - perhaps a bug in the parser?


#15

So the Web-UI command line is indeed the culprit here. I ran test-filters in bash as per @JonMikelV’s instructions and all the appropriate files were indeed excluded! Thanks guys!


#16

I suggest you check by running test-filters from the actual command-line using the following steps:

  1. Export the commandline like Jon explained and copy it to a text editor.
  2. Replace backup after Duplicati.CommandLine.exe with test-filters
  3. Remove the argument immediately after test-filters which should be your backup destination. Also remove your passphrase to make sure it is not saved in your terminal’s history.
  4. Paste this text in a terminal window and analyse the output.

#17

In the commandline Web-UI all options must be on a separate line.

The error that @mohak reported, could be related to this:


#18

The options look fine to me with and without the Edit as text mode on. Here are the screenshots for your reference:

  1. Normal mode

  1. Edit as text