Suggestion to the user manual regarding filters and wildcards

Site Feedback
1

This should be mentioned in the user manual.

Here are 2 points that should be mentioned in the user manual regarding selection filters and use of wildcards:

  1. If you use a filter to select files in a folder by use of wildcards it is not sufficient to create the filter including the total path. You also have to separately select the folder in which the wildcards are to be applied - either by using the file tree or by typing the path directly.

  2. If you use a filter to make a selection and you further want to select other folders or drives you must do this by creating a filter for each of these.

Here is an illustrative example:

I had a backup for my Home drive and a Data drive (both auto mounted in fstab). I selected these drives in the file tree. The backup worked fine.

Then I wanted also to backup my wine desktop files, so I added the path string/usr/share/applications/wine*.desktop.
No desktop files were backed up? After some digging through the manual and forums I came across a statement that wildcards could not be used in path strings.

Aha, then I made an including filter:
/usr/share/applications/wine*.desktop
Error stating that no folder was selected!!!? (hmm, I had defined the full path to the folder in the filter?) Then it crossed my mind that maybe there should also be a path definition, so I entered the path in the path field:
/usr/share/applications/

Now I got the desktop files backed up
BUT
now my Home and Data drives were suddenly no longer backed up. They only had 2 warnings stating that permission was denied to “lost+found”.

These warnings always used to be there, but the backup of the rest should not be aborted.

To make a rather long story short I finally defined a filter for both drives while still selecting their paths:
Path: /home/
Filter: /home/
and
Path: /Data_Linux/
Filter: /Data_linux/

Now the backup worked as intended.

Conclusion:
If you start using a filter to select files in a folder, you have to use filters on all source folders even though you want to include all files in the folders. Additionally you must select their paths either in the path statement box or in the file tree.

This is not obvious and should be mentioned in the user manuals filter section.

System info:
Duplicati version 2.2.0.1 stable
Fedora 43, GNOME
Lenovo Think Pad P1
16GB RAM
2 x 500GB SSD

2

I am very open to updating the docs, and we are happy to accept PRs as well:

However, I don’t think the descriptions you suggest clarify the issue (at least how I read it).

The way Duplicati works with filters is in two steps:

  • Enumerate source paths
  • Apply filters

The first step is needed as we want to be able to define what data is included in the backup. Any machine likely has multiple folders that it makes little sense to backup. Instead you need to supply the source paths which define all the known files in the “backup universe”.

Within the “backup universe” you can filter (remove) things that you don’t want. This is more flexible than having only sources, as you can choose to include/exclude things that do not yet exist.

When I read your “illustrative example”, I get the sense that you would like filters to work as sources or at least some kind of mixing.

What happens when you add the path /usr/share/applications/wine*.desktop it will match that path literal (including the *) and should fail because that path does not exist.

When you add the filter that will then be applied, but since the desired files are never added to the “backup universe” it does not have the desired effect.

We can discuss if this is a good way of working, but it does not currently, and I can see some issues with a source path like *.txt where we would need to enumerate every single accessible folder on the machine to honor that filter.

When you added the filter, you hit a “convenience convention”. If you only have filters that include things, then the assumption is that you want to exclude everything else (otherwise the filter has no effect as all paths are included by default).

Include rules are a bit advanced as it relies on filter order, but can be used to create “exclude all, except these” rules without regex contortions.

With this information, and reviewing the filters page, I think we can update it with:

  • Information about how source and filters work together
  • Mention the include-only filter convention
  • Mention that include rules should only be used to create “exclude, except” patterns

If you have text that would have worked for you, I am open to suggestions as I think we get better documentation if it is understandable by someone who is not deep into the inner workings.

3

Hi kenkendk

Thankyou for your answer.

Point 1.
Regarding use of filters I think what is needed to clarify what is not obvious now is by adding something like this:

"
Note: When you define a filter you additionally have to define the path to the folder you refer to in the filter - either by typing at path statement or by selecting the folder in the file tree.

Example:
Filter: Including expression: /usr/share/applications/wine*.desktop
Path: /usr/share/applications/ (remember end slash).
"

Your comment on the filter function being a 2 step process might also be useful for the understanding.

I think this will help regarding my point 1.

Point 2.
However my point 2 is neither logical or understandable. It is very surprising that a functioning backup not using filters, suddenly stops working by adding a(n including) filter. Whether this is intentional or a bug, it should be explained very clearly that it is so. It is very hard to figure out what is wrong when this happens.

An additional explanation could be something like:
"
If a filter is used to select files in a backup set, then all selections in that backup must be done with a filter, even though you are adding entire folders without filtering.

Example:
You want to backup all wine .desktop files in /usr/share/applications/ and
you want to backup your entire home folder.

For this you need to define the following:
Path: /usr/share/applications/
Including filter: /usr/share/applications/wine*.desktop
Path: /home/
Including filter: /home/*
"

The reason behind my understanding of the manual:
When I wrote the path /usr/share/applications/wine*.desktop as you refer to, I was convinced this was a valid statement for 2 reasons:
First:
I have been using backup systems of different kinds since the very DOS days, and all systems, I have come across, had a folder selection proces in a file tree and some kind of more or less advanced inlcude/exclude filtering accepting wildcards. So “mixing” selection and filtering was obvious to me.
Second:
When reading the importance of distinguishing between paths and files described in the manual section “Path representations” I was convinced that typing a path statement withou an end slash would tell the system to interpret the “wine*.desktop” as a file name. With reference to this section in the manual I don’t understand your explanation that the program would interpret the full expression as a folder when there is no end slash.

Finally.
I have no preference whether the selection/filtering process should be done one way or the other. The important thing to me is, that the way it works is explained clearly.