Suggestion to the user manual regarding filters and wildcards

Site Feedback
1

This should be mentioned in the user manual.

Here are 2 points that should be mentioned in the user manual regarding selection filters and use of wildcards:

  1. If you use a filter to select files in a folder by use of wildcards it is not sufficient to create the filter including the total path. You also have to separately select the folder in which the wildcards are to be applied - either by using the file tree or by typing the path directly.

  2. If you use a filter to make a selection and you further want to select other folders or drives you must do this by creating a filter for each of these.

Here is an illustrative example:

I had a backup for my Home drive and a Data drive (both auto mounted in fstab). I selected these drives in the file tree. The backup worked fine.

Then I wanted also to backup my wine desktop files, so I added the path string/usr/share/applications/wine*.desktop.
No desktop files were backed up? After some digging through the manual and forums I came across a statement that wildcards could not be used in path strings.

Aha, then I made an including filter:
/usr/share/applications/wine*.desktop
Error stating that no folder was selected!!!? (hmm, I had defined the full path to the folder in the filter?) Then it crossed my mind that maybe there should also be a path definition, so I entered the path in the path field:
/usr/share/applications/

Now I got the desktop files backed up
BUT
now my Home and Data drives were suddenly no longer backed up. They only had 2 warnings stating that permission was denied to “lost+found”.

These warnings always used to be there, but the backup of the rest should not be aborted.

To make a rather long story short I finally defined a filter for both drives while still selecting their paths:
Path: /home/
Filter: /home/
and
Path: /Data_Linux/
Filter: /Data_linux/

Now the backup worked as intended.

Conclusion:
If you start using a filter to select files in a folder, you have to use filters on all source folders even though you want to include all files in the folders. Additionally you must select their paths either in the path statement box or in the file tree.

This is not obvious and should be mentioned in the user manuals filter section.

System info:
Duplicati version 2.2.0.1 stable
Fedora 43, GNOME
Lenovo Think Pad P1
16GB RAM
2 x 500GB SSD

Using filters and wildcards in Duplicati
2

I am very open to updating the docs, and we are happy to accept PRs as well:

However, I don’t think the descriptions you suggest clarify the issue (at least how I read it).

The way Duplicati works with filters is in two steps:

  • Enumerate source paths
  • Apply filters

The first step is needed as we want to be able to define what data is included in the backup. Any machine likely has multiple folders that it makes little sense to backup. Instead you need to supply the source paths which define all the known files in the “backup universe”.

Within the “backup universe” you can filter (remove) things that you don’t want. This is more flexible than having only sources, as you can choose to include/exclude things that do not yet exist.

When I read your “illustrative example”, I get the sense that you would like filters to work as sources or at least some kind of mixing.

What happens when you add the path /usr/share/applications/wine*.desktop it will match that path literal (including the *) and should fail because that path does not exist.

When you add the filter that will then be applied, but since the desired files are never added to the “backup universe” it does not have the desired effect.

We can discuss if this is a good way of working, but it does not currently, and I can see some issues with a source path like *.txt where we would need to enumerate every single accessible folder on the machine to honor that filter.

When you added the filter, you hit a “convenience convention”. If you only have filters that include things, then the assumption is that you want to exclude everything else (otherwise the filter has no effect as all paths are included by default).

Include rules are a bit advanced as it relies on filter order, but can be used to create “exclude all, except these” rules without regex contortions.

With this information, and reviewing the filters page, I think we can update it with:

  • Information about how source and filters work together
  • Mention the include-only filter convention
  • Mention that include rules should only be used to create “exclude, except” patterns

If you have text that would have worked for you, I am open to suggestions as I think we get better documentation if it is understandable by someone who is not deep into the inner workings.