i just came across duplicati and it’s really great! thanks for working on this!
i have a filter question though.
i have a directory structure like that:
dir1
subdir1
stuffidontneed1
stuffidontneed2
stuffidontneed3
ineedthis
subdir1
subdir2
subdir3
subdir2
subdir3
dir2
subdir1
subdir2
subdir3
stuffidontneed1
stuffidontneed2
stuffidontneed3
ineedthis
subdir1
subdir2
subdir3
dir3
subdir1
subdir2
ineedthis
subdir3
i would like to backup the ineedthis directories with all their contents. i got this working but to make things more complicated i have new requirements now. i would only like to match ineedthis if it is 2 levels deep and dir3 should be completely ignored. can someone help me with the regular expressions and filter rules?
i would be nice if the filter documentation contained some more examples.
my current try (without regular expressions for now) is this:
-*dir3\
+*ineedthis\
-\
it excludes dir3 correctly but it includes everything else. i would only like it to include the ineedthis directories. i thought the last line excludes everything else?
i am slow with regular expressons and need to experiment a lot but i think i could figure out the 2 level deep thing myself later.
what i have trouble with is understanding the exclude/include filter logic.
I’m confident I can make a regex filter that matches “ineedthis” at only certain folder depth, but it’s not working in my testing.
At first I thought I could just specify a source folder (the top of the tree) and then use the --include filter, but when I test it, ALL folders are included.
Duplicati is testing against my regex filter and also *\ which matches everything. Not sure why it does that:
PS C:\Program Files\Duplicati 2> .\Duplicati.CommandLine.exe test-filter 'C:\Test' --include='[.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]'
Including source path: C:\Test\
Including path due to filter: C:\Test\dir1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir3\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir3\subdir1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir3\subdir2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir3\subdir3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir3\subdir2\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\stuffidontneed1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\stuffidontneed2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\stuffidontneed3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\ineedthis\subdir1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\ineedthis\subdir2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir2\subdir3\ineedthis\subdir3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\stuffidontneed1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\stuffidontneed2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\stuffidontneed3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\ineedthis\subdir1\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\ineedthis\subdir2\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Including path due to filter: C:\Test\dir1\subdir1\ineedthis\subdir3\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]) || (*\)
Matched 0 files (0 bytes)
As a test I ran it as an --exclude filter, you can see that the regex properly matches “ineedthis” at only the correct directory depth. Of course this is the opposite of what you want:
PS C:\Program Files\Duplicati 2> .\Duplicati.CommandLine.exe test-filter 'C:\Test' --exclude='[.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\]'
Including source path: C:\Test\
Including path as no filters matched: C:\Test\dir1\
Including path as no filters matched: C:\Test\dir2\
Including path as no filters matched: C:\Test\dir3\
Including path as no filters matched: C:\Test\ineedthis\
Including path as no filters matched: C:\Test\dir3\ineedthis\
Including path as no filters matched: C:\Test\dir3\subdir1\
Including path as no filters matched: C:\Test\dir3\subdir2\
Including path as no filters matched: C:\Test\dir3\subdir3\
Excluding path due to filter: C:\Test\dir3\subdir2\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\])
Including path as no filters matched: C:\Test\dir2\subdir1\
Including path as no filters matched: C:\Test\dir2\subdir2\
Including path as no filters matched: C:\Test\dir2\subdir3\
Excluding path due to filter: C:\Test\dir2\subdir3\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\])
Including path as no filters matched: C:\Test\dir2\subdir3\stuffidontneed1\
Including path as no filters matched: C:\Test\dir2\subdir3\stuffidontneed2\
Including path as no filters matched: C:\Test\dir2\subdir3\stuffidontneed3\
Including path as no filters matched: C:\Test\dir1\subdir1\
Including path as no filters matched: C:\Test\dir1\subdir2\
Including path as no filters matched: C:\Test\dir1\subdir3\
Excluding path due to filter: C:\Test\dir1\subdir1\ineedthis\ => ([.:(\\[a-zA-Z0-9\ ]+){3}\\ineedthis\\])
Including path as no filters matched: C:\Test\dir1\subdir1\stuffidontneed1\
Including path as no filters matched: C:\Test\dir1\subdir1\stuffidontneed2\
Including path as no filters matched: C:\Test\dir1\subdir1\stuffidontneed3\
Matched 0 files (0 bytes)
the problem with the additional *\ include filter only seems to happen while testing using the command line. the final backup doesn’t seem to use this.
now i additionally would have to exclude dir3. but combining exclude and include filters is beyond me. i guess i will have to try to combine everything into one single regex.
Although the Filters web UI seems a bit glitchy here, its dropdown has an Exclude regular expression option that might help you. You can see what it generates by using its three-dot menu then Edit as text.
After saving, you can open job’s Commandline option to see if it set up your –exclude the way you wanted. Returning to the Filters web UI sometimes seems to guess wrong in terms of reversing it to the dropdown, however what really matters is that Commandline looks right – so presumably the backup will respect it…
yes, but if i just use one include filter, everything else will get excluded by default.
as soon as i add an exclude filter to the include filter, everything else will be included by default.
this makes it tricky. or am i missing something?
thanks for the command line option tip! that’s helpful for experimenting.
Filters documentation makes it sound like you control the processing order by specifying the --include and --exclude order (or where applicable, the equivalent plus/minus single-string syntax). I haven’t verified that, haven’t done much with filters, but will point to some of the claims that the Filters documentation has said:
When filter rules have been defined the first folder is taken and the filter rules are processed one by one. The first rule that matches is applied and the following rules are not processed anymore. For instance, if the first rule excludes a folder, then this folder and all files within will be excluded from the backup even if following rules include this folder or its files.
In the special case that all rules are include rules (which does not make sense when all files and folders are included per default) Duplicati assumes that all other files and folders are meant to be excluded (this had to be defined as another rule in Duplicati 1.3 but most people found that confusing so we changed that in Duplicati 2.0).
