What if a file is contained in more than one of my backup config.?

I made two backup configurations. Not sure that they include no common files. How can I find out easily? What if in fact there is a file common to both?

This question I ask for two reasons:

  1. Such a case makes a priori little sense (at least: waste of execution time and storage space)
  2. More important: I guess that such a situation might make problems, may-be at executing one of the backups and even more in case of a restore. Is this guess justified?

To find out, an unpractical way is to choose “modify” (first one config, then the other) until one is on the page about file selection where one can go into details but risks to oversee that a green mark on a folder might not mean that all contents of it has been selected - theoretically it is a “tree” of subfolders to explore. Forgetting the special purpose (look only, no change) one could be tempted to change something in this “opportunity” BTW isn’t changing the selection nearly always risky? Which changes are for sure not creating potential problems? (End of btw)

Moreover the selection shown might be ambiguous: if all subitems of a folder (i.e. files and subfolders contained in the folder) are marked, that might mean: everything in the folder is selected including future content (which will be backed up too after its creation by the same config.) OR only the current content of the folder is selected (and future content will be ignored at execution of the backup config. until this is completed to include the new items). I wonder whether these two possibilities exist. In case only the second one exists, no ambiguity, but this I would describe by saying Duplicati is strictly file-oriented, there is no folder-orientation.and this would be unpractical: imagine for ex.: I open a new period (year) in my accounting SW, which creates a new file and I forget to add it to the backup config. that backs up the older years - and this file will not be protected against losses by backup!

I hoped that using Duplicati from the Command Line might help here - both for my basic question in the title and the mentioned ambiguity - but what is written about that usage in the manual doesn’t seem to confirm this hope, and it doesn’t work any way: the command Duplicati.CommandLine.exeissues a message that this thing is not found. BTW finding a place to enter such command lines took me much time on my Windows 11 which hides it while it was so simple before (for single commands) so accessing to what they call the Windows shell will again take much time if I try again.

Duplicati treats each backup completely isolated. There is no interaction or communication between backup configurations.

You can make a backup of the same file(s) with many configurations if you like to store the backups at different locations.

For that reason, there are no tools that will compare two different backup configurations.

One way to achieve this, would be to use the FIND command on the two different backups, and then finding common strings in the two:

Duplicati.CommandLine.exe find * file://unused --dbpath=path-to-config1.sqlite > text1.txt
Duplicati.CommandLine.exe find * file://unused --dbpath=path-to-config2.sqlite > text2.txt
Findstr /i /x /g:text1.txt text2.txt

Note that the file://unused needs to be there for validation reasons, but only the database path is used. The findstr command finds common strings.

If you have problems getting a terminal, press Win+R, type “cmd” and press enter.
In the terminal, type:

cd "C:\Program Files\Duplicati 2"
Duplicati.CommandLine.exe help find

There is nothing in Duplicati that has problems with this. Only problem I could see would be if you restore two backups into a folder, and they both restore the same file. You can use the option to not overwrite existing files in this case, and you will get a file with a timestamp in the name (providing that the files are actually different versions).

Not sure I fully understand this, but Duplicati will by default include everything from below the selected top folder(s). If you add/remove/move/modify folders or files inside one of the top folders, they will be included.

The only way to prevent this is to use filters, which can either be static (a single name) or dynamic (wildcards or regular expressions).

What’s a common file? If you mean due to common config, just compare configs.

Creating a new backup job in old manual is a bit long, and suggests:

You can review your selections under Source data in the file picker.

which probably refers to third section of tree, which avoids having to open others:

If you prefer text, the three button menu has a Show advanced editor for that use.

Generally, you won’t have common files without common configs that leads to them.

Thanks to both who answered. They help a lot. Especially useful is the thing with the use of keys Windows+R which makes visible things now hidden in Win 11 - in Win 10 one would find that easily, but because I had not frequently used the Windows key,I did not have the idea to try it with Win 11. Still new to this, I need also general help in this OS. (For such things I found a forum in the StackExchange group of sites but it was difficult to get answers there.)

Looking at the Source Data section of the file selection page for finding what is in a backup config. → I should have had that idea myself ! I consider that it is risky to define the selection there (if it is at all possible) but it is the better place to just look what is contained. BTW my two backups I have defined now DO have common files - by which I mean they occur in both configs. This was not intended but you say it’s not a real problem, so I leave it like that.

The three dot function showing the extended editor I was a bit afraid to apply. Now I saw rhe result. I dont think I’d use it for modifications without a very good reason because the syntax is a bit special, so again high risk of making errors.

For what I consider an ambiguity, may-be I’ll come back later with an example if I feel I really need to know. For the moment: here is behind that question:

In a backup SW I used before on older PCs but which is not well suited for cloud storage. there was a clear possibility to select the whole content of a folder, which must be understood dynamically. After all, not only individual files change their content. But folders can have a modified list of their content list: files included and subfolders. Often one will want to backup all this. Example: Take a folder with accounting data, for each period (i.e. year in my case) a file is created - at least that is how it works with my SW. If I’ll open 2026 in about 11 months (from now January 24) and forget to complete my backup selection, the 2006-file will not be saved until I add it in the backup config. concerned. That is how I expect that it could be in Duplicati. The simplest way, static. If it is not like that, the question is how can one see (in an existing config.) or decide (in a newly being created config.) which case (static or dynamic) one has / wants? Combined cases might make sense too → all this may need to be defined using exclusions and filters - for these I don’t have good examples

I think this was already answered as not how it works. Is there any confusion about the below?

Furthermore the “Source data” contents in the GUI tree (or looking at the text version of that) is what determines the configuration (along with what you put in Filters and Exclude – be careful).

“User data” and “Computer” (preferably not intermixed) are to help you set up the configuration, however something that you see in an expanded tree, unchecked, is not part of a configuration.

If you click a folder to select it, you can see it added to “Source data”, but that whole tree is not, because at backup you get whatever is in the tree at the time. I guess you’d call that “dynamic”.

From a GUI viewpoint, though, the contents of the tree get checked, I guess partly to show what would happen (but not to imply that it’s static), but also to let you apply automatic Filter by check again, which turns it from a green check mark to a red X. Below, note the red mark, and its Filter:

The “Source data” has FOLDER_A, and is backed up (and filtered) with content at backup time.

Source data from new manual:

In the source picker view you can choose the files and folders you would like to back up. If you pick a folder, all subfolders and files in that folder will be included. You can use the UI to uncheck some items that you want to exclude, and they will show up with a red X.

and what it doesn’t detail is it does the exclude by adding an entry in Filters that does it for you.

1 Like

There is or was some confusion. But I decided to get a clear answer by TESTing: this is allways better than asking when there is no risk and one isn’t looking for hidden things. First I thought to make a test with execution of a backup config. but I eventually realized that just going into the selection and looking what happens when one does this or that can already help a lot . so that’s all I tested for the moment.

The hint concerning the possible generation of filters out of what ones does in the tree was quite decisive. I looked if I had such things (before I had not the feeling that I need to enter something there in my case for the moment, and not knowing about such an automatism, I had no reason to open that paragraph) and yes something was there … in the folder with my accounting data - yes that one again - and this had to do with the fact that my accounting SW creates files with the extension BAK containing the old state of a normal acc. file after each change (to have a solution in case the changed file is somehow corrupted) and I find it does not make much sense to backup these BAK-files with Duplicati. To these exclusion filters corresponded in some circumstances red marks in the tree - I dont think I made them, but rather put green marks only on the normal files. It would go a bit far to tell here all that happened while I manipulated these things. But my final conclusion out of these tests is:

The Duplicati SW seems to contain optimizations far beyond what one might expect a priori. How they work exactly might be different from my ideas I mention in what follows but it shows the probable direction all this goes.

When the part of the content of a folder which should be backed up is nearly everthing, it is more optimal to declare: the whole content minus a few excptions; when what is wanted to backup is only a small part of the content i.e. just a few items, then it is more optimal to declare only these few items (positively i.e. green) and in both cases this is probably what is really wanted. Of course this is vague and there may be some subtle aspects and the user himself might act differently in the two cases, so he might be the true source of this surprising optimization.

Any way this - although it does not concern the case in which I saw an ambiguity - paradoxically lets me hope that Duplicati will understand my choices “dynamically” in the ambiguous situation. This has yet to be confirmed - may-be by a test WITH execution … The said ambiguity exists also when part of the content is selected and part not, in case a new item is created and put into the folder e.g. by the function NEW (or SAVE AS …) in a text treatment SW… What matters is finally the choice between all except and only these.and this one can see in the source data part of the selection tree combined with filters that may have been generated.

Thanks for the help.

Please test to confirm for yourself what has been explained. Further notes on configuration:

I suggested looking at Source data in tree view to refute an idea static enumeration exists.

There is also a Commandline button in job. Source data goes in Commandline arguments.

Export

The option to export “As commandline…” is not covered here, but allows you to get a string that can be used with the Duplicati CLI executable.

This is equivalent to the GUI job, meant for those who for some reason run CLI independently.
You will clearly see there that there is no full file enumeration. It should look like Source data.

All of these should show that there is no enumeration of entire config kept in job configuration.
That would be a very large config to use. What if you made a large backup with a million files?

Folders are walked at backup time. That’s also why the status bar tells you it’s “Counting files”.

From backup perspective, the source data selection that will be used is found in Source data (potentially modified by Filters and Exclude) but the automatically applied checkmarks below a selected folder are not part of the job configuration, and in fact you cannot deselect – only filter.

In the GUI, look for green checks on folders. There’s actually a half-green mark for partial trees.

In the first example, I checked walk. The content below are checked as they’ll be backed up if folder stays the same. Because walk is what I chose, it gets walked at backup time, so if sub1 vanishes and sub3 shows up, that’s fine. It backs up what I checked personally, which is walk.

In the second example, I did not check top-level folder walk2. I opened it and checked A and B, meaning those get walked at backup time. I never personally checkmarked walk2. Note the GUI doesn’t put a check mark there either. That’s how you can see that it won’t be walked at backup.

image

Commandline:

image

Export As Command-line:

"C:\backup source\walk\\" "C:\backup source\walk2\A\\" "C:\backup source\walk2\B\\"

Although you might not be doing these by hand, the slash at the end means a folder, and the doubled slash before a closing double quote is something that Windows commandline needs.

TL;DR

Ignoring things you might try with Filters and Exclude, if you want a folder backed up with what’s there at backup time, click it, see it marked with a check mark, and see it put into Source data. This means if you have a folder whose growing content you want, click the folder not its content.

Thanks. One more detail Excluding only the current *.BAK files (they have already changed due to my actions in the accnt. SW) isnt really what I want. Moreover this SW sometimes creates other special files - probably among other possibilities at a forced immediate shutdown, to save data in work - and these I also dont want to backup. So I decided to replace the automatically created exclusion filters by a selfmade include filter naming the only files in the concerned folder to be backed up - and of course taking this folder completely out of the tree (to avoid unneeded double backups). The filter looks like this
C:\Banana-Accnt\UKe*.ac2
where ac2 is the extension of the “normal” files in this folder i.e. those I want to backup with Duplicati. Is this syntax doing what I want? (If I receive a NO answer to this in time - before next execution (see below) - I will be able to correct my input. Otherwise I’ll see what happens …)

I also tried the export function, but it seems to me it would may-be not necessarily produce error messages for wrong input - especially of course when what I enter is formally OK but does not have the wanted effect …
Looking at the content of the exported file, I see that what corresponds to my include filter has many more double back slashes than what is shown to me in the last contribution to this thread.
The next execution of the concerned B. config. - planned for today 6 p.m. in CH time - will any way serve as a test and show whether all is OK as needed.

I suggested Export As Command-line to keep you from reading a file not meant for humans.

Introducing JSON explains how backslash is character escape, including escaping backslash.

I would suggest going lightly on Filters (if space allows), as wrong filters can make big problems. Filters in Duplicati talks about uses, but doesn’t cover exactly what the many GUI dropdowns do.
Three-dot menu to the right of Filters will show what the GUI built. You can match that to the doc. Additionally, advanced Filters will not always color the GUI tree right, as JavaScript regex differs.

You don’t say where that is, as there are lots of choices on the dropdown, and also three-dot raw.
If you put it in “Include expression”, it makes +C:\Banana-Accnt\UKe*.ac2 in Edit as textand looks more or less good to me. Note that a * here can proceed across backslash path separators. Regardless, I also don’t know your file naming conventions, so you must also assess for yourself.

As for whether it’s what you want, note that this limits your entire backup. You want nothing else?

The TEST-FILTERS command is a way to test filters without backups, but either way, please test.

Even better perhaps, limit use of Filters (and Exclude even more so), especially when starting out. Saving a tiny amount of space is not worth the risk of discovering a bad backup when you need it.

The test has happened and it is NOT OK. (I was occupied elsewhere nearly until time of planned backup, so I read the last answer to my Q. here only after the test.) First about the resuts:

The (new) backup of 18:00 local time today is supposed to have been successful. And a new file has been created in the folder of my cloud space (target of my Duplicati backups) but I guess this might just be a kind of header which is now being used by the GUI to produce the things it shows me about the results. In case the backup job had produced an error message, where could I see it? I am not expecting one because bad symptoms don’t necessarily mean there was a formal error but just valid input that has somehow correctly produced an “empty” result.

Following symptoms seem to show that no new backup virtual files have been produced: 1) for the next exec. of the same config. the “forecast” is 0 Bytes 2) If I choose Restore as if I wanted to make one (but I don’t) I see that there are no files to choose 3) may-be the fact that except the probable header no new file has been put into the backup folder in my cloud space allows the same conclusion (no new virtual backup files) I talk about virtual files because what matters to me isn’t the real files there (so-called blocks or something like that) but the things hidden in them (even more due to encryption) that will become a restored file if I do a restore, these things I call virtual files.

Now why is that? I am nearly absolutely sure that if my interpretation of the GUI - espec. concerning Filters - were correct, some non void backup should have occurred. Namely the accounting file for the current period contains records which I must have entered after the preceding backup of the same config. was executed.(some of these records document information that I did not have at that older time) but was already there when the last backup job (today) was run. So a real change of the file occurred from the P.O.V. of Duplicati - this tells me that the file changed was not selected by the last job.

So let’s look at my input. It is (probably) not under “include expression” because the German item in the dropdown menu which must have that function translates to
Regular expression (include)
which I did not select because I don’t know yet what is meant by regular here.
Instead I used the menu item which translates to
Filter (include).
The file which I pretend has been changed is what one would name for my Windows 11
C:\Banana-Accnt\UKe\2025.ac2
so it should match the selection defined by my filter with the choice I made in the dropdown menu; or do you see this otherwise? . I did not put the plus sign you mention, but I consider it would be weird to have to put that, after all it is something to include. BUT the job did not select the file … how come? Now I have an idea! May-be my “expression” is not allowed under the chosen menu item (only individual files and folders?) To do a new test that might be better, I displaced it from where I put it to Regular expression (include) (my translation from DE as I said). I cannot test this without execution - except may-be with your TEST FILTERS thing where again I have no experience (lots of misunderstanding possible).so it’s better if you just confirm my idea and that the new version should be OK. In that case I might start an execution by hand. Otherwise the next planned exec. is on Wed 29.1.25 again 18:00 local time.

Now what do you mean by
this limits your entire backup. You want nothing else?
Must I understand that the entries into Filters with Include deactivate the whole selection in the tree although the part Source data isn’t empty? I’d find this stupid, and it eliminates good possibilities for the selection of files to be backed up. In that case, is it at least possible to change this default into what I want (both tree & Filters with Include)? If not, I would have to split my main backup into two …

When saying things like that, please either name it or say if name has dlist, dblock, dindex.

What sort is it? At least say the Storage type you put on Destination, but more info is better.

If however, this backup was working before (was it?), then the new error may be elsewhere.

If you got a red error message popup at bottom of screen, click the Show button for a job log:

image

If no job log was produced, you should check in the server log at About → Show log → Stored.

Oh. That’s an error. It’s not a regular expression. I’d stay away from those until you understand.

If I do that, I get this:

and no job log, but server log for that run time explains error this way:

Jan 26, 2025 5:26 PM: Failed while executing Backup "test 1" (id: 2)
System.Text.RegularExpressions.RegexParseException: Invalid pattern 'C:\Banana-Accnt\UKe*.ac2' at offset 17. Unrecognized escape sequence \U.
   at System.Text.RegularExpressions.RegexParser.ScanCharEscape()
   at System.Text.RegularExpressions.RegexParser.ScanBasicBackslash(Boolean scanOnly)
   at System.Text.RegularExpressions.RegexParser.ScanBackslash(Boolean scanOnly)
   at System.Text.RegularExpressions.RegexParser.CountCaptures(RegexOptions& optionsFoundInPattern)
   at System.Text.RegularExpressions.RegexParser.Parse(String pattern, RegexOptions options, CultureInfo culture)
   at System.Text.RegularExpressions.Regex..ctor(String pattern, RegexOptions options, TimeSpan matchTimeout, CultureInfo culture)
   at System.Text.RegularExpressions.Regex..ctor(String pattern, RegexOptions options)
   at Duplicati.Library.Utility.FilterExpression.FilterEntry..ctor(String filter)
   at Duplicati.Library.Utility.FilterExpression.<>c.<.ctor>b__11_0(String n)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Duplicati.Library.Utility.FilterExpression.Compact(IEnumerable`1 items)
   at Duplicati.Library.Utility.FilterExpression..ctor(IEnumerable`1 filter, Boolean result)
   at Duplicati.Library.Utility.FilterExpression..ctor(String filter, Boolean result)
   at Duplicati.Server.Runner.<>c.<ApplyFilter>b__20_2(<>f__AnonymousType2`2 <>h__TransparentIdentifier0)
   at System.Linq.Enumerable.SelectIPartitionIterator`2.MoveNext()
   at System.Linq.Enumerable.Aggregate[TSource](IEnumerable`1 source, Func`3 func)
   at Duplicati.Server.Runner.ApplyFilter(IBackup backup, IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

Question regarding file filter discusses the German translation. For your syntax, you use this:

Filter (einschlieĂźen)

Later discussion talks about the special unusual case where only Include rules are specified.

Since you’re saying you did use Filter (include) (translation back to English), it “should” work.

Note that Filters filter Source data, so you also have to make sure you have right folder there.

You don’t type it in the dropdown field, but the dropdown puts it into Edit as text if you look.

That’s what plus sign is intended to suggest. A minus sign is an exclude. Sounds fine to me.

Filters apply to the Source data. What else do you think Filters and Exclude should apply to?

Why don’t you try the advice I wrote to just do something like backup C:\Banana-Accnt\UKe?
You can get yourself too confused with Filters until you understand them, and it’s dangerous.

I don’t understand what “part Source data” means. As noted, filter works with all Source data.

Not clear what Source data you want. If you just want C:\Banana-Accnt\UKe, then you’re fine.
If you want to get stuff from all over, then the include-without-exclude implies exclude the rest.

You can look through the other post I cited, but the information from the author on that is here.
You can put together combinations of include and exclude for most uses, but it’s not simple…

EDIT:

So to test what you’re trying to do that isn’t working (please try to find the error message), I set:

C:\BANANA-ACCNT
└───Uke
        2025.ac2
        2025.BAK

image

Restore looks like:

because by asking for only include of C:\Banana-Accnt\UKe*.ac2, I exclude walk and .BAK.

It seems to me that the problem is that I de-selected the whole C:\Banana-Accnt\UKe\ folder in the tree, thinking that otherwise the files contained there would all be selected by the backup job plus once more those contained in the inclusion filter (which are the ones I want), I guess now this was a misunderstanding. I’ll assume now that despite it’s not very logical from my point of view, any folder touched by a filter (or several of them, some may-be making an exclusion) is first emptied (of selected files) before the filter(s) apply. I imagine that this isn’t the case for folders NOT touched by a filter, i.e. for them, what is in the tree is what counts.

Based on this, I will put back my include filter to where it was before (to Filter(include) ) AND select again the C:\Banana-Accnt\UKe\ folder in the tree (complete)

The reason why I have chosen an include filter is that excluding the unwanted ".ac2.bak files (I consider that ac2 is not part of their extension because it it is followed by a dot plus bak) would keep selected all other files in the folder (the accounting SW making sometimes other unwanted files). Of course, considering the space available, this isn’t really important, but I am learning now and will apply later what I think about the way Duplicati works to cases where it does matter. For the moment, no really large files - executables or videos - are on my plan, and a lot of other folders I still don’t intend to backup. But I might change my mind on that and then it might be critical

That’s an error. If you want anything from there, you have to have it in Data sources.

A filter, whether include or exclude, filters Data sources. It doesn’t add anything new.

It’s also not true. Data sources produce a set of paths. Filters say which to keep or not.
Maybe I misunderstand “touched by a filter”. Filter touches everything in Data sources.

That should get you the files you want and omit what you don’t, but cut off any other files.
I don’t know if you have anything else you want in Data sources. If so, see my example.

I made a test. This time the result was not “empty” but still restricted to the folder mentioned in the include filter. Here is a picture (made via screenshot in the GUI and Windows’ Paint) showing Source Data and my filter:

Now the following picture shows the result obtained by executing the backup config’ containing these data.- by showing what a restore job would do after the backup just shown:

Look at these pictures, you might better understand what I say below …

I can understand that unchanged files do appear: they might have changed in a way I don’t see - by the specialized SW I use to produce them. Or they are there in case one wants to also restore from earlier backups made with the “same” config. (but this I changed several times, so I mean same name in same GUI folder) … as those files aren’t generally proposed, why here? - this although the theory says that Restore can search for older backups if nothing is found in the last … after all, these backups are supposed to be incremental.

More important is: why nothing else (among contents of other folders) is available? I guess because nothimg else was backed up before - which BTW is true only for the last two executions of the “same” config. (again a contradiction with the theory just mentioned!) At first sight this seems to prove that the filter eliminates everything in folders other than the one mentioned in the filter. OK the filter restricts files to those with extension ac2, and for sure no other folder contains files with this extension (long ago there were other subfolders other than UKe which stands for my person, at least this was planned if never realized, and those had or would have had files with the same extenson) BUT the filter explicitly mentions only one folder … so an unique include filter suffices to first deactivate the whole source data? I’ll assume this and therefore abandon all filters except those one doesnt input directly but are generated by excludes in the Source data.

Some symptoms mentioned here seem to tell me that ALL chaanges in the selection are to avoid because there might be an irreversibility effect !! My next change will eliminate all selfmade filters and replace them partially by excludes in the Source data. Not what I really want. But acceptable because I will probably never have such data for more than about 20 MB. If next test isnt OK I will have to begin again with the beginning: make a new config. and delete the old one.

The UI is not 100% accurate in this area, but if you edit and view the UKe folder, it should show you real-time what files are included and excluded. Does the UI show the .ac2 files with a red cross?

Can you click the three dots, and copy-paste (or screenshot) the filters:

The term “regular expression” here refers to regular expressions. They are very expressive but have a bit of a learning curve.

The term “filter” here refers to a globbing expression.