It seems like in the Source data view I can select the same directories from more than one location. Which one is the better?
For example in the screenshots below I was trying to ignore c:\Users\<my user>\AppData\Roaming\discord\ directory.
I turned it off in User data > Application Data > discord
But that dir is still enabled in User data > Home > AppData > Roaming > discord
Which one will take priority? Where should I turn it off?
Similar to Windows File Explorer, I suppose. Several ways to get to the same place.
How did you set up a priority question? Normally you’d only want to “check” one way.
For me, the different ways of filling the green checkmark trees worked independently.
Looking at an Export As Command-line should show what you actually got, in theory.
exclude should apply to all of the source paths specified for The BACKUP command.
This means that it should hold even if you GUI-green-check the same path two ways,
however it still seems a safer plan to just pick one way of selecting, and just use that.
If you’re familiar with Windows folder environment variables, Duplicati uses them too:
Environment.SpecialFolder Enum (.NET Framework API)
The check boxes in the file picker are only an approximation of what will be included.
Only the highest green check in the tree is added as a source path (in your case Application Data, Home, …). You can also look at the source list directly with the three dots above the file picker. Everything inside a source directory is processed with the specified filters to see if it should be included. If a directory is excluded, all of the contents are also skipped (this is marked with a red x).
In the backup job, environment variables are used to substitute the full path to the user directory. The filters only run on the full paths, so either method will exclude the folder. The UI uses a different method, so it does not show up correctly.
I could be mistaken, but I think Windows File Explorer does it for some legacy compatibility reason with other apps. Duplicati doesn’t really need to do that.
I didn’t set up any priority. I don’t see where that’s possible to do.
I didn’t manually check both ways. This is how Duplicati added my profile home dir.
I didn’t GUI-green-check the same path two ways. Duplicati did it when I added my profile dir.
I don’t understand how is that related to understanding which checkbox takes a priority if a path is listed in two different places.
approximation?!
Then how am I supposed to ensure that my data will be backed up?! “approximation” is not an acceptable way! Duplicati must show EXACTLY what will be backed up, not approximately what.
Duplicati adds nothing on its own. Are you saying you clicked the Home button - only?
That green-checks Home tree. How did the rest of your items get checked, if not you?
Testing Home button by itself here:
Your screenshot has every item above Home checked, even though they’re in Home.
If you didn’t manually check those (redundantly), then please give your exact checks.
These different GUI trees seem independent to me – unless you actually check them.
For the initial confusion, Windows Application Data is Home → AppData → Roaming.
Read my citation or use %APPDATA% in Explorer or Command Prompt to verify that.
Don’t list them in two places. I’m still thinking you clicked more then Home to get that.
Maybe you could start a new job and figure out exactly how you got to those pictures.
It does. You’re misquoting text that referred only to the “convenience” check boxes.
I strongly suggest you do some or all of these things. Most were mentioned already:
Nowhere will you see all of the lower level folders enumerated. Only the top is named.
The GUI illustrates this by showing checks further down. Uncheck-to-red adds a Filter.
You can see exclude in the Filters section or by the Export or Commandline methods.
Here I checked a green Downloads folder to make it red, so you can see added Filter:
After a Save, here’s what that turned into, using the Commandline screen to view it:
You wouldn’t want the checkmarks to be only what’s backed up. What if you add a file?
The actual backup will pick up new files because it walks down from a top level folder.
If something is supposed to be excluded, it excludes it, if it happens to come across it.
The top level folders say which trees to walk, the filters say which subparts to exclude.
The TEST-FILTERS command can illustrate how it walks a folder, filtering as it goes…
It’s not a priority thing. It’s processing order where filter affects all trees, i.e. filter wins.
It looks like the GUI’s green checkmarks are also not sensitive to the Exclude settings.
AppData is a hidden folder, but AppData stays green if I say to exclude Hidden files.
Duplicati would exclude it though. It knows all the details, but the green checks do not.
They’re a visual convenience, but do not tell the entire story. Most are not even stored.
I do not remember the exact steps I took when I switched to Duplicati after the shutdown of CrashPlan in 2017 or 2018.
But seeing that lately Duplicati’s development slowed to a crawl I started exploring other backup solutions, which made me look back at my Duplicati setup. Maybe I checked those boxes myself. Maybe it was a bug in Duplicati back then. It was many years ago.
There was never a confusion about these paths and where they point to.
My question was: if the same location is exposed in Duplicati in two different places and in one place it’s checked and another unchecked, which one takes priority.
Maybe I clicked them. But I clicked them because they were presented to me in the UI by the backup software. It is unrealistic to expect all users, especially new users to fully know and understand all the relevant internals of the system.
If those options were presented to me I clicked them because I wanted to ensure that my data gets backed up.
As a new user, how am I supposed to interpret if a backup software shows me an important location in my profile? Should I just leave it unchecked even though I want my data backed up?
Then it’s a defect in GUI which does not accurately represent what is and what is not being backed up.
Again, if the same location is shown in Duplicati in 2 different places, then its status whether it will or will not be backed up must be the same. Otherwise, to the end user it’s confusing to understand what takes precedent: 
or 
and from which of the 2 locations.
If Duplicati officially provides GUI, and it is officially documented, telling people to go dig into cli output is just a tiny bit better than telling them to go into through the source code on github to figure out how it works.
Consider everything that’s below a checked folder to be artificially colored to ease administration.
The red check is a real filter applied during the walk of the top level folder, so it takes priority over
convenience-colored green of everything below the top level filter, because it’s done later and the
artificially colored green subfolder isn’t even stored in the source data configuration which is seen
below the multi-method selection trees in its own unexpandable view (but you can uncheck them).
Here, I intentionally clicked two top level folders. Source data tree at bottom reflect my checking:
Ideally you recognize that Duplicati gives you up to three ways to get to a folder, whichever you like.
I suppose we could document that unchecked is neither an include nor exclude, but look elsewhere.
Backups should of course always be tested, but at the very least see if the Restore tree looks right.
Navigating from Computer through the no-convenience-shortcuts starting from C:\ is also possible.
Allowing the user multiple ways to do things (instead of allowing one way) is done a lot in Duplicati. Sometimes it helps, sometimes it’s confusing. Some commercial backup programs aim for “simple”.
Some (such as Backblaze Personal Backup) even make that a selling point. Very little to decide on.
Duplicati also has a huge number of Advanced options, rather than its author deciding the direction.
Once you give people multiple ways, they get used. I don’t see Source picker ever allowing just one.
If you like, open an Issue on it and it can get in the queue, but I wouldn’t count on it being prioritized.
Possibly someone really good at JavaScript could figure out how to make one way color other ways.
That would have avoided your question, and maybe nobody will complain about attribute Exclude…
Expanding the manual a bit might help, but covering every last detail would probably be too verbose.
If you now understand the situation, maybe you can suggest how to explain living with current GUI…
The manual is an independent project (needing volunteers). You could file a pull request or an issue.
Creating a new backup job already talks about green and red, and dips in different ways to navigate:
To select your personal libraries, don’t use the My Documents/Music/Pictures/Videos/Desktop items, but drill down through the file system, probably
C:\Users\<Username>\Documentsetc.
That seems a good start. It tries to suggest picking one way, but it doesn’t explain use of many ways,
and fact that Application Data resides under Home isn’t explained for those not knowing the layout.
You can review your selections under Source data in the file picker.
and a look at the screen suggests this isn’t the “all child items” view that was mentioned a little earlier.
There are times when the difference matters, for example a missing top level folder produces an error.
If one of the green-to-the-bottom items vanishes from the drive, it’s just considered to be a deleted file.
Somewhere down under all these subtleties is your question. Is there a quite simple way to answer it? The red check exclude takes priority, but then you’d have to explain how one would set up the conflict.
Green checks are probably additive, and I hope Duplicati doesn’t actually walk the trees multiple times.
Safest way is to pick one way, and that avoids the question of conflicting red versus green deep below.
You can ensure your data will be backed up by understanding how files will be selected.
If you are not sure, there is the ability to look at the restore screen and see what it contains (that will still only be a snapshot). It is not possible to convey the information you want in the file tree, because everything inside the source folders can change:
These are just some examples of things that duplicati can’t know when it shows you the file picker. That is what I meant by approximation. You need to understand the process to be sure. I agree that it is not intuitive if the same file is shown as included and excluded in different parts of the tree, and it could be changed to show that for the current filesystem snapshot.
Then the user would still need to consider cases like moving the folder in the settings, where it is important which part of the tree you clicked to remove.
UI should to show what’s explicitly included and what’s included by inheritance from a parent directory.
But somebody who’s just starting to use the app can’t be expected to “recognize” how an app treats neutral/unselected directories.
Testing a restore is not the same as manually verifying that every directory and file you assume would be backed up, was backed up.
I’m not saying user has 0 responsibility.
But having obtuse UI elements makes it harder for the user to ensure that they will get the desired results.
This is one of the reasons why I started exploring alternative backup solutions…There doesn’t seem to be that much development going on anymore. I’m not blaming anybody for that, it is what it is.
I have never seen a project where people said that it had too much documentation.
So takes priority of an inherited
But what about explicitly checked dir in one place and explicitly excluded dir in the other place?
If it can’t be resolved programmatically, then UI or backup job at run time should throw a warning/error that there are conflicting selections, and let the user know to correct it.
Deprecating these “convenience shortcuts” and restricting selections only to direct file paths would help prevent users shooting themselves in the foot with conflicting selections or as the documentation states problems that arise when you switch how backup is run from user to service/“SYSTEM”.
which is exactly why I proposed that change. If you like it, great, but that sounds like pushback.
Frankly extended debates like this that involve more than half the Duplicati resources are tiring.
By being in a separate section of both GUI and CLI, and done in folder walk, my answer is “yes”.
You can certainly test it to see if that’s so.
You are going from an accidental corner case to a further hypothetical corner case. You can test it.
For things this obscure, the answers could change even if the fine manual explained corner cases.
I have a vague recollection this came up somewhat recently, and had explicit versus implicit tones.
More nightmares on how complicated or impossible it may be to make the GUI guidance definitive:
--hardlink-policy (Enumeration): Hardlink handling
Use this option to handle hardlinks (only works on Linux/OSX). The "first"
option will record a hardlink ID for each hardlink to avoid storing
hardlinked paths multiple times. The option "all" will ignore hardlink
information, and treat each hardlink as a unique path. The option "none"
will ignore all hardlinks with more than one link.
* values: First, All, None
* default value: All
--symlink-policy (Enumeration): Symlink handling
Use this option to handle symlinks differently. The "store" option will
simply record a symlink with its name and destination, and a restore will
recreate the symlink as a link. Use the option "ignore" to ignore all
symlinks and not store any information about them. The option "follow"
will cause the symlinked target to be backed up and restored as a normal
file with the symlink name. Early versions of Duplicati did not support
this option and bevhaved as if "follow" was specified.
* values: Store, Follow, Ignore
* default value: Store
“Perfect” is not the goal. The current GUI tree is, IMO, generally useful. Compare it to some others.
Maybe (IDK) that’s easier than getting color changes through different ways coordinate their checks, although programmatically recognizing a conflict could also be very similar to resolving such conflict.
There might be other programmatic solutions, but remember there are 2 devs and a thousand+ asks.
Convenience shortcuts have their price, but they’re convenient, just as they’re convenient in the OS:
I don’t know early history, but Windows is still using shortcuts, and it’s more confusing on Windows 11 which pushes OneDrive which makes it harder to fathom. Watch The Problem With OneDrive Backup.
You’re apparently not looking at the GitHub Pull request queue. I hope some of those will be accepted.
Duplicati development is volunteer resource limited, but I think that’s typical. The “it is what it is” fits the situation at any given time. You can ask for changes (docs are easier), help, tolerate, or look for better.
Asking for change is best done in a GitHub issue, as there are 6000 forum topics but no queue system. There is a UX category that I can move this to, if you like. We’ll see if other people have the same “ask”.
Mark folders with excludes inside is the closest I could find, and by that link it will also link back to here.
If you don’t wish to help work out how to improve the manual, that’s probably the end of this discussion.
Ultimately I decided to redo my whole backup set and avoid using these ambiguous “shortcuts” and directly select files/dirs from the each drive’s root.
But it’s still unclear if I should be backing up c:\Users\username\AppData\Local\Duplicati\ itself or if that’s redundant because it’s part of the backup’s structure already. This is not documented or explained anywhere.
Backing up the jobs databases (the files with the weird names) is not very useful because when the backup happens the files themselves are still in the process of being changed, so they are never up to date and restoring them can lead to weird problems.
IMO the best way is to export the job(s) as json files and securing them somewhere different of the main backup, then you have everything you need to recreate the whole setup if you lose the source computer (the job databases can be recreated from the backend). You can also save the main server database if you want.
Yes, I keep a copy each of my backup jobs’ configs in BitWarden’s secure notes.