Need Help with Duplicati Backup - Tricky Path Issues

basata · January 21, 2024, 7:43pm

Hey Duplicati Community,

I’m wrestling with setting up a backup and could use some expertise.

Issue: I’m trying to back up folders from a remote Z: drive, specifically those under folders starting with “Projects” with a “03_Production” sub-folder.

Attempts: I dropped in this source path:

Z:\Projects\Projects*\**\03_Production\

But it throws an “invalid path” error. I also tried a forum answer adding it in the “include” part, but it says “No files found at the destination.”

Explicit listing works though.

Other Tries:

Z:\Projects\*
Swapped slashes
Messed with path elements like Z:\Projects\Projects*03_Production\ , Z:\Projects\Projects*\03_Production\

Setup:

Duplicati GUI - 2.0.7.1_beta_2023-05-25
Windows Server 2022 VM running Windows 10.

Request for Help: Any tips on how to nail this setup? Am I missing something with wildcards?

Edit:
Every asterisk is replaced with folders that are changeable.

gpatel-fr · January 21, 2024, 8:35pm

Hello

Maybe you could find some tips in this thread

basata · January 22, 2024, 1:50pm

Thank you for referring that thread.

I tried understanding it as much as possible but still failed after many trials

According to my understanding the structure should be as follows?

+Z:\Projects\Projects*\\
+Z:\Projects\Projects*\*\\
+Z:\Projects\Projects*\*\03_Production\*

-Z:\Projects\*\\
-Z:\Projects\Projects*\*\\
-Z:\Projects\Projects*\*\*\\
-Z:\Projects\Projects*\*\*\*\\
-Z:\Projects\Projects*\*\*\*\*\\
-Z:\Projects\Projects*\*\*\*\*\*\\
-Z:\Projects\Projects*\*\*\*\*\*\Renders\*

As you can see I have multiple sub-directories that’s also another reason it makes me confused.

I would really be grateful for any help explaining it, if I didn’t get it right.

Also if it’s already explained verbosely excuse my lack of understanding as English isn’t my first language

Thank you

ts678 · January 22, 2024, 2:36pm

It makes me confused too. What are you trying to backup anyway? At first, I feared the existence of 03_Production was the signal to backup the whole Project* that it’s in, or is it backup of files below?

A tree with notes might help. My other fear was you wanted ** to work like the globstar in bash…

basata · January 22, 2024, 3:33pm

Excuse me if I seem to misunderstand something as I’m still a beginner at this.

Yes I thought it would act as a globstar.

What I need backed up is folder “Xproject/03_Production” but I want to exclude the folder “Renders” which is a descendant of multiple folders down the tree.

I tried to structure the tree as clear as I can as it contains many sub-directories.

Z:
└───Projects
    ├───Projects2023
    |  ├───Xproject
    |  |  ├───01_folder
    |  |  ├───02_folder
    |  |  ├───03_Production
    |  |  └───04_folder
    |  └───...
    └───...

An example of a sub-tree containing the folder “Renders”

├───03_Production
   ├───folder_1
   └───folder_2
        ├───folder_2.a
        |    ├───folder_2.a.a
        |    |   ├───folder_2.a.a.a
        |    |   ├───Renders  
        |    |   └───...
        |    └───...
        └───...

Thank you and I hope my situation now is presented to you clearly.

gpatel-fr · January 22, 2024, 4:11pm

Then how about just scrolling in the UI to search the folders that you want to backup and check them?

basata · January 22, 2024, 4:16pm

I’m sorry I didn’t explain this at the beginning, all the directories replaced with asterisk contain multiple changeable folders with different names (we’re talking ~50 projects per year) and there are new folders that are added constantly down the tree each day and I need them all backed up so choosing them explicitly from the UI would need me to edit the configuration before each backup so it would be time consuming and counterproductive.

ex.:

Projects*: Projects2023, Projects2024,etc.
*: Xproject, Yproject, etc.

the only constants are the ones I wrote the name of.

basata · January 22, 2024, 4:27pm

The only option that I could think of that could work is generating a script with variables that lookup all folders I set it to and list them accordingly.
But I asked here first believing that there could be some config I’m not aware of that would work.

gpatel-fr · January 22, 2024, 6:45pm

Well, it’s possible that a combination of include and exclude and possibly regular expressions can fit your use case, as the thread I have pointed you shows that quite complicated configurations can be used.
I don’t think that fitting complicated setups to specific use cases is all right to ask to unpaid helpers, whose time is better spent pointing at simple adaptations for usual problems. In your case, maybe your ‘generated’ files are fitting a pattern of having a specific extension, in this case these files could be excluded by extension instead of a folder filter.

basata · January 22, 2024, 7:09pm

Ok, as I said, I tried whatever I could think of before I got here.

I studied the thread you pointed to and failed, that’s why I got back.

I apologize if it was my lack of understanding.

I just needed guidance or any advice to steer me in the right direction.

I never intended for any unpaid service to my specific use case and I apologize for it.

I’m in constant state of learning Duplicati and I keep studying over the docs and discussions on the forum, and I just asked for help from experts with my situation and I apologize for making it appear as if I’m just asking for a solution.

And I would never like for anyone to spend their time helping me when they have better things to do.
If you’ve got better things to do please just ignore me.

Thank you

ts678 · January 22, 2024, 7:49pm

Example Scripts shows how run-script-before can seemingly set up from folders that you find.

github.com

duplicati/duplicati/blob/e7c191389cdb3fdb9500d83ebd06e0ae99634fd1/Duplicati/Library/Modules/Builtin/run-script-example.bat#L110-L114


      
          REM DUPLICATI__LOCALPATH

          REM This is the path to the folders being backed up or restored. This variable

          REM is empty operations  other than backup or restore. The local path can 

          REM contain : to separate multiple folders. This value can be changed by echoing

          REM --localpath = "new value".

Yes and no. As I understand it, the Source list is processed recursively and Filters acts on the findings.
--include filter is still a filter, so I think not the same as changing the Source list. Could use an expert.

What’s a little odd IMO about Duplicati wildcard * is that it can cross path separators like a bash ** can.
A globstar might be a nice way to look for 03_Production folders. Unfortunately, you’re using Windows.

If you can supply the 03_Production folders, ignoring Renders below those is a really simple exclude, similar to what the GUI would make when you exclude a Folder from dropdown. See in three-dot menu.

What gets weird is when you have combinations of includes and excludes, similar to the other situation. Although I’d still want the documentation to cover why it works, I guessed my way to a possible solution provided that it doesn’t get any stranger. The varied levels look below 03_Production. Top part is firmer?

C:\Program Files\Duplicati 2>tree /f Z:\Projects
Folder PATH listing

Z:\PROJECTS
│   nope.txt
│
├───Projects2023
│   │   nope.txt
│   │
│   ├───XProject
│   │   │   nope.txt
│   │   │
│   │   ├───01_folder
│   │   │       nope.txt
│   │   │
│   │   ├───02_folder
│   │   ├───03_Production
│   │   │   │   want.txt
│   │   │   │
│   │   │   ├───folder_1
│   │   │   │       want.txt
│   │   │   │
│   │   │   └───folder_2
│   │   │       │   want.txt
│   │   │       │
│   │   │       └───folder_2.a
│   │   │           │   want.txt
│   │   │           │
│   │   │           └───folder_2.a.a
│   │   │               │   want.txt
│   │   │               │
│   │   │               └───Renders
│   │   │                       nope.txt
│   │   │
│   │   └───04_folder
│   └───YProject
└───Projects2024
C:\Program Files\Duplicati 2>Duplicati.CommandLine.exe test-filters Z:\Projects --include="Z:\tmp\Projects\\" --include="Z:\Projects\Projects*\\" --include="Z:\Projects\Projects*\*Project\\" --exclude="*\Renders\*" --include="Z:\Projects\Projects2023\XProject\03_Production\*" --exclude="*" | find "want"
Including path due to filter: Z:\Projects\Projects2023\XProject\03_Production\want.txt => (Z:\PROJECTS\PROJECTS2023\XPROJECT\03_PRODUCTION\*)
Including path due to filter: Z:\Projects\Projects2023\XProject\03_Production\folder_2\want.txt => (Z:\PROJECTS\PROJECTS2023\XPROJECT\03_PRODUCTION\*)
Including path due to filter: Z:\Projects\Projects2023\XProject\03_Production\folder_2\folder_2.a\want.txt => (Z:\PROJECTS\PROJECTS2023\XPROJECT\03_PRODUCTION\*)
Including path due to filter: Z:\Projects\Projects2023\XProject\03_Production\folder_2\folder_2.a\folder_2.a.a\want.txt => (Z:\PROJECTS\PROJECTS2023\XPROJECT\03_PRODUCTION\*)
Including path due to filter: Z:\Projects\Projects2023\XProject\03_Production\folder_1\want.txt => (Z:\PROJECTS\PROJECTS2023\XPROJECT\03_PRODUCTION\*)

C:\Program Files\Duplicati 2>Duplicati.CommandLine.exe test-filters Z:\Projects --include="Z:\tmp\Projects\\" --include="Z:\Projects\Projects*\\" --include="Z:\Projects\Projects*\*Project\\" --exclude="*\Renders\*" --include="Z:\Projects\Projects2023\XProject\03_Production\*" --exclude="*" | find "nope"
Excluding path due to filter: Z:\Projects\nope.txt => (*)
Excluding path due to filter: Z:\Projects\Projects2023\nope.txt => (*)
Excluding path due to filter: Z:\Projects\Projects2023\XProject\nope.txt => (*)
Excluding path due to filter: Z:\Projects\Projects2023\XProject\03_Production\folder_2\folder_2.a\folder_2.a.a\Renders\nope.txt => (*\RENDERS\*)
Excluding path due to filter: Z:\Projects\Projects2023\XProject\01_folder\nope.txt => (*)

I haven’t looked at it that carefully, but superficially it seems to be finding what I want, and not the rest.
My Z: drive is done by subst to a path on C:. I’m also worried about your remote Z:. Remote is worse.

Jojo-1000 · January 22, 2024, 10:43pm

To summarize @ts678 for what to put in the UI (slightly different format from command line):

Source:

Z:\Projects\

Filters: (edit as text)
Order is important, because the first match determines the result.

+Z:\Projects\Projects*\
+Z:\Projects\Projects*\*Project\
-*\Renders\*
+Z:\Projects\Projects2023\XProject\03_Production\*
-*

There was also Z:\tmp\Projects\ as include, but I don’t see why.

Note that the UI markings (x or check) are not shown correctly in combination with wildcards, unless you use the latest canary (2.0.7.100). In the canary you should be able to confirm the filters by looking at the file tree.

ts678 · January 22, 2024, 11:00pm

That was a conversion typo. Z: was C:\tmp before I did the subst. It seems to work without that though.

Do you understand this well enough to say why I need to build a trail this way instead of just having an include override an exclude? I know order in the list matters, and Duplicati also has processing orders.

EDIT:

My guess about the need to ladder your way down is the include needs to override the exclude enough that one can see the next rung on the way down. Filters says below, but doesn’t detail how to escape it:

For instance, if the first rule excludes a folder, then this folder and all files within will be excluded from the backup even if following rules include this folder or its files.

Also it is recommended to write the folder rules one directory level at a time.

and it probably means going down. It’s not clear what a folder rule is though, especially for a messy RE.

basata · January 23, 2024, 1:09pm

Thank you all for explaining this stuff and helping me out with the backup setup.

I can see now that I was not aware on how to put those paths together in order and just listed them redundantly.

Thank you so much for your patience putting up with my path confusion.