Strange behaviour after after upgrade to 2.1.0.4

[ts678]

Works fine here (admittedly I’m trying the brand new 2.1.0.5, but it should behave the same).

I backed up a folder with a subfolder called sub1 initially excluded it by clicking UI (turns red).

Export As Command-line shows it (properly quoted for Windows Command Prompt) like this:

--exclude="C:\backup source\filter\sub1\\"

I make parameters-file with that exclude minus the stuff added for Command Prompt, so

--exclude=C:\backup source\filter\sub1\

Run it while also keeping CLI option. As expected from the help text, it tells me not to do that:

Unable to read the parameters file "C:\Duplicati\duplicati-2.1.0.5_stable_2025-03-04-win-x64-gui\RUN\parameters-file.txt", reason: Filters cannot be specified on the commandline if filters are also present in the parameter file. Use the special --replace-filter, --append-filter, or --prepend-filter options to specify filters inside the parameter file. Each filter must be prefixed with either a + or a -, and multiple filters must be joined with ;

I drop back to just using the parameters-file without the can’t-use-both CLI option. Works fine.

Return to the CLI option to see if I can --replace-filter it with itself. The file content now has:

--replace-filter=-C:\backup source\filter\sub1\

Works fine.

Now try to replace it with something actually different. Until now, nothing has changed in backup.

--replace-filter=-C:\backup source\filter\nope\

finally undoes my original filter, as evidenced by the first appearance of sub1 folder for Restore.

This is certainly not testing everything, but everything tested seems to be operating as expected.

EDIT 1:

Without checking GUI to see if a version was updated (none implies nothing changed), CLI says

Data uploaded: 0 bytes until the final change where I actually did something to change filters.

EDIT 2:

  Uploading file duplicati-b73888592114c429faa9b56eb130c3f07.dblock.zip (777 bytes) ...
  Uploading file duplicati-ic29e79b0abf44a1a82221e5c8ef0bba0.dindex.zip (626 bytes) ...
  Uploading file duplicati-20250305T005138Z.dlist.zip (1.12 KB) ...

then happens. The dlist file has the UTC time of the backup version. Restore shows local time.

EDIT 3:

I’m going to delete that last version so I can watch a verbose log when file in sub1 goes in again.

Running commandline entry
Finished!

            
  Listing remote folder ...
  Deleting file duplicati-20250305T005138Z.dlist.zip  (1.12 KB) ...
  Deleting file duplicati-b73888592114c429faa9b56eb130c3f07.dblock.zip  (777 bytes) ...
  Deleting file duplicati-ic29e79b0abf44a1a82221e5c8ef0bba0.dindex.zip  (626 bytes) ...
These filesets were deleted:
0: 3/4/2025 7:51:38 PM
Return code: 0

Use Command Prompt history to run again adding --console-lg-level=verbose and see it say

Adding directory C:\backup source\filter\sub1\
Checking file for changes C:\backup source\filter\sub1\subfile.jpg, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 12/21/2021 6:47:27 PM vs 1/1/0001 12:00:00 AM
New file C:\backup source\filter\sub1\subfile.jpg

which reminds me of what you saw (I think – feel free to check), so that’s how an added file looks.

The COMPARE command run in GUI Commandline:

Listing changes
  1: 3/4/2025 7:42:57 PM
  0: 3/4/2025 8:23:20 PM

Size of backup 1: 45 bytes

  1 added entries:
  + C:\backup source\filter\sub1\subfile.jpg


  Added files:       1
  No changes found
Size of backup 0: 45 bytes
Return code: 0

Everything seems fine.

[kenkendk]

@lipp_f

with log verbose I can see thousands of entries like this:

Checking file for changes E:\Data\Invest\data\tmp\fws-553361_20240222070132.xml, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 2024-02-21 12:25:19 PM vs 0001-01-01 12:00:00 AM

or like this

Checking file for changes E:\Users\****\Documents\Samsung\SmartSwitch\backup\SM-A505W\SM-A505W_18076321955\SM-A505W_20240421154354\APKFILE\atws.app.png, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 2024-04-21 7:46:50 PM vs 0001-01-01 12:00:00 AM

These are actually excluded from gui and cli with

--exclude="E:\Data\Invest\data\tmp\\"
--exclude="E:\Users\****\Documents\Samsung\\"'

I can see here that it thinks the files are “new” meaning that the path was not part of the previous backup.

The only explanation I can come up with is that the filters are simply not working correctly from the CLI instance. If this is the case, it explains the issues you see:

• GUI run: exclude folders
• CLI run: include folders, 8000 new files
• GUI run: exclude folders, 8000 deleted files

You should be able to see verbose messages concerning the evaluation of filters, where it says something like “Including file because no filters matched”.

@lipp_f

The help does not say that --exclude should not be in the parameters file. Also tried --replace-filter with colon and you can imagine the result when the some of the filters start with e:\ .

This part is quite dated and not well documented. It has not been changed for several years, but is confusing when it comes to filters. The problem is that the order of filters matters, so you need to be able to choose the order of filters, which is not important for the other options.

The “special” options for the parameters file are:

• source: Adds a source to the list of inputs
• target: Changes the destination
• append-filter: Appends a filter to the end of the existing list of filters
• prepend-filter: Prepends a filter before the existing filters
• replace-filter: Replaces all filters

In your case, you would likely want to use append-filter.

You cannot directly see the applied filters, which makes it a bit difficult to use. But you can use the test-filters command of the CLI together with the parameters file to see when it is working.

@lipp_f

I am using same script with same filters for about 8 years, always worked until this upgrade.

That seems really odd. The code around filters and the parameters file has not been changed. Any chance this happened before but was not noticed?