Problems using test-filter

I’m trying to test my filters as they don’t seem to be properly excluding directories. To do my testing I did the following:

  1. Click on the backup job
  2. Click on Commandline
  3. Select test-filter
  4. Clear the target URL
  5. Set Commandline arguments to be the directory to test “/root”
  6. Leave everything else alone, this has exclude folder “/root/.config/Duplicati/”
  7. Click run test-filter…

When I get the output I still see files in /root/.config/Duplicati/ in the output. I don’t see anything about filters applied either. Am I not running the test command properly?

Is that saying job editing used Add filter of Exclude folder with value /root/.config/Duplicati/ or that Commandline screen shows that value? If it shows an --exclude of */root/.config/Duplicati/*/ I think there may be a GUI entry bug that makes Exclude folder dropdown treat your value like Exclude directories whose names contain dropdown. If one goes back to look at Filters, that’s what’s shown.

One can click the three-dot-menu, or switch screens to adjacent screens 2 or 4 then back, to watch the dropdown change. Setting the dropdown again can make it stay put. Three-dot menu can confirm results.

Another possible point of confusion (for those who read the documentation) is that it’s not clear if a trailing slash used to imply folder is supposed to be entered for a folder dropdown, or if it’s added automatically.

The first. I created the job and used Add filter, Exclude folder with the value ‘/root/.config/Duplicati’.
Edit as text shows
-/root/.config/Duplicati/ -*/.cache

I left the change in and the backup last night gave me this

System.NullReferenceException: Object reference not set to an instance of an object at Duplicati.Library.Utility.FileBackedList`1[T].Add (T value) [0x00075] in <621233224d6648a0a1cd0bb505473fc4>:0 
 at Duplicati.Server.WebServer.RESTMethods.CommandLine+LogWriter.WriteLine (System.String value) [0x00070] in <fe28905ee30b422e8d475f1cfdb85515>:0 
 at System.IO.TextWriter.WriteLine (System.String format, System.Object arg0, System.Object arg1) [0x0000f] in <8f2c484307284b51944a1a13a14c0266>:0 
 at Duplicati.CommandLine.ConsoleOutput.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x00053] in <04206d56f2084515b87543fcb90a7e00>:0 
 at Duplicati.Library.Main.MultiMessageSink.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x0000b] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
 at Duplicati.Library.Main.ControllerMultiLogTarget.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x00046] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
 at Duplicati.Library.Logging.LogScope.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x0001e] in <2fa046254e6b430abcb53256f54d51b3>:0 
 at Duplicati.Library.Logging.Log.WriteMessage (Duplicati.Library.Logging.LogMessageType type, System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00026] in <2fa046254e6b430abcb53256f54d51b3>:0 
 at Duplicati.Library.Logging.Log.WriteErrorMessage (System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00000] in <2fa046254e6b430abcb53256f54d51b3>:0 
 at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00266] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
 at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00068] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
 at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00307] in <fe28905ee30b422e8d475f1cfdb85515>:0 

Job --> Commandline --> Edit as text here gives me something that looks like it’s roughly right, except it gave it the Exclude directories whose names contain rewrite that I didn’t ask for, thus is too broad.

--exclude=*/root/.config/Duplicati*/

I’m also not sure why you got the plus and minus syntax instead of multiple options like I get. They seem mainly intended for places where you can’t have multiple options, so must have a sequence in a string…

C:\Program Files\Duplicati 2>Duplicati.CommandLine.exe help filters

Duplicati can apply globbing and regex filter rules to backup and restore
specific files only. Globbing filters can be used in file names. To specify a
regex filter put the filter in [brackets].
  Globbing: *.txt
  Regex: [.*test\.txt]

  --include=<filter>
    Adds an include filter
  --exclude=<filter>
    Adds an exclude filter

Duplicati also has several built-in groups of filters, which include commonly
excluded files and folders for different operating systems. These can be
specified by putting the name of the group in {curly brackets}.
  {None}
  {DefaultExcludes}
  {DefaultIncludes}
  {Applications}
  {CacheFiles}
  {OperatingSystem}
  {SystemFiles}
  {TemporaryFiles}
For more details about these groups, including exactly what files they cover,
see 'filter-groups'.

Inside scripts and the commandline --parameters-file, where multiple filters
are supplied with a single option, filters must be prefixed with either a
plus (+) or a (-), for include and exclude respectively. Example:
  +*.txt;-[.*\.xyz];+*



C:\Program Files\Duplicati 2>

I’m not even sure the plus and minus syntax is legal outside of the scenarios described above (at bottom). Below is a test showing test-filters handling an --exclude properly, but choking on a raw filter with a minus:

HP4-VirtualBox ~ # duplicati-cli test-filters /root/.config --exclude=/root/.config/Duplicati/ | grep Duplicati
Excluding path due to filter: /root/.config/Duplicati/ => (/root/.config/Duplicati/)
HP4-VirtualBox ~ # duplicati-cli test-filters /root/.config -/root/.config/Duplicati/ | grep Duplicati

System.IO.IOException: The source folder -/root/.config/Duplicati/ does not exist, aborting backup
  at Duplicati.Library.Main.Controller.ExpandInputSources (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x002c4] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass31_0.<TestFilter>b__1 (Duplicati.Library.Main.TestFilterResults result) [0x00011] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00271] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.Library.Main.Controller.TestFilter (System.String[] paths, Duplicati.Library.Utility.IFilter filter) [0x0008e] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.CommandLine.Commands.TestFilters (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x00040] in <04206d56f2084515b87543fcb90a7e00>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00313] in <04206d56f2084515b87543fcb90a7e00>:0 
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <04206d56f2084515b87543fcb90a7e00>:0 
HP4-VirtualBox ~ # 

Basically, I’m not sure how your Edit as text got what it got, and even odder, how -*/.cache` got into it. Regardless, maybe you can relocate the two exclude filters into separate --exclude settings using the UI. Testing using an actual shell is where I started my study of filters, before getting into the GUI operation…

I added “~~~” around your message (and a few carriage returns) to make it easier to read and it looks to me like the issue was related to the to what was backed up.

Was this reported as an error (backup failed) or a warning (backup succeeded)?

The plus/minus syntax came from the settings for the job when I edit the filters as text.

The exception was backup failed.

Trying again with the command line test.

  1. Entered the command line
  2. Changed the command to test-filter
  3. Clicked edit as text
  4. cleared the target URL - leaving it in appears to make Duplicati unhappy
  5. Changed command line arguments to just “/root/”, removing the other directories.
  6. Under advanced options removed everything except:
--exclude=/*root/.config/Duplicati*/
--exclude=*/.cache

Results:

System.NullReferenceException: Object reference not set to an instance of an object
  at Duplicati.Library.Utility.FileBackedList`1[T].Add (T value) [0x00075] in <621233224d6648a0a1cd0bb505473fc4>:0 
  at Duplicati.Server.WebServer.RESTMethods.CommandLine+LogWriter.WriteLine (System.String value) [0x00070] in <fe28905ee30b422e8d475f1cfdb85515>:0 
  at System.IO.TextWriter.WriteLine (System.String format, System.Object arg0, System.Object arg1) [0x0000f] in <8f2c484307284b51944a1a13a14c0266>:0 
  at Duplicati.CommandLine.ConsoleOutput.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x00053] in <04206d56f2084515b87543fcb90a7e00>:0 
  at Duplicati.Library.Main.MultiMessageSink.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x0000b] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.Library.Main.ControllerMultiLogTarget.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x00046] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.Library.Logging.LogScope.WriteMessage (Duplicati.Library.Logging.LogEntry entry) [0x0001e] in <2fa046254e6b430abcb53256f54d51b3>:0 
  at Duplicati.Library.Logging.Log.WriteMessage (Duplicati.Library.Logging.LogMessageType type, System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00026] in <2fa046254e6b430abcb53256f54d51b3>:0 
  at Duplicati.Library.Logging.Log.WriteErrorMessage (System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00000] in <2fa046254e6b430abcb53256f54d51b3>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00266] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.Library.Main.Controller.TestFilter (System.String[] paths, Duplicati.Library.Utility.IFilter filter) [0x0008e] in <c6c6871f516b48f59d88f9d731c3ea4d>:0 
  at Duplicati.CommandLine.Commands.TestFilters (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x00040] in <04206d56f2084515b87543fcb90a7e00>:0 
  at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00313] in <04206d56f2084515b87543fcb90a7e00>:0 
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <04206d56f2084515b87543fcb90a7e00>:0 
Return code: 100

That results stack trace looks similar to Backups fail with error: Object reference not set to an instance of an object except that yours started in TestFilter while the other started in Backup. They ended in the same way, which is possibly a failure in collecting the output lines, but my previous attempts to debug the failure dead-ended. Interestingly both failures are Linux. Sometimes updating to latest mono can solve surprising things. [Solved] Failed to connect: Only http or https scheme is allowed in 2.0.4.10_canary_2018-12-29 is the latest case where I wished I’d suggested trying it. I’m not sure if you want to try updating mono though.

If you happen to have a Windows system showing a stack trace like that, my chances of debug get higher. You could also file an issue for the developers, especially if you can come up with a very simple test case.

An alternative suggestion is to try real shell instead of GUI one, being careful to quote strings as necessary. While I’d like to solve the Object reference failure, I think I’ll have better luck at getting test-filters running. Following your steps didn’t break Commandline (but my /root is almost certainly smaller – does it matter?) however one oddity is that adding the cache exclude broke the Duplicati exclude. Under shell, it still works:

# duplicati-cli test-filters /root/ --exclude="/*root/.config/Duplicati*/" --exclude="*/.cache" | grep Duplicati
Excluding path due to filter: /root/.config/Duplicati/ => (/*root/.config/Duplicati*/) || (*/.cache)
# 
# duplicati-cli test-filters /root/ --exclude="*/.cache" --exclude="/*root/.config/Duplicati*/" | grep Duplicati
Excluding path due to filter: /root/.config/Duplicati/ => (*/.cache) || (/*root/.config/Duplicati*/)
# 

I haven’t yet tried reversing order in the GUI. I wonder if this unusual case of repeating options won’t work? OK, I had to try reversing, and it seems like perhaps the last --exclude option it gets is the only one it does. Seems like a possible bug (although a somewhat obscure one) in however Commandline is implemented. For the clear bug of NullReferenceException, does it also fail with short output e.g. /root/.config/ not /root/?

I looked at this, and it seems to be a shutdown race of sorts, where the log is disposed, and there is a message written to the log after.

The logic will not allow the commandline process to spend more than 5 minutes without output. I have fixed this for the next canary, but for the beta, you need to use the real commandline, like @ts678 suggest.

Using duplicati-cli I get the same results that you do. I notice that the command line only has “–exclude” while the UI has “Exclude folder” and “Exclude expression”. Perhaps there’s a difference here?

In the end what I’m trying to accomplish is having duplicati not backup it’s local database because that’s causing an exception as well saying something about issue 1400. See this post New backup state Error: Checking errors, related to #1400

@kenkendk

I’m also seeing this error in backups. If I have the following in my source data filters:

  • “Exclude folder” “/root/.config/Duplicati
  • “Exclude expression” “*/.cache”

I always had the “Exclude expression” “*/.cache” and everything was working fine. I added the “/root/…” to try and avoid this 1400 error that was showing up. Adding that filter is when I started getting this null reference exception. If I change the “Exclude folder” to “Exclude expression” I still get the exception.

The error doesn’t show up in the log for the job, it only shows up in the system log (About -> Show log).

I just removed the filter for “/root…” and still got the exception. So something has duplicati in a very unhappy state.

~Then it should be a different stack trace (and different source of the problem) because the stack trace you posted explictily mentions calling from the RESTMethods.CommandLine class, which is not called when running backups normally.~

Edit: I take that back. It seems that there is a lingering log destination that is somehow invoked when you run the backup. Restart the Duplicati (server) instance and the problem should go away.

The UI tries to make it simpler to use filters, by “hiding” some things for you. If you click the ... button you can see the actual filters generated by the UI.

Can you post the actual filters you use? Or pehaps try to export to commandline to see if there are some hidden settings that mess it up.

There is a slightly longer explanation for how filters work here:

Restarted the server and got rid of the null reference exception. Then ran a backup, this gave the 1400 error again. This was expected because I didn’t have a live filter for the duplicati database. I just added that back. Here’s the command line

mono /usr/lib/duplicati/Duplicati.CommandLine.exe backup "webdavs://server//path?auth-username=user&auth-password=password" /etc/ /home/ /root/ /usr/local/ /opt/ --thread-priority=belownormal --send-mail-level="Warning,Error,Fatal" --send-mail-to="email" --send-mail-url="smtp://localhost:25/?starttls=never" --throttle-download=5MB --throttle-upload=5MB --send-mail-subject="host: Duplicati %OPERATIONNAME% report for %backup-name% - %RESULT%" --backup-name="name" --dbpath=/root/.config/Duplicati/MOLZFSKPZF.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50MB --passphrase="password" --retention-policy="1W:1D,4W:1W,12M:1M" --disable-module=console-password-input --exclude="*/.cache" --exclude=/root/.config/Duplicati/ 

Did another backup and no 1400 error.

1 Like