Jail on TrueNAS has stopped running. Duplicati crashing

Support
1

It appears that Duplicati is now crashing inside the TrueNAS core jail and I really can’t figure out why. This process had been running very well for quite a long time. As part of an update, the jail got updated and now the process won’t run. I decided to back it all out and start from scratch and so I recreated the jail from scratch only to discover that the same issue still occurs. I’m not entirely sure what is going on. If I try to invoke the service manually it throws the following error:

Unable to start up, perhaps another process is already running?
Error message: System.InvalidOperationException: object_op ---> System.BadImageFormatException: Method has no body
  at System.IO.CoreFX.FileSystemWatcher.StartRaisingEvents () [0x0001c] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
  at System.IO.CoreFX.FileSystemWatcher.StartRaisingEventsIfNotDisposed () [0x00019] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
  at System.IO.CoreFX.FileSystemWatcher.set_EnableRaisingEvents (System.Booleanvalue) [0x0001d] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
  at (wrapper remoting-invoke-with-check) System.IO.CoreFX.FileSystemWatcher.set_EnableRaisingEvents(bool)
  at System.IO.CoreFXFileSystemWatcherProxy+<>c.<StartDispatching>b__9_0 (System.IO.CoreFX.FileSystemWatcher internal_fsw, System.IO.FileSystemWatcher fsw) [0x00048] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
  at System.IO.CoreFXFileSystemWatcherProxy.Operation (System.Action`4[T1,T2,T3,T4] map_op, System.Action`2[T1,T2] object_op, System.Object handle, System.Action`2[T1,T2] cancel_op) [0x00184] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
   --- End of inner exception stack trace ---
  at System.IO.CoreFXFileSystemWatcherProxy.Operation (System.Action`4[T1,T2,T3,T4] map_op, System.Action`2[T1,T2] object_op, System.Object handle, System.Action`2[T1,T2] cancel_op) [0x001a6] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
  at System.IO.CoreFXFileSystemWatcherProxy.StartDispatching (System.Object handle) [0x00004] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
  at System.IO.FileSystemWatcher.Start () [0x0001b] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0
  at System.IO.FileSystemWatcher.set_EnableRaisingEvents (System.Boolean value)[0x0003d] in <3cd1e5eafcb04e86ab91ae63b4afeccc>:0

It doesn’t generate any logfiles. Im assuming because the process doesn’t even get that far. Even running mono with --debug doesn’t produce anything more that the exception above. Anyone have any thoughts?

2

FileSystemWatcher is only used in the server, for control_dir_v2 which is supposed to prevent multiple instances running.

BadImageFormatException happens when something with the DLLs is wrong. In this case, it looks like some of the system libraries are broken (which is a mono issue).

Different error, but note the object_op and similar stack trace:

The filesystem watcher seems to be a part of mono that breaks sometimes. In that case it was solved by downgrading mono. What version do you have?

1 Like
3

This is the current version:

# mono --version
Mono JIT compiler version 5.20.1.34 (5.20.1.34 Sun Oct 22 22:47:55 UTC 2023)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notification:  kqueue
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          supported, not enabled.
        Suspend:       preemptive
        GC:            sgen (concurrent by default)

Seems like if there is mismatch here I need to push this back up to the plugin maintainer.

4

From an up-to-date Ubuntu computer:

mono --version
Mono JIT compiler version 6.12.0.200 
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        Interpreter:   yes
        LLVM:          yes(610)
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

your version dates from 2019.

5

This isn’t really something I can completely control from the package bundle installer for the plugin. The whole thing is supposed to be an end-to-end tested deployment.

6

As far as I know there is no official Duplicati package for TrueNAS, so someone else has produced your ‘end to end tested deployment’. Maybe they would be the best people to solve it ?
If this is it:

it does not seem to be updated for last Duplicati version.
Maybe it was auto-upgraded to last Duplicati version and it does not work with old Mono ?

however if I understand correctly, this thing seems to bring the standard system mono package (it looks a bit like the Docker Duplicati version) and it can’t be this old.

7

I’m not so sure. FreeBSD does (or tries to do) the mono port themselves, as mono project doesn’t do it.

Bug 238089 - lang/mono: Update to 5.20 (5.10.0 dates to 2018-02)
The actual commit was on 2023-08-08, and the current final comment expressed wish to remain on 5.x, despite some requests in the thread to go higher. The number of potential versions is somewhat limited.

Bug 251795 - lang/mono: convert to metaport refers to MONO_DEFAULT. Could end users modify that?
Sorry I don’t know much about the FreeBSD ports system which I assume got used to get Oct 22 mono:

If you can’t change MONO_DEFAULT, is there somewhere you can edit mono to mono5.10 before building?

https://www.freshports.org/lang/mono5.10/ has some info on that version. Maybe it will work better here.
For an API level, Duplicati needs .NET Framework 4.7.1 or later. 5.10 release note says that it qualifies.

8