Add source data: opening Home folder crashes Duplicati

Hi,

I cannot add new folders to my backup. Whenever I go to

Backup > my backup > Edit… > 3: Source Data

and click the triangle that opens the Home folder, Duplicati crashes.
In detail:

When I try to open the Home folder, first the icon vanishes:

02%20Open%20Home%20-%20icon%20vanishes

Then the Duplicati server crashes. The UI shows a Connection Lost message:

Afer restarting Duplicati, this error message occurs:

04%20Empty%20error%20message

I already repaired the database but this had no effect.

Prior to this, a couple of backups gave me warnings about a missing fileset, for example:

Expected there to be a temporary fileset for synthetic filelist (13, duplicati-b015a3573200c46c19fa3a5f38be96ae3.dblock.zip.aes), but none was found?

However, I suppose these are not related to the data source crash issue but I want to mention them here just in case.

How can I fix this issue?

Thanks,
Christoph

Hi @christophberger, welcome to the forum!

I’ve never heard of this happening before so we’re going to have to do a little info gathering / testing to figure out what’s going on

  • what version of Duplicati are you using?
  • on what OS are you running it? (Looks like Windows?)
  • does it crash if you use “Computer” to browse to it?
  • does anything show up in the global (main menu About -> Show logs) paste? (If using Live log select “Profiling”.)

Hi @JonMikelV, thank you for the reply, and apologies for failing to include basic information.

  • my Duplicati version is 2.0.3.3.
  • OS is macOS 10.13.4
  • Mono JIT compiler version 5.4.1.6 (tarball Mon Dec 11 14:59:42 GMT 2017), installed via Homebrew

Duplicati crashes as soon as I open the Home folder, no matter how I navigate to the folder. (So yes, also via “Computer”.) Maybe an important detail: the Home folder is the only one that contains subfolders backed up by Duplicati.

I have not been able to collect log output, as the crash wipes the live log completely. I also tried keeping the live log open in a separate tab while replicating the crash. The last live log output was about the backup done recently. No error about selecting source data.

Does Duplicati store its log files permanently somewhere? I already checked the usual places that also appear in Console.app, like /var/log, ~/Library/Logs, etc. but found no Duplicati log files.

Thanks,
Christoph

Update: From another forum thread I learned that Duplicati writes a stack trace to stdout when started from the command line. I cannot upload .txt files here, but here are the first few lines:

~> /Applications/Duplicati.app/Contents/MacOS/duplicati                                                                                                                                            
Server has started and is listening on 127.0.0.1, port 8200
Stacktrace:

at <unknown> <0xffffffff>
at (wrapper managed-to-native) System.IO.MonoIO.FindFirstFile (string,string&,int&,int&) [0x00000] in <6e9b92f0d119424382ef180639777acb>:0
at System.IO.FileSystemEnumerableIterator`1<TSource_REF>.CommonInit () [0x0001d] in <6e9b92f0d119424382ef180639777acb>:0
at System.IO.FileSystemEnumerableIterator`1<TSource_REF>..ctor (string,string,string,System.IO.SearchOption,System.IO.SearchResultHandler`1<TSource_REF>,bool) [0x000d6] in <6e9b92f0d119424382ef180639777acb>:0
at System.IO.FileSystemEnumerableFactory.CreateFileNameIterator (string,string,string,bool,bool,System.IO.SearchOption,bool) [0x00009] in <6e9b92f0d119424382ef180639777acb>:0
at System.IO.Directory.EnumerateFileSystemNames (string,string,System.IO.SearchOption,bool,bool) [0x00000] in <6e9b92f0d119424382ef180639777acb>:0
at System.IO.Directory.InternalEnumerateFileSystemEntries (string,string,System.IO.SearchOption) [0x00000] in <6e9b92f0d119424382ef180639777acb>:0
at System.IO.Directory.EnumerateFileSystemEntries (string) [0x0000e] in <6e9b92f0d119424382ef180639777acb>:0
at Duplicati.Server.WebServer.RESTMethods.Filesystem/<>c__DisplayClass3_0.<ListFolderAsNodes>b__0 (string) [0x00008] in <670b0e4a9ae144208688fcb192d20146>:0
...

Then a native stacktrace follows:

Native stacktrace:

w32error-unix.c: unknown error (6) "Device not configured"
	0   mono-sgen                           0x0000000106d6705e mono_handle_native_crash + 242
	1   libsystem_platform.dylib            0x00007fff768dbf5a _sigtramp + 26
	2   ???                                 0x0000000116768558 0x0 + 4671833432
	3   libsystem_c.dylib                   0x00007fff766791ae abort + 127
	4   mono-sgen                           0x0000000106ef68e0 mono_log_write_logfile + 351
	5   mono-sgen                           0x0000000106f0a18b monoeg_g_logv + 83
	6   mono-sgen                           0x0000000106f0a27e monoeg_g_log + 120
	7   mono-sgen                           0x0000000106ddad42 mono_w32error_unix_to_win32 + 239
	8   mono-sgen                           0x0000000106dd4e9d _wapi_set_last_path_error_from_errno + 61
	9   mono-sgen                           0x0000000106dd5df1 mono_w32file_find_first + 1053
	10  mono-sgen                           0x0000000106e00c8a ves_icall_System_IO_MonoIO_FindFirstFile + 49
	...

After that, some gdb output follows.

Debug info from gdb:

(lldb) command source -s 0 '/tmp/mono-gdb-commands.gUjQC1'
Executing commands in '/tmp/mono-gdb-commands.gUjQC1'.
(lldb) process attach --pid 99620
Process 99620 stopped
* thread #1, name = 'tid_307', queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
	frame #0: 0x00007fff76714246 libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
->  0x7fff76714246 <+10>: retq
	0x7fff76714247 <+11>: nop

libsystem_kernel.dylib`semaphore_wait_signal_trap:
	0x7fff76714248 <+0>:  movq   %rcx, %r10
	0x7fff7671424b <+3>:  movl   $0x1000025, %eax          ; imm = 0x1000025
Target 0: (mono-sgen) stopped.

Executable module set to "/usr/local/Cellar/mono/5.4.1.6/bin/mono-sgen".
Architecture set to: x86_64h-apple-macosx.
(lldb) thread list
Process 99620 stopped
...

At the end of the output, this message occurs:

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Shell error code is 134. I am posting this information here in the hope it may help understanding the scenario better.

Thanks for the detailed info!

Unfortunately, I’m not strong on the MacOS side so I’m going to see if maybe @kenkendk or @Pectojin have any thoughts on what might be causing it.

Oh - are you able to back up OTHER folders (not in Home) or does selecting anything though the GUI cause the crash?

Yes, adding other folders works fine, and I can open and edit the tree without problems, even after another sync.

BTW, I switched to the advanced editor and removed all subtrees of the Home folder, then saved the change, and then edited the source data again in the tree view. However, opening the Home folder still causes the same crash. So it seems the crash does not depend on the list of selected subtrees.

Very odd - I don’t suppose you’ve got any unusual characters in your home path that could be causing parsing errors when it’s passed to the server?

@kenkendk & @Pectojin, are either of you aware of any mono issues with FindFirstFile?

If umlauts are considered unusual, then I do have unusual characters in some subfolders of my home path, for example, Bücher/.

Just recently I learned that Apple’s APFS file system does not normalize Unicode characters anymore; that is, the same character (for example, ü) can have two different encodings. Can this be a problem here? (But then, I guess Duplicati would already have crashed when I selected those paths for backup.)

Not necessarily. I haven’t checked the code but it’s possible different API calls are being made for the different parts of the UI so that could expose different potential failure points.

I’m not sure about the umlauts - my non-Windows test VMs are not usable at the moment so I don’t have a good place to test, but hopefully some other users can step in and let us know if they have issues with umlauts on 2.0.3.3.

That is a bug in the Mono runtime. You can see that only the last line is in Duplicati, after that it goes into the Mono framework.
We cannot fix it in Duplicati because it actually crashes the Mono runtime, making it impossible to make any recovery.

My best guess is that it is a variant of this Mono bug:
https://bugzilla.xamarin.com/show_bug.cgi?id=60422

The AppleFS returns an unknown status code, which is not handled by Mono and then it crashes.

1 Like

Thank you @kenkendk for the analysis. It could well be a bug in Mono…

HOWEVER…

Today I started Duplicati, and everything works fine again.

The ONLY relevant change to the system that I did yesterday was a fix in my config.fish file (I am using Fish shell). A few of the environment variables were not visible to child processes, including the PATH variable!
(The wrong configuration was not obvious, and I advise anyone against using a non-POSIX shell. In the mid term, I plan to migrate from Fish to zsh which is POSIX compliant.)

This raises a question…

Does Duplicati or Mono call out to the shell when listing the source tree? If so, WHY?! I never would have mentally connected the Duplicati problem to the shell problem.

Duplicati does not, and I cannot imagine that Mono would do either. The crash was simply listing the contents of a folder, and it would surprise me if that invokes an external process.

This is strange as I am pretty sure that I have not done anything substantial on my machine except fixing my fish.config. Anyway, I guess we can safely assume for now that this is not a Duplicati bug but rather a possible bug in Mono caused by an issue with my shell configuration that had to get fixed anyway.

Thanks @kenkendk and @JonMikelV for looking into this. Who would have guessed that the cause for this issue is so far outside Duplicati…

Thanks for the follow-up.

It’s completely up to you, but if you’d be willing to test “un-fixing” you fish.config to see if the Duplicati issue returns I would feel more confident marking your previous post as the “solution”.

Either way, glad it’s working for you now!

I am in the midst of a long backup (running since days) but when it will have completed, I will do that test.

1 Like