Suddenly started crashing - mono-sgen pthread_mutex_lock.c

Support
1

I’ve been running duplicati-2.0.6.3-2.0.6.3_beta_20210617.noarch on Fedora 35 for at least a month now. It’s been backup up on schedule to a host on my LAN using SFTP. Then about a week ago, the jobs stopped completing. Duplicati itself would crash a little way into the job running. I don’t know what’s useful, so I’ll paste entries from /var/log/messages.

Any help in fixing this will be greatly appreciated.

Aug  9 20:35:42 fedora systemd[1]: Started duplicati.service - Duplicati web-server.
Aug  9 20:35:42 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=duplicati comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Aug  9 20:59:29 fedora duplicati-server[259594]: mono-sgen: pthread_mutex_lock.c:94: ___pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
Aug  9 20:59:29 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:29 fedora duplicati-server[259594]: #011Native Crash Reporting
Aug  9 20:59:29 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:29 fedora duplicati-server[259594]: Got a SIGABRT while executing native code. This usually indicates
Aug  9 20:59:29 fedora duplicati-server[259594]: a fatal error in the mono runtime or one of the native libraries
Aug  9 20:59:29 fedora duplicati-server[259594]: used by your application.
Aug  9 20:59:29 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:29 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:29 fedora duplicati-server[259594]: #011Native stacktrace:
Aug  9 20:59:29 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x55a74d099028 - /usr/bin/mono-sgen :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x55a74d04451e - /usr/bin/mono-sgen :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x55a74d09121c - /usr/bin/mono-sgen :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6e3ea70 - /lib64/libc.so.6 :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6e8ec4c - /lib64/libc.so.6 :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6e3e9c6 - /lib64/libc.so.6 : raise
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6e287f4 - /lib64/libc.so.6 : abort
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6e2871b - /lib64/libc.so.6 :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6e37576 - /lib64/libc.so.6 :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6e902b0 - /lib64/libc.so.6 : pthread_mutex_lock
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c71c6736 - /lib64/libnss_sss.so.2 :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c71c71fe - /lib64/libnss_sss.so.2 :
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c71c9036 - /lib64/libnss_sss.so.2 : _nss_sss_getpwuid_r
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93c6edc631 - /lib64/libc.so.6 : getpwuid_r
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x7f93bd79b035 - /lib64/libMonoPosixHelper.so : Mono_Posix_Syscall_getpwuid_r
Aug  9 20:59:29 fedora duplicati-server[259594]: #0110x4246a939 - Unknown
Aug  9 20:59:29 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:29 fedora duplicati-server[259594]: #011Telemetry Dumper:
Aug  9 20:59:29 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:29 fedora duplicati-server[259594]: Pkilling 0x140272281318976x from 0x140271233791552x
[ deleting a whole pile of these ]
Aug  9 20:59:29 fedora duplicati-server[259594]: Pkilling 0x140272495220288x from 0x140271233791552x
Aug  9 20:59:29 fedora duplicati-server[259594]: Entering thread summarizer pause from 0x140271233791552x
Aug  9 20:59:29 fedora duplicati-server[259594]: Finished thread summarizer pause from 0x140271233791552x.
Aug  9 20:59:29 fedora duplicati-server[259594]: Failed to create breadcrumb file (null)/crash_hash_0xfffda88dc3ecd76f
Aug  9 20:59:29 fedora duplicati-server[259594]: Waiting for dumping threads to resume
Aug  9 20:59:30 fedora duplicati-server[265988]: =================================================================
Aug  9 20:59:30 fedora duplicati-server[265988]: #011External Debugger Dump:
Aug  9 20:59:30 fedora duplicati-server[265988]: =================================================================
Aug  9 20:59:30 fedora duplicati-server[265988]: /usr/bin/gdb: warning: Couldn't determine a path for the index cache directory.
Aug  9 20:59:31 fedora duplicati-server[265988]: [New LWP 259595]
[ Delete another pile ]
Aug  9 20:59:31 fedora duplicati-server[265988]: [New LWP 265927]
Aug  9 20:59:31 fedora duplicati-server[265988]: warning: File "/usr/bin/mono-sgen-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
Aug  9 20:59:31 fedora duplicati-server[265988]: To enable execution of this file add
Aug  9 20:59:31 fedora duplicati-server[265988]: #011add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py
Aug  9 20:59:31 fedora duplicati-server[265988]: line to your configuration file "$HOME/.gdbinit".
Aug  9 20:59:31 fedora duplicati-server[265988]: To completely disable this security protection add
Aug  9 20:59:31 fedora duplicati-server[265988]: #011set auto-load safe-path /
Aug  9 20:59:31 fedora duplicati-server[265988]: line to your configuration file "$HOME/.gdbinit".
Aug  9 20:59:31 fedora duplicati-server[265988]: For more information about this security protection see the
Aug  9 20:59:31 fedora duplicati-server[265988]: "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
Aug  9 20:59:31 fedora duplicati-server[265988]: #011info "(gdb)Auto-loading safe path"
Aug  9 20:59:31 fedora duplicati-server[265988]: [Thread debugging using libthread_db enabled]
Aug  9 20:59:31 fedora duplicati-server[265988]: Using host libthread_db library "/lib64/libthread_db.so.1".
Aug  9 20:59:31 fedora duplicati-server[265988]: 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  Id   Target Id                                            Frame
Aug  9 20:59:31 fedora duplicati-server[265988]: * 1    Thread 0x7f93c7089040 (LWP 259594) "mono-sgen"       0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  2    Thread 0x7f93bfbff640 (LWP 259595) "SGen worker"     0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  3    Thread 0x7f93bddff640 (LWP 259596) "Finalizer"       0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  4    Thread 0x7f93bc5fe640 (LWP 259599) "Timer-Scheduler" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  5    Thread 0x7f93bc3fd640 (LWP 259600) "UpdatePollThrea" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  6    Thread 0x7f93affff640 (LWP 259601) "WorkerThread<IR" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  7    Thread 0x7f93afdfe640 (LWP 259602) "TaskScheduler"   0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  8    Thread 0x7f93af7ff640 (LWP 259603) "mono-sgen"       0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  9    Thread 0x7f93af3ff640 (LWP 259604) "Thread Pool I/O" 0x00007f93c6f0588f in poll () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  10   Thread 0x7f93ae9fa640 (LWP 259648) "Thread Pool Wor" 0x00007f93c6f0588f in poll () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  11   Thread 0x7f93ae3f7640 (LWP 259651) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  12   Thread 0x7f93ae1f6640 (LWP 259795) "Backend Async W" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  13   Thread 0x7f9372a5d640 (LWP 259819) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  14   Thread 0x7f937265b640 (LWP 259846) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  15   Thread 0x7f934bbff640 (LWP 259884) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  16   Thread 0x7f9371a55640 (LWP 259898) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  17   Thread 0x7f934b7fd640 (LWP 261945) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  18   Thread 0x7f93710ff640 (LWP 261946) "Thread Pool Wor" 0x00007f93c6edcbdf in wait4 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  19   Thread 0x7f9372259640 (LWP 262747) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  20   Thread 0x7f937285c640 (LWP 263261) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  21   Thread 0x7f93aedfc640 (LWP 263643) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  22   Thread 0x7f93ae5f8640 (LWP 263645) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  23   Thread 0x7f9371854640 (LWP 264105) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  24   Thread 0x7f93aeffd640 (LWP 264210) "Thread Pool Wor" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]:  25   Thread 0x7f93af1fe640 (LWP 265927) "Thread Pool I/O" 0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 25 (Thread 0x7f93af1fe640 (LWP 265927) "Thread Pool I/O"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e8c504 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d302dfe in mono_os_cond_timedwait ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d30c6c4 in sleep_interruptable.lto_priv ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x000055a74d29f7b7 in monitor_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #6  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #7  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 24 (Thread 0x7f93aeffd640 (LWP 264210) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 23 (Thread 0x7f9371854640 (LWP 264105) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 22 (Thread 0x7f93ae5f8640 (LWP 263645) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 21 (Thread 0x7f93aedfc640 (LWP 263643) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e8c504 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d302dfe in mono_os_cond_timedwait ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d32cfc5 in mono_w32handle_timedwait_signal_handle.constprop ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x000055a74d260ccb in mono_w32handle_wait_one ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x000055a74d324482 in mono_w32handle_wait_multiple.constprop ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #6  0x000055a74d1d2c4f in ves_icall_System_Threading_WaitHandle_Wait_internal_raw ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #7  0x0000000041ffad42 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #8  0x00007f93bfe030d8 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #9  0x00007f93bfe323c0 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #10 0x0000000000000000 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 20 (Thread 0x7f937285c640 (LWP 263261) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 19 (Thread 0x7f9372259640 (LWP 262747) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 18 (Thread 0x7f93710ff640 (LWP 261946) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6edcbdf in wait4 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x000055a74d0992ab in dump_native_stacktrace.lto_priv ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d04451e in mono_handle_native_crash ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d09121c in sigabrt_signal_handler ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  <signal handler called>
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6e8ec4c in __pthread_kill_implementation () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #6  0x00007f93c6e3e9c6 in raise () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #7  0x00007f93c6e287f4 in abort () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #8  0x00007f93c6e2871b in __assert_fail_base.cold () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #9  0x00007f93c6e37576 in __assert_fail () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #10 0x00007f93c6e902b0 in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #11 0x00007f93c71c6736 in sss_nss_mc_get_ctx () from /lib64/libnss_sss.so.2
Aug  9 20:59:31 fedora duplicati-server[265988]: #12 0x00007f93c71c71fe in sss_nss_mc_getpwuid () from /lib64/libnss_sss.so.2
Aug  9 20:59:31 fedora duplicati-server[265988]: #13 0x00007f93c71c9036 in _nss_sss_getpwuid_r () from /lib64/libnss_sss.so.2
Aug  9 20:59:31 fedora duplicati-server[265988]: #14 0x00007f93c6edc631 in getpwuid_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #15 0x00007f93bd79b035 in Mono_Posix_Syscall_getpwuid_r () from /lib64/libMonoPosixHelper.so
Aug  9 20:59:31 fedora duplicati-server[265988]: #16 0x000000004246a939 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #17 0x00007f93c6de8310 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #18 0x00007f93bfd71358 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #19 0x00000000000003e8 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #20 0x00007f93bfd75008 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #21 0x00007f93710fd0f8 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #22 0x00007f93bfd71690 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #23 0x00007f93bfd71440 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #24 0x00007f93710fe7f8 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #25 0x00007f93710fd1c0 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #26 0x00007f93710fd020 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #27 0x0000000000000000 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 17 (Thread 0x7f934b7fd640 (LWP 261945) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 16 (Thread 0x7f9371a55640 (LWP 259898) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 15 (Thread 0x7f934bbff640 (LWP 259884) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 14 (Thread 0x7f937265b640 (LWP 259846) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 13 (Thread 0x7f9372a5d640 (LWP 259819) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 12 (Thread 0x7f93ae1f6640 (LWP 259795) "Backend Async W"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e8c210 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d302e5d in mono_os_cond_timedwait ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d32cfc5 in mono_w32handle_timedwait_signal_handle.constprop ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x000055a74d260af7 in mono_w32handle_wait_one ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x000055a74d324482 in mono_w32handle_wait_multiple.constprop ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #6  0x000055a74d1d2c4f in ves_icall_System_Threading_WaitHandle_Wait_internal_raw ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #7  0x0000000041ffad42 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #8  0x00007f93bfffb620 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #9  0x00007f93ac7ab950 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #10 0x0000000000000000 in ?? ()
Aug  9 20:59:31 fedora duplicati-server[265988]: Thread 11 (Thread 0x7f93ae3f7640 (LWP 259651) "Thread Pool Wor"):
Aug  9 20:59:31 fedora duplicati-server[265988]: #0  0x00007f93c6e89a19 in __futex_abstimed_wait_common () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #1  0x00007f93c6e94a3b in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #2  0x000055a74d29e973 in worker_thread ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #3  0x000055a74d24b5a8 in start_wrapper ()
Aug  9 20:59:31 fedora duplicati-server[265988]: #4  0x00007f93c6e8ce2d in start_thread () from /lib64/libc.so.6
Aug  9 20:59:31 fedora duplicati-server[265988]: #5  0x00007f93c6f121b0 in clone3 () from /lib64/libc.so.6
[ Deleting many lines of logs because my post is too large ] 
=================================================================
Aug  9 20:59:31 fedora duplicati-server[259594]: #011Basic Fault Address Reporting
Aug  9 20:59:31 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:31 fedora duplicati-server[259594]: Memory around native instruction pointer (0x7f93c6e8ec4c):0x7f93c6e8ec3c  04 00 44 89 e2 89 de 89 c7 b8 ea 00 00 00 0f 05  ..D.............
Aug  9 20:59:31 fedora duplicati-server[259594]: 0x7f93c6e8ec4c  89 c5 f7 dd 3d 00 f0 ff ff b8 00 00 00 00 0f 47  ....=..........G
Aug  9 20:59:31 fedora duplicati-server[259594]: 0x7f93c6e8ec5c  c5 e9 70 ff ff ff 66 0f 1f 44 00 00 4c 89 ef e8  ..p...f..D..L...
Aug  9 20:59:31 fedora duplicati-server[259594]: 0x7f93c6e8ec6c  10 b0 ff ff e9 44 ff ff ff 0f 1f 00 4c 89 ef e8  .....D......L...
Aug  9 20:59:31 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:31 fedora duplicati-server[259594]: #011Managed Stacktrace:
Aug  9 20:59:31 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at <unknown> <0xffffffff>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.Native.Syscall:sys_getpwuid_r <0x000a8>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.Native.Syscall:getpwuid_r <0x00077>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.UnixUserInfo:.ctor <0x00087>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.UnixFileSystemInfo:get_OwnerUser <0x0006b>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at FileInfo:.ctor <0x000b3>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at UnixSupport.File:GetUserGroupAndPermissions <0x00057>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Duplicati.Library.Common.IO.SystemIOLinux:GetMetadata <0x001eb>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Duplicati.Library.Snapshots.NoSnapshotLinux:GetMetadata <0x0005a>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Duplicati.Library.Main.Operation.Backup.MetadataGenerator:GenerateMetadata <0x00121>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at <<Run>b__0>d:MoveNext <0x01803>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at MoveNextRunner:InvokeMoveNext <0x0009e>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.ExecutionContext:RunInternal <0x001f9>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.ExecutionContext:Run <0x00047>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at MoveNextRunner:Run <0x00103>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.AwaitTaskContinuation:RunOrScheduleAction <0x00086>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.Task:FinishContinuations <0x0017b>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.Task:FinishStageThree <0x00087>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.Task`1:TrySetResult <0x000f3>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:SetResult <0x000c7>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at <ReadAsync>d__30:MoveNext <0x018cf>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at MoveNextRunner:InvokeMoveNext <0x0009e>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.ExecutionContext:RunInternal <0x001f9>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.ExecutionContext:Run <0x00047>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at MoveNextRunner:Run <0x00103>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.AwaitTaskContinuation:RunOrScheduleAction <0x00086>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.Task:FinishContinuations <0x0017b>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.Task:FinishStageThree <0x00087>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.Task`1:TrySetResult <0x000f3>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.TaskCompletionSource`1:TrySetResult <0x0003f>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.Tasks.TaskCompletionSource`1:SetResult <0x00033>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at <>c__DisplayClass49_0:<SetResult>b__0 <0x00047>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at <>c__DisplayClass0_0:<QueueItem>b__0 <0x00032>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.QueueUserWorkItemCallback:WaitCallback_Context <0x00063>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.ExecutionContext:RunInternal <0x001f9>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.ExecutionContext:Run <0x00047>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.QueueUserWorkItemCallback:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem <0x00067>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading.ThreadPoolWorkQueue:Dispatch <0x002f8>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at System.Threading._ThreadPoolWaitCallback:PerformWaitCallback <0x00063>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at <Module>:runtime_invoke_bool <0x00086>
Aug  9 20:59:31 fedora duplicati-server[259594]: =================================================================
Aug  9 20:59:33 fedora systemd[1]: duplicati.service: Main process exited, code=exited, status=134/n/a
Aug  9 20:59:33 fedora systemd[1]: duplicati.service: Failed with result 'exit-code'.
Aug  9 20:59:33 fedora audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=duplicati comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Aug  9 20:59:33 fedora systemd[1]: duplicati.service: Consumed 2min 36.460s CPU time.
2

To add a bit more information:

  • I start the duplicati server, and the running job continues
  • It may run for a few seconds, but typically completes counting the files (there are 180,000 of them) and then proceeds to backup.
  • It will run for anything from 5 minutes to 3 hours before it crashes.
  • It’s never on the same file or folder.
  • It’s always the same error, with a 134 exit code at the end.
  • I’ve downgraded glibc, and upgraded it.
  • I don’t have an alternative for mono-sgen available.

I’m really at a loss!!

3

the pthread message, or the backtrace:

Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.UnixFileSystemInfo:get_OwnerUser <0x0006b>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at FileInfo:.ctor <0x000b3>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at UnixSupport.File:GetUserGroupAndPermissions <0x00057>

it matters, because pthread error message is meaningless, however getting always security related backtraces could be significant.

4

Thanks for your response.

When checking /var/log/messages, I get those three lines repeatedly, exactly as shown. However, I notice that when I run it directly from the command line - i.e. not starting it via systemctl - that when it crashes I don’t get the above three lines. Maybe it’s not occurring every time it crashes, or else systemctl starting it has something to do with it.

I’m not sure how to interpret those values, but hex 6b = decimal 107 = user “qemu”.

Duplicati is running as “root” though, so I can’t see why security should matter.

As a test I’ve exported and imported the backup job as a brand new job, into a new empty structure. So it’s now running the backup (200 GB). Let’s see how far it gets. Perhaps Duplicati’s old backup files have been corrupted somehow? (One of the firtst things I done was rebuild the database - but that didn’t help.)

5

because of something called SeLinux maybe ?

6

I know that pain in the neck well! But I run my system in Permissive mode, so it only logs but doesn’t stop anything.

My fresh backup ran for an hour, and then crashed in the same way. I had started it with systemctl, and I see the errors you spotted.

The odd thing is it was all working smoothly in July. It seems the last successful backup had been on 30 July. Later that day, I updated patches - this is what was applied, but I can’t see anthing that looks like it could cause issues:

    Install  boost-json-1.76.0-5.fc35.x86_64                         @updates
    Upgrade  boost-1.76.0-5.fc35.x86_64                              @updates
    Upgrade  boost-atomic-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-chrono-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-container-1.76.0-5.fc35.x86_64                    @updates
    Upgrade  boost-context-1.76.0-5.fc35.x86_64                      @updates
    Upgrade  boost-contract-1.76.0-5.fc35.x86_64                     @updates
    Upgrade  boost-coroutine-1.76.0-5.fc35.x86_64                    @updates
    Upgrade  boost-date-time-1.76.0-5.fc35.x86_64                    @updates
    Upgrade  boost-fiber-1.76.0-5.fc35.x86_64                        @updates
    Upgrade  boost-filesystem-1.76.0-5.fc35.x86_64                   @updates
    Upgrade  boost-graph-1.76.0-5.fc35.x86_64                        @updates
    Upgrade  boost-iostreams-1.76.0-5.fc35.x86_64                    @updates
    Upgrade  boost-locale-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-log-1.76.0-5.fc35.x86_64                          @updates
    Upgrade  boost-math-1.76.0-5.fc35.x86_64                         @updates
    Upgrade  boost-nowide-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-numpy3-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-program-options-1.76.0-5.fc35.x86_64              @updates
    Upgrade  boost-python3-1.76.0-5.fc35.x86_64                      @updates
    Upgrade  boost-random-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-regex-1.76.0-5.fc35.x86_64                        @updates
    Upgrade  boost-serialization-1.76.0-5.fc35.x86_64                @updates
    Upgrade  boost-stacktrace-1.76.0-5.fc35.x86_64                   @updates
    Upgrade  boost-system-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-test-1.76.0-5.fc35.x86_64                         @updates
    Upgrade  boost-thread-1.76.0-5.fc35.x86_64                       @updates
    Upgrade  boost-timer-1.76.0-5.fc35.x86_64                        @updates
    Upgrade  boost-type_erasure-1.76.0-5.fc35.x86_64                 @updates
    Upgrade  boost-wave-1.76.0-5.fc35.x86_64                         @updates
    Upgrade  chromium-103.0.5060.114-1.fc35.x86_64                   @updates
    Upgrade  chromium-common-103.0.5060.114-1.fc35.x86_64            @updates
    Upgrade  composer-2.3.10-1.fc35.noarch                           @updates
    Upgrade  fetchmail-6.4.31-1.fc35.x86_64                          @updates
    Upgrade  git-2.37.1-1.fc35.x86_64                                @updates
    Upgrade  git-core-2.37.1-1.fc35.x86_64                           @updates
    Upgrade  git-core-doc-2.37.1-1.fc35.noarch                       @updates
    Upgrade  gnupg1-1.4.23-18.fc35.x86_64                            @updates
    Upgrade  gsmartcontrol-1.1.4-1.fc35.x86_64                       @updates
    Upgrade  hplip-3.22.6-1.fc35.x86_64                              @updates
    Upgrade  hplip-common-3.22.6-1.fc35.x86_64                       @updates
    Upgrade  hplip-libs-3.22.6-1.fc35.x86_64                         @updates
    Upgrade  ibus-typing-booster-2.18.0-1.fc35.noarch                @updates
    Upgrade  inih-56-1.fc35.i686                                     @updates
    Upgrade  inih-56-1.fc35.x86_64                                   @updates
    Upgrade  koji-1.29.1-1.fc35.noarch                               @updates
    Upgrade  libffado-2.4.6-1.fc35.x86_64                            @updates
    Upgrade  libidn2-2.3.3-1.fc35.i686                               @updates
    Upgrade  libidn2-2.3.3-1.fc35.x86_64                             @updates
    Upgrade  libsane-hpaio-3.22.6-1.fc35.x86_64                      @updates
    Upgrade  lightdm-1.32.0-1.fc35.x86_64                            @updates
    Upgrade  lightdm-gobject-1.32.0-1.fc35.x86_64                    @updates
    Upgrade  lightdm-qt5-1.32.0-1.fc35.x86_64                        @updates
    Upgrade  mutter-41.8-2.fc35.x86_64                               @updates
    Upgrade  perl-Git-2.37.1-1.fc35.noarch                           @updates
    Upgrade  perl-Image-ExifTool-12.42-1.fc35.noarch                 @updates
    Upgrade  perl-Module-CoreList-1:5.20220720-1.fc35.noarch         @updates
    Upgrade  perl-Module-CoreList-tools-1:5.20220720-1.fc35.noarch   @updates
    Upgrade  pipewire-0.3.56-1.fc35.i686                             @updates
    Upgrade  pipewire-0.3.56-1.fc35.x86_64                           @updates
    Upgrade  pipewire-alsa-0.3.56-1.fc35.i686                        @updates
    Upgrade  pipewire-alsa-0.3.56-1.fc35.x86_64                      @updates
    Upgrade  pipewire-gstreamer-0.3.56-1.fc35.x86_64                 @updates
    Upgrade  pipewire-jack-audio-connection-kit-0.3.56-1.fc35.x86_64 @updates
    Upgrade  pipewire-libs-0.3.56-1.fc35.i686                        @updates
    Upgrade  pipewire-libs-0.3.56-1.fc35.x86_64                      @updates
    Upgrade  pipewire-pulseaudio-0.3.56-1.fc35.x86_64                @updates
    Upgrade  python3-koji-1.29.1-1.fc35.noarch                       @updates
    Upgrade  python3-rpm-4.17.1-2.fc35.x86_64                        @updates
    Upgrade  rpm-4.17.1-2.fc35.x86_64                                @updates
    Upgrade  rpm-build-4.17.1-2.fc35.x86_64                          @updates
    Upgrade  rpm-build-libs-4.17.1-2.fc35.x86_64                     @updates
    Upgrade  rpm-libs-4.17.1-2.fc35.x86_64                           @updates
    Upgrade  rpm-plugin-selinux-4.17.1-2.fc35.x86_64                 @updates
    Upgrade  rpm-plugin-systemd-inhibit-4.17.1-2.fc35.x86_64         @updates
    Upgrade  rpm-sign-libs-4.17.1-2.fc35.x86_64                      @updates
    Upgrade  sane-backends-1.1.1-6.fc35.x86_64                       @updates
    Upgrade  sane-backends-daemon-1.1.1-6.fc35.x86_64                @updates
    Upgrade  sane-backends-doc-1.1.1-6.fc35.noarch                   @updates
    Upgrade  sane-backends-drivers-cameras-1.1.1-6.fc35.i686         @updates
    Upgrade  sane-backends-drivers-cameras-1.1.1-6.fc35.x86_64       @updates
    Upgrade  sane-backends-drivers-scanners-1.1.1-6.fc35.i686        @updates
    Upgrade  sane-backends-drivers-scanners-1.1.1-6.fc35.x86_64      @updates
    Upgrade  sane-backends-libs-1.1.1-6.fc35.i686                    @updates
    Upgrade  sane-backends-libs-1.1.1-6.fc35.x86_64                  @updates
    Upgrade  wireplumber-0.4.11-2.fc35.x86_64                        @updates
    Upgrade  wireplumber-libs-0.4.11-2.fc35.x86_64                   @updates
    Upgrade  xorg-x11-server-Xephyr-1.20.14-7.fc35.x86_64            @updates
    Upgrade  xorg-x11-server-Xorg-1.20.14-7.fc35.x86_64              @updates
    Upgrade  xorg-x11-server-Xvfb-1.20.14-7.fc35.x86_64              @updates
    Upgrade  xorg-x11-server-Xwayland-21.1.4-2.fc35.x86_64           @updates
    Upgrade  xorg-x11-server-common-1.20.14-7.fc35.x86_64            @updates
7

I don’t see any obvious culprit in your patches either.
2 points:

  • do you use containers (stuff like lxc) ? I could imagine how trying to access files belonging to a container could lead to problems if the software tries to lookup the container file user id in the host passwd file.

  • maybe you could indeed:

To enable execution of this file add
#011add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py
line to your configuration file "$HOME/.gdbinit".

if all else fails, try to debug :slight_smile:

8

No, I don’t use containers, just a KVM virtual machine.

Initially I checked, but don’t have a .gdbinit file. I’ll create one and add the settings this evening. I need to get some work done now!

9

Sadly, I don’t know what I’m doing here. I don’t know gdb at all.

“/root/.config/gdb” doesn’t exist, so I added “set auto-load safe-path /” to a brand new gdb folder and file “/root/.config/gdb/gdbinit”. I also hard linked that to “/root/.gdbinit”, but it still gives the same error message.

A strange behaviour is as follows:

  • I have THREE jobs defined. “Virtual Machines” backs up all my VMs and their associated definition files on a weekly basis. Daily backups up my Documents and Downloads and mail spool file on a daily basis. Lastly Bulk backs up everything else I need to be able to recover, excluding the sections backed up by the other two, and this is also run weekly.
  • If I run “Virtual Machines” it seems to succeed every time.
  • If I run Daily or Bulk, it crashes at a random time into the backup. By random, I mean 10 seconds to 3 hours.

Are my two large jobs failing because of too many files?

10

sorry I had done a copypasta of your error message and reading it again I don’t understand why there is a #011 at the beginning of the instruction. Should not be IMO. So your /root/.gdbinit should be something like
add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py.
However I think that to have this file you should install mono-devel. Wait, on my test install I have installed mono-devel and I don’t have it. Ah yes, should install mono-dbg.

About your backup, do you use LVM ? IIRC it’s not used with Fedora but I may be wrong.

11

Yes, I do use LVM for my OS files, and /root. My /home folder is on another disk, and is not LVM.

There isn’t a mono-dbg package. Here is what I have:

# dnf list mono\*
Last metadata expiration check: 2:16:03 ago on Mon 15 Aug 2022 03:45:00 PM SAST.
Installed Packages
mono-addins.x86_64                         1.3.3-1.fc36                          @fedora
mono-complete.x86_64                       6.12.0-6.fc36                         @fedora
mono-core.x86_64                           6.12.0-6.fc36                         @fedora
mono-data.x86_64                           6.12.0-6.fc36                         @fedora
mono-data-oracle.x86_64                    6.12.0-6.fc36                         @fedora
mono-data-sqlite.x86_64                    6.12.0-6.fc36                         @fedora
mono-devel.x86_64                          6.12.0-6.fc36                         @fedora
mono-extras.x86_64                         6.12.0-6.fc36                         @fedora
mono-locale-extras.x86_64                  6.12.0-6.fc36                         @fedora
mono-mvc.x86_64                            6.12.0-6.fc36                         @fedora
mono-reactive.x86_64                       6.12.0-6.fc36                         @fedora
mono-wcf.x86_64                            6.12.0-6.fc36                         @fedora
mono-web.x86_64                            6.12.0-6.fc36                         @fedora
mono-winforms.x86_64                       6.12.0-6.fc36                         @fedora
mono-winfx.x86_64                          6.12.0-6.fc36                         @fedora
monodoc.x86_64                             6.12.0-6.fc36                         @fedora
Available Packages
mono-addins-devel.i686                     1.3.3-1.fc36                          fedora 
mono-addins-devel.x86_64                   1.3.3-1.fc36                          fedora 
mono-basic.x86_64                          4.7-9.fc36                            fedora 
mono-basic-devel.i686                      4.7-9.fc36                            fedora 
mono-basic-devel.x86_64                    4.7-9.fc36                            fedora 
mono-bouncycastle.x86_64                   1.8.10-2.fc36                         fedora 
mono-cecil.x86_64                          0.10.4-6.fc36                         fedora 
mono-cecil-devel.i686                      0.10.4-6.fc36                         fedora 
mono-cecil-devel.x86_64                    0.10.4-6.fc36                         fedora 
mono-cecil-flowanalysis.x86_64             0.1-0.40.20110512svn100264.fc36       fedora 
mono-cecil-flowanalysis-devel.i686         0.1-0.40.20110512svn100264.fc36       fedora 
mono-cecil-flowanalysis-devel.x86_64       0.1-0.40.20110512svn100264.fc36       fedora 
mono-core.i686                             6.12.0-6.fc36                         fedora 
mono-devel.i686                            6.12.0-6.fc36                         fedora 
mono-mvc-devel.i686                        6.12.0-6.fc36                         fedora 
mono-mvc-devel.x86_64                      6.12.0-6.fc36                         fedora 
mono-reactive-devel.i686                   6.12.0-6.fc36                         fedora 
mono-reactive-devel.x86_64                 6.12.0-6.fc36                         fedora 
mono-reactive-winforms.x86_64              6.12.0-6.fc36                         fedora 
mono-reflection.i686                       0.1-0.25.20110613git304d1d.fc36       fedora 
mono-reflection.x86_64                     0.1-0.25.20110613git304d1d.fc36       fedora 
mono-reflection-devel.i686                 0.1-0.25.20110613git304d1d.fc36       fedora 
mono-reflection-devel.x86_64               0.1-0.25.20110613git304d1d.fc36       fedora 
mono-tools.x86_64                          4.2-25.fc36                           fedora 
mono-tools-devel.i686                      4.2-25.fc36                           fedora 
mono-tools-devel.x86_64                    4.2-25.fc36                           fedora 
mono-tools-gendarme.x86_64                 4.2-25.fc36                           fedora 
mono-tools-monodoc.x86_64                  4.2-25.fc36                           fedora 
mono-web-devel.i686                        6.12.0-6.fc36                         fedora 
mono-web-devel.x86_64                      6.12.0-6.fc36                         fedora 
mono-zeroconf.x86_64                       0.9.0-33.fc36                         fedora 
mono-zeroconf-devel.i686                   0.9.0-33.fc36                         fedora 
mono-zeroconf-devel.x86_64                 0.9.0-33.fc36                         fedora 
monobristol.x86_64                         0.60.3.1-21.fc36                      fedora 
monochrome-icon-theme.noarch               16.10-13.20180421bzr625.fc36          fedora 
monocypher.i686                            3.1.2-3.fc36                          fedora 
monocypher.x86_64                          3.1.2-3.fc36                          fedora 
monocypher-devel.i686                      3.1.2-3.fc36                          fedora 
monocypher-devel.x86_64                    3.1.2-3.fc36                          fedora 
monodevelop.x86_64                         5.10.0-22.fc36                        fedora 
monodevelop-debugger-gdb.x86_64            5.0.1-11.fc36                         fedora 
monodevelop-devel.i686                     5.10.0-22.fc36                        fedora 
monodevelop-devel.x86_64                   5.10.0-22.fc36                        fedora 
monodoc-devel.i686                         6.12.0-6.fc36                         fedora 
monodoc-devel.x86_64                       6.12.0-6.fc36                         fedora 
monosim.x86_64                             1.5.2-25.fc36                         fedora 
monotone.x86_64                            1.1-42.fc36                           fedora 
monotone-server.x86_64                     1.1-42.fc36                           fedora 
[root@fedora ~]#
12

Mono 6.12.0 Release Notes says that’s a November 2020 version. Maybe try a newer download?
Although I couldn’t find a relevant mono project issue, one never knows. Technically it died in libc.
getpwuid_r might be trying to fetch names (reverse-lookup from number) for the owner and group.
Looking at a sample block of metadata from a Linux dblock file, its format does store such names.

Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.Native.Syscall:sys_getpwuid_r <0x000a8>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.Native.Syscall:getpwuid_r <0x00077>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.UnixUserInfo:.ctor <0x00087>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Mono.Unix.UnixFileSystemInfo:get_OwnerUser <0x0006b>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at FileInfo:.ctor <0x000b3>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at UnixSupport.File:GetUserGroupAndPermissions <0x00057>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Duplicati.Library.Common.IO.SystemIOLinux:GetMetadata <0x001eb>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Duplicati.Library.Snapshots.NoSnapshotLinux:GetMetadata <0x0005a>
Aug  9 20:59:31 fedora duplicati-server[259594]: #011  at Duplicati.Library.Main.Operation.Backup.MetadataGenerator:GenerateMetadata <0x00121>

I don’t know why a lookup would fail though. Perhaps ltrace could confirm the call where it aborts.

That’s too bad, otherwise I’d wonder if it found a file that killed it, and ask for narrowing area down.
There’s still a chance it’s file-specific. You could run a verbose log-file to see how random death is.

Channel Pipeline tries to explain internals. Source code has GenerateMetadata uses in these files

https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/Operation/Backup/FilePreFilterProcess.cs
https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/Operation/Backup/MetadataPreProcess.cs

So there are some debug ideas to try to find why mono is dying. Of course it’s not supposed to die…

Aug 9 20:59:33 fedora systemd[1]: duplicati.service: Main process exited, code=exited, status=134/n/a

From Google search, 134 may be what mono does when it dies this way. That’s not a Duplicati code.

13

I think I am having a closely related problem. My counting files never gets past a couple of thousand, then it hangs with connecting to server error. It aborts, and recycles the process in an endless loop.

Reading your thread prompted me to check journalctl. I found this line at the beginning of each loop iteration:
mono-sgen: pthread_mutex_lock

I also just came across this in the log, not sure if it is meaningful or not:
abrt-server[21904]: Blacklisted package ‘mono-core’

14

Seems packages don’t have the same name as for Ubuntu :slight_smile:
Do you have a /usr/bin/mono-sgen-gdb.py file or not ?

re: LVM I was thinking that snapshot could cause problem, but it’s logical that your backup is running in non snapshot mode since your backup set include a non LVM device.

Do you use LDAP ? I don’t quite see how an access to the passwd file could cause random problems un less some weird security stuff such as SeLinux could be involved (in your place I’d try to check sestatus anyway), but if there is a network access involved, it could be a reason.

About the problem appearing after an update, it could be that the update forced a reboot, and this reboot triggered some previous change to be effectively applied. Wild ideas I know but trying to debug an intermittent problem remotely can be challenging. It’s challenging enough when dealing with it directly.

15

The randomness of it all makes me think it’s read errors on your source. If nothing else actually changed, that’s my bet.

Can you try to backup/copy you’re important data using another method, if you can, chance are your source is fine and there is a problem with Duplicati, if not then you know where you’re problem lies.

16

So I’ve tried a few things:
@ts678 suggested uprading mono, which I did, but that made no difference. I then ran the backup logging to a file with verbose as level. I ran the job twice, and the first time it got further than the second time. It just stops. No error recorded on the log itself - only the crash in /var/log/messages. The 134 code seems to show only when running as a service, not when running from CLI, so I think it’s a systemd exit code.
@gpatel-fr I do have /usr/bin/mono-sgen-gdb.py - what do I do with that? I don’t use LDAP. I have a standalone Linux workstation that I have a lot of data on. I run Duplicati on it, and backup to an old PC with Fedora 36 installed and a 12 TB drive - I use SFTP/SSH as connection method.

To try to eliminate some options, I did the following test: I used the Fedora ISO to create a new KVM VM on Fedora 36. I didn’t patch it. I installed Duplicati. I mounted the three folders I back up with SSHFS onto the VM, so they appear as part of the local file system. I then backed that up to the same destination server successfully, four times. So I believe the source to 100%. My latest experiment is I renamed the job’s database files, and renamed the folder on the destination. So now I am effectively running the job as if it were a first time backup. It’s going to take some time, then I will report back. If it succeeds it means my Duplicati backup (destination) files somehow got corrupted.

17

Half an hour ago, the process crashed. It had been running for 5 hours. Again, this is a different file to before.

What I did notice in LOOKING at /var/log/messages instead of just grepping it, is that it actually wrote a systemd-coredump for the mono-gen process. I have an 18 MB file in /var/lib/systemd/coredump. Would that be any use to anyone in the thread?

(It in fact wrote a number of them over the last two weeks.)

Alternatively, can someone guide me in what to do to analyze the problem? I don’t mind doing the grunt work, I just don’t know what to do.

18

My problems started after I did a dnf upgrade. It updated a few dozen packages that I have installed.

By any chance did your problems also start after upgrading? If so, maybe we can compare lists of what packages were affected.

Thanks

19

as I understand it, when the application crashes it would allow to use gdb but I’d admit that if it does not start automatically I’m not sure how to force it :-/
about the crash dump, you are supposed to open them with gdb, so it’s actually useful to know it. Basically you learn the executable name by using ‘file core’, and then ‘gdb executable_name core’. After that you are in gdb.

Edit: thinking more about it, using gdb would be productive only for a specific class of users - mostly developers. Outside of that, it would not get more info that a crash report such as the one you showed in your first post. Only thing I would have liked a complete trace. Why ? because it could possibly have exposed a badly managed access problem between threads. As some threads are not shown, it’s not possible to say. If you have still the complete trace, you could take a look if the getpwuid appears in 2 threads. In this case it could show a problem in Duplicati - or Mono. Not very likely but who knows.

20

List was posted earlier.