Is there a background maintenance task or job that gets run by the Duplicati service?
I ask because I have two Linux machines (Fedora and Debian) that will after several weeks of perfect service, even with restarts of the machine during this time, suddenly cannot be connected to via the GUI but looking via SSH are busy running numerous Duplicati tasks. There is no backup scheduled during this time and none running that I can see as from the emailed logs the previously scheduled ones ran without incident. This is what I see.
I’d look at the logs on the machine but there aren’t any so I am at a loss to see what Duplicati is actually doing - can I enable file logging in some way so I can see what is going on when the GUI is not responding?
The issue does eventually get resolved, the machines are kept busy for many hours, sometimes a couple of days, then I will be alerted that the Duplicati service was restarted and all is well again, I check on the GUI and it’s as if nothing had happened.
It at least checks for updates (one forum post says once a week), but I’d expect that to be quite fast.
mono can likely do memory garbage collection when it likes, but all those mono-sgen look odd to me. Perhaps you could get a process tree, or at least run ps to sample parent PIDs to check their origins.
Other people here are on Linux more, but I’m used to seeing one mono-sgen even on busy Duplicati.
You could try using netstat or lsof to see what has port 8200. Also confirm you can telnet-connect. Duplicati doesn’t log routine web server traffic because it’s constantly polled when things are working. Packet capture such as tcpdump, wireshark, or tshark could find what’s going on, if you’re into that…
Duplicati.Server.exe has --log-file and --log-level options, if you want to make a big file to read. Generally this is redundant with job log, but you’re wondering what may be going on between job runs.
Thanks for the insight, interestingly for all my Linux installs the check for updates is failing:
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at /build/mono-6.12.0.200/external/boringssl/ssl/handshake_client.c:1132
at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <a8a996a78a804d888710c9e2575d78c8>:0
at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <a8a996a78a804d888710c9e2575d78c8>:0
at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <a8a996a78a804d888710c9e2575d78c8>:0
at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <a8a996a78a804d888710c9e2575d78c8>:0
--- End of inner exception stack trace ---
at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <a8a996a78a804d888710c9e2575d78c8>:0
at Mono.Net.Security.MonoTlsStream.CreateStream (System.Net.WebConnectionTunnel tunnel, System.Threading.CancellationToken cancellationToken) [0x0016a] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x001ba] in <a8a996a78a804d888710c9e2575d78c8>:0
--- End of inner exception stack trace ---
at System.Net.WebConnection.CreateStream (System.Net.WebOperation operation, System.Boolean reused, System.Threading.CancellationToken cancellationToken) [0x0021a] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebConnection.InitConnection (System.Net.WebOperation operation, System.Threading.CancellationToken cancellationToken) [0x00141] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebOperation.Run () [0x0009a] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebCompletionSource`1[T].WaitForCompletion () [0x00094] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebClient.DownloadBits (System.Net.WebRequest request, System.IO.Stream writeStream) [0x000e6] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00088] in <a8a996a78a804d888710c9e2575d78c8>:0
at System.Net.WebClient.DownloadFile (System.String address, System.String fileName) [0x00008] in <a8a996a78a804d888710c9e2575d78c8>:0
at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadFile(string,string)
at Duplicati.Library.AutoUpdater.UpdaterManager.CheckForUpdate (Duplicati.Library.AutoUpdater.ReleaseType channel) [0x000ee] in <283f74f2f6054493bec137d989f56043>:0
I’m running 2.0.7.2_canary_2023-05-25
I’ve enabled the logging by adding the options to service config so I will see if it can help the next time it happens, and also try those commands to get more info.
yes similar because htop show the threads, while top -H shows the same threads but with their names instead of the process path. I think that there are a lot of threads displayed because Duplicati creates more worker threads when there are more cores.
Obviously not as up-to-date as they could be even though I check every week. I fixed it for all the machines failing by running this and if necessary unticking the old X3 certificate mentioned in those posts:
dpkg-reconfigure ca-certificates
Then for good measure restarting Duplicati and checking for updates.
… and labels them PID, which ps uses for processes. Why Does htop Show More Processes Than ps.
So possibly these are threads of one process. FWIW pstree has nice displays with or without threads.
Mystery is still what started them all, and why they’re chewing up system resources.
Does this view help? The machine is just idle so nothing out of the ordinary running at the moment - no backups or anything. I can compare it later if it happens again
