Purge Broken Files Running for 36 Hours

Support
1

I had an error in a checksum and removed the offending dblock file, then running --purge-broken-files (via the GUI) as recommended in these forums. The process has been running for 36 hours now and there has been no log output at all (even on Explicit or Profiling). Running htop I see two mono-sgen Duplicati processes each at 95.1% and 94.5% with not much else as this box’s only function in life is backups. Please let me know if I can expect this to finish, I’d rather not lose the version history.

The destination storage is a CIFS mount to a file server on the same gigabit switch.

Duplicati version: 2.0.4.5_beta_2018-11-28
Source: 8.80 GB
Backup: 16.68 GB / 47 Versions
OS: Ubuntu 18
CPU: Core 2 Duo T9550 2.66GHz
RAM: 8GB

2

This is still running. At this point I think I just need to stop it and perhaps try a rebuild, what is the safest way out of this without (hopefully) losing my backup version history?

3

I have little opinion on the best way to stop in the middle of things. In theory, Duplicati uses database transactions to keep consistency even with interruptions, but I’m not sure how well that theory holds.

Your version history is in the destination files, so if database Recreate works, you should still have it. Keeping database copies as fixes proceed would be a good idea in case a different path gets worse.

Recreate can take awhile, but you ought to see some logs, e.g. at Information level. I’m not sure why there’s nothing now (Live log, right?). You could run an ls -lc or ls -lu on DB for signs of access.

4

Is “recreate” the same as “purge broken files” functionally, or maybe “recreate” is part of the process?

When I was poking around yesterday trying to get some kind of blip of output, I checked the data store and no files had been touched there. The sqlite database was also untouched. I killed the process today and the timestamp of the sqlite database changed to the time of the “kill” but is the same size. None of the dlist or dblock files have been updated since before I started the purge-broken-files run.

I started another purge-broken-files from the command line with the --verbose switch and there hasn’t been a peep for over two hours now, but the same 95%+ CPU usage.

I fired up an strace about 5 minutes ago and this is all it’s produced so far:

root@wopr:~/.config/Duplicati# strace -f -p 3238
strace: Process 3238 attached with 5 threads
strace: [ Process PID=3238 runs in x32 mode. ]
[pid 3246] futex(0x55f88635c900, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished …>
[pid 3243] futex(0x55f886175640, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff <unfinished …>
[pid 3239] futex(0x55f886184a08, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished …>
[pid 3253] restart_syscall(<… resuming interrupted poll …>

5

The precise function is not documented, but purge-broken-files code doesn’t obviously do any recreates. These are different functions anyway. Recreate is a database reconstruction done using destination files. –list-broken-files and –purge-broken-files sound like analysis and removals of broken files currently there.

I’m a bit surprised –verbose isn’t complaining. New log system with filters says it should be gone by now. There’s some information in that article on the effects --verbose used to have before the new log system.

strace not finding anything is really weird. That should trace every system call, right? Windows Duplicati generally comes in paired processes, where the parent is mostly to launch a child with the latest update. Mono is possibly different in process organization. There are usually two but I’m not sure if it’s equivalent.

6

This appears to be a problem only with the Linux version. I just copied everything over to a Windows VM and used Duplicati.CommandLine.exe. It was done in about 20 seconds. Something is wrong with the Mono/Linux version.

To expand: I did NOT copy the sqlite database. I copied the dblock/dlist store, did a rebuild, and then successfully purged the broken files. I just ran a full-remote-verification and all is well, at least with the data store.

7

… or with Mono. Sometimes a newer mono helps, but Error with Debian mono 5.18 #3620 gives me pause.
Duplicati does not start up #3572 is related. Yours, though, appeared to start, but then seemed to stall out… Perhaps you can make more sense of those, check your version, etc. I generally run Duplicati on Windows.

8

I’ve been running backups for many months on that system and haven’t noticed similar problems. I’ll see about a newer Mono version and whether that changes things (after I back up my backups). :wink:

I just did another copy of everything to the Windows VM and this time used the “old” database and again everything worked fine. It appears the bits in what is backed up are intact and it’s just certain operations on those bits that are choking.

Mono JIT compiler version 4.6.2 (Debian 4.6.2.7+dfsg-1ubuntu1)
Copyright © 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen

9

I added the repo and updated. I’m still seeing the same behavior. Obviously it could still be a Mono issue but updating to the latest stable version isn’t a fix at this time. I guess I have a big re-think about backups as a new project, at the very least how to wedge Windows into the process.

Mono JIT compiler version 5.18.0.240 (tarball Wed Jan 16 09:10:16 UTC 2019)
Copyright © 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(600)
Suspend: preemptive
GC: sgen (concurrent by default)

10

You could try using forum search or (maybe better) an external search engine to see if you can find any similar purge-broken-files stalled-running cases. I tried earlier (but only to see if it suffers from the high-resource-usage-to completion that Recreate suffers from). It seemed like it ought to be finishing faster.

Another possibly slow option is to use the Database option to find its path, rename it, and try Recreate. Coming back to debug --purge-broken-files further would then still be an option if Recreate won’t finish.

I don’t know how likely it could be that most of Duplicati would work, with this not working, but reinstalls sometimes help. Here, Windows install without prior uninstall can break. Duplicati files can be missing.

Mono has some tracing options, but it’s been a long time since I tried them. In addition to strace (which doesn’t show anything at all going on?!), you might have ltrace if you really want another view of things.

Duplicati code is, as far as I know, largely the same code on different OS’s, with some OS differences such as which way the slashes go, low-level things like file accessing, and a few OS-specific features.

11

Quick update, in case anyone else comes across this. The issue is not related to CIFS, it appears. From some searching, I thought there might be some Mono vs. CIFS bugginess but I just picked another one of the backups on the machine that goes to S3 (minio). While I can run a “Verify files” in the GUI and backups seem to be running as normal, if I attempt “test” with “–full-remote-verification=true” it just eats the CPU and never even connects to the minio server (watching for TCP connections there from the Duplicati host).

It’s the same behavior, nothing is logged still even with “log-level” cranked all the way up and “verbose” on.

Currently working on a migration plan away from Duplicati (which makes me sad, I love the concept and design). Thankfully it was still a test install and not anything important that isn’t backed up elsewhere.

I don’t think I was doing anything unexpected. Bog standard Ubuntu and then I attempted the latest stable Mono. If there’s a developer who wants more info, I’ll keep the install around for a bit. Let me know…

12

As I essentially have a sandbox of backups to play with now, I thought I’d experiment a bit more. Oddly, restores from the existing backups seem to be working fine. I would have thought that the full verification test and actual file restores would exercise similar or the same paths in the code. Maybe this is somehow a key to the issue? Why would restores work but full remote verifications just spin CPU and do nothing?

13

is a deeper version of the test command, but the sample is the same – one set unless you say otherwise.
–full-remote-verification describes in more detail what it does, and almost sounds like you need to set it to false manually if you don’t want it on the test command. I suppose you could try that to see if it helps any.

The test is not a restore, but tests the raw dblock, dindex, and dlist files that would be used for the restore. What they have in common is downloads of files, and I think even a restore download gets hash-checked. Forcing --full-remote-verification=false for a test might help things if somehow that was on, and breaking. Possibly restore doesn’t unzip the entire file, but only what it needs. I don’t know the code at such detail…

You could experiment with mono tracing. It’s in the man page. Maybe you can see where yours is spinning. Below is what my mono-sgen looked like when it’s just sitting waiting for web commands and other things. This output doesn’t redirect well, so it’s probably best to cut-and-paste the text from the screen scrollback.

HP4-VirtualBox / # ps -e | grep mono
 4453 pts/0    00:00:00 mono
 4457 pts/0    00:11:50 mono-sgen
HP4-VirtualBox / # kill -quit 4457
HP4-VirtualBox / # Full thread dump:

"Thread Pool I/O Selector"
"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Server.EventPollNotify.Wait (long,int) [0x00042] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTMethods.RequestInfo.LongPollCheck (Duplicati.Server.EventPollNotify,long&,bool&) [0x00115] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTMethods.ServerState.GET (string,Duplicati.Server.WebServer.RESTMethods.RequestInfo) [0x0000c] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo,string,string,string) [0x00140] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (HttpServer.IHttpRequest,HttpServer.IHttpResponse,HttpServer.Sessions.IHttpSession,string,string,string) [0x00009] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTHandler.Process (HttpServer.IHttpRequest,HttpServer.IHttpResponse,HttpServer.Sessions.IHttpSession) [0x000b6] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at HttpServer.HttpServer.HandleRequest (HttpServer.IHttpClientContext,HttpServer.IHttpRequest,HttpServer.IHttpResponse,HttpServer.Sessions.IHttpSession) [0x0003e] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpServer.ProcessRequest (HttpServer.IHttpClientContext,HttpServer.IHttpRequest) [0x00121] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpServer.ProcessRequestWrapper (HttpServer.IHttpClientContext,HttpServer.IHttpRequest) [0x0000b] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpServer.OnRequest (object,HttpServer.RequestEventArgs) [0x0004b] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) System.EventHandler`1<TEventArgs_REF>.invoke_void_object_TEventArgs (object,TEventArgs_REF) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.HttpListenerBase.OnRequestReceived (object,HttpServer.RequestEventArgs) [0x00000] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) System.EventHandler`1<TEventArgs_REF>.invoke_void_object_TEventArgs (object,TEventArgs_REF) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.HttpContextFactory.OnRequestReceived (object,HttpServer.RequestEventArgs) [0x00000] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) System.EventHandler`1<TEventArgs_REF>.invoke_void_object_TEventArgs (object,TEventArgs_REF) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.HttpClientContext.OnRequestCompleted (object,System.EventArgs) [0x00052] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) <Module>.invoke_void_object_EventArgs (object,System.EventArgs) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.Parser.HttpRequestParser.OnRequestCompleted () [0x00000] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.Parser.HttpRequestParser.Parse (byte[],int,int) [0x00175] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpClientContext.OnReceive (System.IAsyncResult) [0x00059] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <b3d41b23de534128a4f18a6e1312f79c>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0001e] in <2943701620b54f86b436d3ffad010412>:0

"Thread Pool Worker"
"<threadpool thread>"
"<threadpool thread>"
"TrayIcon Longpoll Thread"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne () [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Library.Utility.AsyncHttpRequest/AsyncWrapper.GetResponseOrStream () [0x00000] in <621233224d6648a0a1cd0bb505473fc4>:0
  at Duplicati.Library.Utility.AsyncHttpRequest.GetResponse () [0x00044] in <621233224d6648a0a1cd0bb505473fc4>:0
  at Duplicati.GUI.TrayIcon.HttpServerConnection.PerformRequestInternal<T_REF> (string,string,System.Collections.Generic.Dictionary`2<string, string>) [0x001c5] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at Duplicati.GUI.TrayIcon.HttpServerConnection.PerformRequest<T_REF> (string,string,System.Collections.Generic.Dictionary`2<string, string>) [0x0000c] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at Duplicati.GUI.TrayIcon.HttpServerConnection.UpdateStatus () [0x00001] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at Duplicati.GUI.TrayIcon.HttpServerConnection.LongPollRunner () [0x00005] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0004d] in <2943701620b54f86b436d3ffad010412>:0

"<unnamed thread>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.RequestQueue.QueueThread () [0x00005] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0004d] in <2943701620b54f86b436d3ffad010412>:0

"TrayIcon Request Thread"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (System.TimeSpan,bool) [0x0002c] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.GUI.TrayIcon.HttpServerConnection.ThreadRunner () [0x000bc] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0004d] in <2943701620b54f86b436d3ffad010412>:0

"TaskScheduler"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Server.Scheduler.Runner () [0x0056f] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0004d] in <2943701620b54f86b436d3ffad010412>:0

"Finalizer"
"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Server.EventPollNotify.Wait (long,int) [0x00042] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTMethods.RequestInfo.LongPollCheck (Duplicati.Server.EventPollNotify,long&,bool&) [0x00115] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTMethods.ServerState.GET (string,Duplicati.Server.WebServer.RESTMethods.RequestInfo) [0x0000c] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo,string,string,string) [0x00140] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (HttpServer.IHttpRequest,HttpServer.IHttpResponse,HttpServer.Sessions.IHttpSession,string,string,string) [0x00009] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.Server.WebServer.RESTHandler.Process (HttpServer.IHttpRequest,HttpServer.IHttpResponse,HttpServer.Sessions.IHttpSession) [0x000b6] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at HttpServer.HttpServer.HandleRequest (HttpServer.IHttpClientContext,HttpServer.IHttpRequest,HttpServer.IHttpResponse,HttpServer.Sessions.IHttpSession) [0x0003e] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpServer.ProcessRequest (HttpServer.IHttpClientContext,HttpServer.IHttpRequest) [0x00121] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpServer.ProcessRequestWrapper (HttpServer.IHttpClientContext,HttpServer.IHttpRequest) [0x0000b] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpServer.OnRequest (object,HttpServer.RequestEventArgs) [0x0004b] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) System.EventHandler`1<TEventArgs_REF>.invoke_void_object_TEventArgs (object,TEventArgs_REF) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.HttpListenerBase.OnRequestReceived (object,HttpServer.RequestEventArgs) [0x00000] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) System.EventHandler`1<TEventArgs_REF>.invoke_void_object_TEventArgs (object,TEventArgs_REF) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.HttpContextFactory.OnRequestReceived (object,HttpServer.RequestEventArgs) [0x00000] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) System.EventHandler`1<TEventArgs_REF>.invoke_void_object_TEventArgs (object,TEventArgs_REF) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.HttpClientContext.OnRequestCompleted (object,System.EventArgs) [0x00052] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at (wrapper delegate-invoke) <Module>.invoke_void_object_EventArgs (object,System.EventArgs) [0x00070] in <2943701620b54f86b436d3ffad010412>:0
  at HttpServer.Parser.HttpRequestParser.OnRequestCompleted () [0x00000] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.Parser.HttpRequestParser.Parse (byte[],int,int) [0x00175] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at HttpServer.HttpClientContext.OnReceive (System.IAsyncResult) [0x00059] in <bed89f1655ee48029f6d6812f54c58ad>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <b3d41b23de534128a4f18a6e1312f79c>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0001e] in <2943701620b54f86b436d3ffad010412>:0

"WorkerThread<IRunnerData>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne () [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Library.Utility.WorkerThread`1<Tx_REF>.Runner () [0x0007e] in <621233224d6648a0a1cd0bb505473fc4>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0004d] in <2943701620b54f86b436d3ffad010412>:0

"UpdatePollThread"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (System.TimeSpan,bool) [0x0002c] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Server.UpdatePollThread.Run () [0x00347] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0004d] in <2943701620b54f86b436d3ffad010412>:0

...

"<unnamed thread>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x0000b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.WaitHandle.WaitOne () [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Server.Program.RealMain (string[]) [0x0071a] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int_object (object,intptr,intptr,intptr) [0x00054] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) [0x0000c] in <2943701620b54f86b436d3ffad010412>:0
  at System.Reflection.MonoMethod.Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo) [0x0003b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Reflection.MethodBase.Invoke (object,object[]) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.RunMethod (System.Reflection.MethodInfo,string[]) [0x00000] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager/<>c__DisplayClass58_1.<RunFromMostRecentSpawn>b__0 () [0x00000] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.WrapWithUpdater (Duplicati.Library.AutoUpdater.AutoUpdateStrategy,System.Action) [0x00157] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.RunFromMostRecentSpawn (System.Reflection.MethodInfo,string[],Duplicati.Library.AutoUpdater.AutoUpdateStrategy) [0x0004b] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.RunFromMostRecent (System.Reflection.MethodInfo,string[],Duplicati.Library.AutoUpdater.AutoUpdateStrategy) [0x0001b] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Server.Program.Main (string[]) [0x00014] in <fe28905ee30b422e8d475f1cfdb85515>:0
  at Duplicati.GUI.TrayIcon.HostedInstanceKeeper/<>c__DisplayClass5_0.<.ctor>b__0 (object) [0x0006c] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <2943701620b54f86b436d3ffad010412>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0004f] in <2943701620b54f86b436d3ffad010412>:0

"<unnamed thread>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) Gtk.Application.gtk_main () [0x00002] in <17f6f103d9a54c8c880a641a2f9bcb57>:0
  at Gtk.Application.Run () [0x00000] in <17f6f103d9a54c8c880a641a2f9bcb57>:0
  at Duplicati.GUI.TrayIcon.AppIndicatorRunner.Run (string[]) [0x0001f] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at Duplicati.GUI.TrayIcon.TrayIconBase.Init (string[]) [0x0003f] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at Duplicati.GUI.TrayIcon.GtkRunner.Init (string[]) [0x00008] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at Duplicati.GUI.TrayIcon.Program.RealMain (string[]) [0x00572] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x0004e] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) [0x0000c] in <2943701620b54f86b436d3ffad010412>:0
  at System.Reflection.MonoMethod.Invoke (object,System.Reflection.BindingFlags,System.Reflection.Binder,object[],System.Globalization.CultureInfo) [0x0003b] in <2943701620b54f86b436d3ffad010412>:0
  at System.Reflection.MethodBase.Invoke (object,object[]) [0x00000] in <2943701620b54f86b436d3ffad010412>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.RunMethod (System.Reflection.MethodInfo,string[]) [0x00000] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager/<>c__DisplayClass58_1.<RunFromMostRecentSpawn>b__0 () [0x00000] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.WrapWithUpdater (Duplicati.Library.AutoUpdater.AutoUpdateStrategy,System.Action) [0x00157] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.RunFromMostRecentSpawn (System.Reflection.MethodInfo,string[],Duplicati.Library.AutoUpdater.AutoUpdateStrategy) [0x0004b] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.Library.AutoUpdater.UpdaterManager.RunFromMostRecent (System.Reflection.MethodInfo,string[],Duplicati.Library.AutoUpdater.AutoUpdateStrategy) [0x0001b] in <3227b98b904042faac2116282e7fbdfd>:0
  at Duplicati.GUI.TrayIcon.Program.Main (string[]) [0x0001c] in <a5f36c71f1df46eb8251c7281146fa4b>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int_object (object,intptr,intptr,intptr) [0x00054] in <a5f36c71f1df46eb8251c7281146fa4b>:0

...

HP4-VirtualBox / #