Backup hangs indefinitely

ThomasChr · May 21, 2025, 6:27am

Hi there!

This is the second time I’m trying Duplicati after 9 months and I still got the same error.
But this time I want to figure out what is wrong!

I need to backup about 600 GB into an SFTP storage. The backup runs fine for a few days but then it just stops. Neither the left file or the left GB display changes. Only the KB/s are getting lower and lower.
Is there anything I can do to hunt this possible bug down?
I’m a programmer and I’m willing to do whatever is necessary, because I really want to like Duplicati!

Thanks for any help,

Thomas

PS: 2.1.0.5_stable_2025-03-04 in Windows 11

ThomasChr · May 21, 2025, 8:16am

These are the threads atm:

ThomasChr · May 21, 2025, 9:09am

And here are the stack traces

Thread (0x134CC):
    [Native Frames]
    Avalonia.Win32!Avalonia.Win32.Win32DispatcherImpl.RunLoop(value class System.Threading.CancellationToken)
    Avalonia.Base!Avalonia.Threading.DispatcherFrame.Run(class Avalonia.Threading.IControlledDispatcherImpl)
    Avalonia.Base!Avalonia.Threading.Dispatcher.PushFrame(class Avalonia.Threading.DispatcherFrame)
    Avalonia.Base!Avalonia.Threading.Dispatcher.MainLoop(value class System.Threading.CancellationToken)
    Avalonia.Controls!Avalonia.Controls.ApplicationLifetimes.ClassicDesktopStyleApplicationLifetime.StartCore(class System.String[])
    Avalonia.Controls!Avalonia.Controls.ApplicationLifetimes.ClassicDesktopStyleApplicationLifetime.Start(class System.String[])
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.AvaloniaRunner.Run(class System.String[])
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.TrayIconBase.Init(class System.String[])
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.AvaloniaRunner.Init(class System.String[])
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.Program.StartTray(class System.String[],class System.Collections.Generic.Dictionary`2<class System.String,class System.String>,class Duplicati.GUI.TrayIcon.HostedInstanceKeeper,class System.String,class System.String)
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.Program.Main(class System.String[])
    Duplicati.GUI.TrayIcon!Duplicati.GUI.TrayIcon.Net8.Program+<>c__DisplayClass0_0.<Main>b__0()
    Duplicati.Library.Crashlog!Duplicati.Library.Crashlog.CrashlogHelper.WrapWithCrashLog(class System.Func`1<!!0>,class System.String,class System.String)
    Duplicati.GUI.TrayIcon!Duplicati.GUI.TrayIcon.Net8.Program.Main(class System.String[])

Thread (0x4BC8):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    Duplicati.Server.Implementation!Duplicati.Server.Program.Main(class System.String[])
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.HostedInstanceKeeper+<>c__DisplayClass5_0.<.ctor>b__0(class System.Object)

Thread (0x13558):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    System.Private.CoreLib.il!System.Threading.TimerQueue.TimerThread()

Thread (0x10D18):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+IOCompletionPoller.Poll()

Thread (0x12590):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.Monitor.Wait(class System.Object,int32)
    Microsoft.Extensions.Logging.Console.il!Microsoft.Extensions.Logging.Console.ConsoleLoggerProcessor.TryDequeue(value class Microsoft.Extensions.Logging.Console.LogMessageEntry&)
    Microsoft.Extensions.Logging.Console.il!Microsoft.Extensions.Logging.Console.ConsoleLoggerProcessor.ProcessLogQueue()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x29B4):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
    System.Private.CoreLib.il!System.Threading.ManualResetEventSlim.Wait(value class System.TimeSpan)
    Microsoft.AspNetCore.Server.Kestrel.Core.il!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x8114):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+GateThread.GateThreadStart()

Thread (0xF628):
    [Native Frames]
    Microsoft.Win32.SystemEvents!Interop+User32.GetMessageW(value class MSG&,int,int32,int32)
    Microsoft.Win32.SystemEvents!Microsoft.Win32.SystemEvents.WindowThreadProc()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x13444):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    Duplicati.Library.RestAPI!Duplicati.Server.UpdatePollThread.Run()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x13364):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
    System.Private.CoreLib.il!System.Threading.Tasks.Task.SpinThenBlockingWait(int32,value class System.Threading.CancellationToken)
    System.Private.CoreLib.il!System.Threading.Tasks.Task.InternalWaitCore(int32,value class System.Threading.CancellationToken)
    System.Private.CoreLib.il!System.Threading.Tasks.Task.Wait(int32,value class System.Threading.CancellationToken)
    System.Private.CoreLib.il!System.Threading.Tasks.Task.Wait()
    CoCoL!CoCoL.ChannelExtensions.WaitForTask(class System.Threading.Tasks.Task)
    CoCoL!CoCoL.ChannelExtensions.WaitForTaskOrThrow(class System.Threading.Tasks.Task)
    Duplicati.Library.Main!Duplicati.Library.Main.Operation.BackupHandler.Run(class System.String[],class Duplicati.Library.Utility.IFilter,value class System.Threading.CancellationToken)
    Duplicati.Library.Main!Duplicati.Library.Main.Controller+<>c__DisplayClass17_0.<Backup>b__0(class Duplicati.Library.Main.BackupResults)
    Duplicati.Library.Main!Duplicati.Library.Main.Controller.RunAction(!!0,class System.String[]&,class Duplicati.Library.Utility.IFilter&,class System.Action`1<!!0>)
    Duplicati.Library.Main!Duplicati.Library.Main.Controller.Backup(class System.String[],class Duplicati.Library.Utility.IFilter)
    Duplicati.Library.RestAPI!Duplicati.Server.Runner.Run(class IRunnerData,bool)
    Duplicati.Server.Implementation!Duplicati.Server.Program+<>c.<SetWorkerThread>b__62_0(class IRunnerData)
    Duplicati.Library.Utility!Duplicati.Library.Utility.WorkerThread`1[System.__Canon].Runner()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x132DC):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOne(int32)
    Duplicati.Library.RestAPI!Duplicati.Server.Scheduler.Runner()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x12538):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.HttpServerConnection.ThreadRunner()

Thread (0x12970):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
    System.Private.CoreLib.il!System.Threading.Tasks.Task.SpinThenBlockingWait(int32,value class System.Threading.CancellationToken)
    System.Private.CoreLib.il!System.Threading.Tasks.Task.InternalWaitCore(int32,value class System.Threading.CancellationToken)
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.HttpServerConnection.PerformRequestInternal(class System.String,class System.String,class System.String,value class System.Nullable`1<value class System.TimeSpan>)
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.HttpServerConnection.PerformRequest(class System.String,class System.String,class System.String,value class System.Nullable`1<value class System.TimeSpan>)
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.HttpServerConnection.UpdateStatus(bool)
    Duplicati.GUI.TrayIcon.Implementation!Duplicati.GUI.TrayIcon.HttpServerConnection.LongPollRunner()

Thread (0x10260):
    [Native Frames]
    Avalonia.Win32!Avalonia.Win32.WinRT.Composition.WinUiCompositorConnection.RunLoop()
    Avalonia.Win32!Avalonia.Win32.WinRT.Composition.WinUiCompositorConnection+<>c__DisplayClass7_0.<TryCreateAndRegisterCore>b__0()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x12CC0):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitAnyMultiple(value class System.ReadOnlySpan`1<class Microsoft.Win32.SafeHandles.SafeWaitHandle>,int32)
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+WaitThread.WaitThreadStart()

Thread (0xA420):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    Duplicati.Library.Utility!Duplicati.Library.Utility.BlockingQueue`1[System.__Canon].Dequeue()
    Duplicati.Library.Main!Duplicati.Library.Main.BackendManager.ThreadRun()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunInternal(class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)

Thread (0x586C):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int32)
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.Wait(int32,bool)
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Thread (0x13930):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int32)
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.Wait(int32,bool)
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Thread (0x14628):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int32)
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.Wait(int32,bool)
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Thread (0x15318):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.WaitHandle.WaitOneNoCheck(int32)
    Duplicati.Library.RestAPI!Duplicati.Server.EventPollNotify.Wait(int64,int32)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Endpoints.V1.ServerState.Execute(value class System.Nullable`1<int64>,bool,class System.String,class Duplicati.WebserverCore.Abstractions.IStatusService,class Duplicati.Server.EventPollNotify)
    Anonymously Hosted DynamicMethods Assembly!dynamicClass.lambda_method265(pMT: 00007FFB5CA2E4A0,pMT: 00007FFB5CBB9860)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Middlewares.HostnameFilter+<InvokeAsync>d__2.MoveNext()
    System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(class Microsoft.AspNetCore.Http.EndpointFilterInvocationContext,class Microsoft.AspNetCore.Http.EndpointFilterDelegate)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Middlewares.LanguageFilter+<InvokeAsync>d__0.MoveNext()
    System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(class Microsoft.AspNetCore.Http.EndpointFilterInvocationContext,class Microsoft.AspNetCore.Http.EndpointFilterDelegate)
    Anonymously Hosted DynamicMethods Assembly!dynamicClass.lambda_method266(pMT: 00007FFB5CA2E4A0,class System.Object,pMT: 00007FFB5C9054A8)
    Microsoft.AspNetCore.Routing.il!Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(class Microsoft.AspNetCore.Http.HttpContext)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Middlewares.WebsocketExtensions+<>c__DisplayClass0_0+<<UseNotifications>b__0>d.MoveNext()
    System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
    Microsoft.AspNetCore.WebSockets.il!Microsoft.AspNetCore.WebSockets.WebSocketMiddleware.Invoke(class Microsoft.AspNetCore.Http.HttpContext)
    Microsoft.AspNetCore.Diagnostics.il!Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.Invoke(class Microsoft.AspNetCore.Http.HttpContext)
    Microsoft.AspNetCore.StaticFiles.il!Microsoft.AspNetCore.StaticFiles.DefaultFilesMiddleware.Invoke(class Microsoft.AspNetCore.Http.HttpContext)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Middlewares.StaticFilesExtensions+<>c__DisplayClass5_0+<<UseDefaultStaticFiles>b__1>d.MoveNext()
    System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
    Duplicati.WebserverCore!Duplicati.WebserverCore.Middlewares.StaticFilesExtensions+<>c__DisplayClass5_0.<UseDefaultStaticFiles>b__1(class Microsoft.AspNetCore.Http.HttpContext,class System.Func`1<class System.Threading.Tasks.Task>)
    Microsoft.AspNetCore.Authorization.Policy.il!Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__11.MoveNext()
    System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
    Microsoft.AspNetCore.Authorization.Policy.il!Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(class Microsoft.AspNetCore.Http.HttpContext)
    Microsoft.AspNetCore.Authentication.il!Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext()
    System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
    Microsoft.AspNetCore.Authentication.il!Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(class Microsoft.AspNetCore.Http.HttpContext)
    Microsoft.AspNetCore.HostFiltering.il!Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware.Invoke(class Microsoft.AspNetCore.Http.HttpContext)
    Microsoft.AspNetCore.Server.Kestrel.Core.il!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__238`1[System.__Canon].MoveNext()
    System.Private.CoreLib.il!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(class System.Threading.Thread,class System.Threading.ExecutionContext,class System.Threading.ContextCallback,class System.Object)
    System.Private.CoreLib.il!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__238`1[System.__Canon]].MoveNext(class System.Threading.Thread)
    System.Private.CoreLib.il!System.Threading.ThreadPoolWorkQueue.Dispatch()
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Thread (0xCB14):
    [Native Frames]
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.WaitForSignal(int32)
    System.Private.CoreLib.il!System.Threading.LowLevelLifoSemaphore.Wait(int32,bool)
    System.Private.CoreLib.il!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

ThomasChr · May 21, 2025, 11:30am

Last Event according to the log:

Procmon only captures FileSystem Request to the Duplicati-server.sqlite File.

ThomasChr · May 22, 2025, 6:08am

I had to restart the PC. After counting the files it is now at:

1819684 files (398.66 GB) to go at 590.92 KB/s

before the restart it was at:

1461093 files (318.00 GB) to go at 625.64 KB/s

And it stopped at 0% Progress at file ‘C:\Users\tchristlieb\AppData\Local\JetBrains\Fleet\download-api\downloads\4abe3eb0279a6044903c69d0ee6b4b759893ec35\fleet-jcef-windows-x64.zip’

Let’s see what will happen next. I’ll keep this thread updated.

ThomasChr · May 22, 2025, 11:26am

Still going!

1238184 files (284.48 GB) to go at 921.67 KB/s

ThomasChr · May 24, 2025, 4:09pm

And it went through. Backup is done. Finally.
It did only hang this one time.
I now try some more (incrementals) and then try recovering.

kenkendk · May 31, 2025, 2:44pm

Awesome level of detail!

I could not see any obvious issue, so it might be something deeper with the way tasks are being scheduled. Since they are not threads, they do not have a regular stack.

What I see are the UI parts (Avalonia, Duplicati.GUI.TrayIcon, HttpServerConnection), the Kestrel webserver parts, the queue and runner, and Duplicati.Library.Main.Operation.BackupHandler.Run which is the cut-off point with synchronous waiting on asynchronous operations.

I would expect all of the threads to be waiting, so there is nothing there I can see would cause issues.

I have not heard widespread reported hangs, so lets hope it was just a fluke

ThomasChr · May 31, 2025, 4:20pm

Thanks @kenkendk !
After the initial hang - which has been resolved by a restart - it works without any problems.
I‘m at 8 Versions now and also restoring worked flawlessly.

I‘m thinking about a PR adding support for “backup when drive gets connected” - let‘s see how that will work out