I’m still unable to reproduce the issue by myself, unfortunately. I’m also not used to running duplicati in a docker-image. May someone give me a simple command to start a docker container that backups a local folder to storj, where I only need to set my own values for the folder, access grant and bucket?
Did you recently update duplicati? Do you know which version did work? Might you be able to get back to the previous version to see if it is working again?
I still did not yet get the error. I have my own WebAPI-project using uplink.NET which crashes randomly when running in a docker container on Ubuntu (24.04 and 20.04). But I fail to get the root cause. All of my demos and trials work. Even using an older uplink-version leads to the same error, so I cannot tell if some update in-between introduced the error.
I still dont get any further. Can someone try to get a core-dump for this issue? I am unable to get one.
I tried now with older versions, I tried different code-paths, I tried extensive logging - but I cannot fetch the error directly. My container crashes silently and restarts but does not generate any dump or exception.
I am also running Duplicati in a container under linux, with a large number of files for backup, using Storj, and experiencing an issue with the container terminating without generation of a dump file. My container is running as root.
I’ve tested a number of things with no noticeable benefit. Things I’ve tried are:
Increasing the memory limit of the container to 32G.
Running the container using the --oom-kill-disable argument
Unfortunately, I see in my system logs that my kernel does not support oom-kill-disable, so that argument is ignored.
Maybe someone else can use this argument on a system that supports that option?
I am using the following command to start my docker instance
Mapping the host dump file location to same path in the container
The dump file location is determined with the following command on the host:
cat /proc/sys/kernel/core_pattern
Running the following command on both the host and within the container to insure core dumps are enabled
ulimit -c unlimited
I have found that the container always terminates while enumerating files for backup.
The last message logged by duplicati before terminating is always: 2025-02-11 00:35:17 +00 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: /data/
The full path for the last message varies.
Last note of possible value. I have found that the duplicati image with the following tag works consistently. However, it isn’t compatible with the database structure in the 2.1.x.x images, so you can’t easily roll back. 2.0.8.0_experimental_2024-04-19
I’d be very interested in seeing results from someone who can run the container with the oom-kill-disable flag.
Yes, there is also a 2.0.8.1_beta_2024_05_07 which is the last version to use Mono, and Storj works there. Both Storj and Duplicati has upgrades, so not sure what is causing what.
I still do not have any real core dump or stack trace, unfortunately. I could overcome my issue locally by extracting the “Download”-Logic into en EXE that I call using “new Process()”. This is totally overkill as it spins up a whole go-runtime and initialises everything. But interestingly enough this is quite fast - and fast enough for me at the moment. But for me it resolves the issue - so it seems it’s happening in the download-part of uplink.Net.
May someone with the error try setting this env variable and see if it changes anything?
GODEBUG=asyncpreemptoff=1
Don’t know if that does help but maybe it is worth a try.
Hi,
Just wanted to report the very same issue. I have a backup setup which is about 450GB and the destination is storj. I have never experienced issues until I upgraded the docker container a couple of weeks ago (running 2.1.0.4_stable now).
Unfortunately, whenever it starts the backup job, it goes on for a while and then crashes and restarts - in a perpetual loop. This is the from the log:
Server has started and is listening on port 8200
Connection to localhost (::1) 8200 port [tcp/*] succeeded!
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HNALR9CV12O8", Request id "0HNALR9CV12O8:00000001": An unhandled exception was thrown by the application.
System.NullReferenceException: Object reference not set to an instance of an object.
at Duplicati.Server.Scheduler.get_WorkerQueue()
at Duplicati.Server.Scheduler.GetSchedulerQueueIds()
at WebserverCore.Services.SchedulerService.GetSchedulerQueueIds()
at Duplicati.WebserverCore.Services.StatusService.GetStatus()
at Duplicati.WebserverCore.Notifications.WebsocketAccessor.SendInitialStatus(WebSocket connection)
at Duplicati.WebserverCore.Notifications.WebsocketAccessor.AddConnection(WebSocket newConnection)
at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
at Duplicati.WebserverCore.Middlewares.StaticFilesExtensions.<>c__DisplayClass5_0.<<UseDefaultStaticFiles>b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.<Invoke>g__AwaitMatcher|10_0(EndpointRoutingMiddleware middleware, HttpContext httpContext, Task`1 matcherTask)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Duplicati.Server.Scheduler.get_WorkerQueue()
at Duplicati.Server.Scheduler.GetSchedulerQueueIds()
at WebserverCore.Services.SchedulerService.GetSchedulerQueueIds()
at Duplicati.WebserverCore.Services.StatusService.GetStatus()
at Duplicati.WebserverCore.Notifications.WebsocketAccessor.<.ctor>b__4_0(Object _, EventArgs _)
at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
at System.Threading.QueueUserWorkItemCallback.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Connection to localhost (::1) 8200 port [tcp/*] succeeded!
Server has started and is listening on port 8200
The Duplicati container is running on an old 20-core Xeon Dell PowerEdge with 78.5GB of RAM running OpenMediaVault 7.7.0-2.
Unfortunately, it’s still not working. The asyncpreemptoff env variable didn’t do anything. Eventually I deleted my backups and re-created them. But as soon as the backup job starts, the container just dies. Now I don’t even see any errors in the logs anymore.
Thank @Tom_DeBom for investigating! So it seems that the switch to .Net8/9 (not using Mono anymore) leads to that issue. That does not mean - unfortunately - that I do know what the issue might be.
It does not seem to be uplink.NET from the binary-side as the uplink-Nuget seem to not be different between 2.0.8.1 and the current master. Am I right @kenkendk?
So maybe there changed something with PInvoke between .Net 4/Monot and .Net8/9. I will try to find something in that direction.
At least this fits in my observations. I have an app that is not yet on .Net8/9 and it does not have that issue. My current problemativ app instead is also using .Net8/9 - so this might be the next part to investigate.
uplink.NET 2.12.3328 uses Go 1.21.4 and uplink-c 1.8.1
uplink.NET 2.13.3484 uses Go 1.22.5 and uplink-c 1.9.0
So there is definitely a binary change in there. It should be easy to test this: just switch back to 2.12.3328 on the latest Duplicati and see if that helps. I would assume it does not change anything (but would be happy to know it does).
Then I would stll assume an issue with the way P/Invoke changed between Mono and .Net5+.