Duplicati crashing when starting backups

[DataBitz]

I have Duplicati (2.1.0.2_beta_2024-11-29) running on Unraid (i.e. Docker) with backups which go to Google drive and Storj.
Backups to Google drive are working fine.
Backups to Storj, when started cause Duplicati to fail completely.
When I start a backup it shows “verifying backend data” then “completing previous backup” and shortly after this the webui shows “connection to server lost” and the following is shown in the logs.
Not sure where to start to troubleshoot this. TIA.

Server has started and is listening on port 8200
Inside getter
Connection to localhost (127.0.0.1) 8200 port [tcp/*] succeeded!
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
Server has started and is listening on port 8200

[kenkendk]

Hi @DataBitz, welcome to the forum.

@DataBitz

Backups to Storj, when started cause Duplicati to fail completely.
When I start a backup it shows “verifying backend data” then “completing previous backup” and shortly after this the webui shows “connection to server lost” and the following is shown in the logs.

Is this a backup that used to work with a previous version of Duplicati?

From the description it sounds like Duplicati crashes completely and restarts.
I would think this is caused by some interaction with the Storj library, but without any additional logging, it is hard to know.

Do you have any indication that the Docker instance restarted? Can you see logs from the end of the previous session?

Looking at the code for Storj, it could be that this is caused by a timeout issue. Can you try to set the option --read-write-timeout=1h?

This gives 1 hour before the connection times out (default is 30s).

@DataBitz

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup

This is a logged message that should not be there, caused by a legacy API decision. It does not have any effect on the backup.

[funkysloth]

Not sure if this helps to narrow things: I am facing a similar behavior (maybe exactly the same) as mentioned here Something wrong with web UI - #3 by kenkendk

Fun fact is that also my remote backups are on Storj. However, they have been working fine for more than one year with daily backup operations.

I am still in favor of thinking that Duplicati upgrade might be the root cause (since we are sharing the timing at which this issue popped up) but it might also be related on some changes that Storj could have been done, even though I am not aware of anything.

[DataBitz]

Thanks @kenkendk
Yes last backups to Storj completed back in August, these backups have been in place for almost 2 years. I did not have Duplicati set to auto start so it has not actually been running probably between around August and this week when I restarted it.
Do you mean --http-readwrite-timeout=1h? Doesn’t seem to make any difference.

Here are the logs after starting the job and having the web ui stop straight after each. Did it twice to show the pattern of logs. These are from the docker terminal.
Uptime on the docker container is 28 hours.

Connection to localhost (127.0.0.1) 8200 port [tcp/*] succeeded!
Server has started and is listening on port 8200
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
Inside getter
Connection to localhost (127.0.0.1) 8200 port [tcp/*] succeeded!
Server has started and is listening on port 8200
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
Inside getter
Connection to localhost (127.0.0.1) 8200 port [tcp/*] succeeded!
Server has started and is listening on port 8200
Inside getter
Connection to localhost (127.0.0.1) 8200 port [tcp/*] succeeded!
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.NullReferenceException: Object reference not set to an instance of an object.
         at Duplicati.Server.Scheduler.get_WorkerQueue()
         at Duplicati.Server.Scheduler.GetSchedulerQueueIds()
         at WebserverCore.Services.SchedulerService.GetSchedulerQueueIds()
         at Duplicati.WebserverCore.Services.StatusService.GetStatus()
         at Duplicati.WebserverCore.Notifications.WebsocketAccessor.SendInitialStatus(WebSocket connection)
         at Duplicati.WebserverCore.Notifications.WebsocketAccessor.AddConnection(WebSocket newConnection)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
warn: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[2]
      The response has already started, the error handler will not be executed.
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HN8U8CAVMIHI", Request id "0HN8U8CAVMIHI:00000001": An unhandled exception was thrown by the application.
      System.NullReferenceException: Object reference not set to an instance of an object.
         at Duplicati.Server.Scheduler.get_WorkerQueue()
         at Duplicati.Server.Scheduler.GetSchedulerQueueIds()
         at WebserverCore.Services.SchedulerService.GetSchedulerQueueIds()
         at Duplicati.WebserverCore.Services.StatusService.GetStatus()
         at Duplicati.WebserverCore.Notifications.WebsocketAccessor.SendInitialStatus(WebSocket connection)
         at Duplicati.WebserverCore.Notifications.WebsocketAccessor.AddConnection(WebSocket newConnection)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
         at Duplicati.WebserverCore.Middlewares.StaticFilesExtensions.<>c__DisplayClass5_0.<<UseDefaultStaticFiles>b__1>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Duplicati.Server.Scheduler.get_WorkerQueue()
   at Duplicati.Server.Scheduler.GetSchedulerQueueIds()
   at WebserverCore.Services.SchedulerService.GetSchedulerQueueIds()
   at Duplicati.WebserverCore.Services.StatusService.GetStatus()
   at Duplicati.WebserverCore.Notifications.WebsocketAccessor.<.ctor>b__4_0(Object _, EventArgs _)
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Inside getter
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
Connection to localhost (127.0.0.1) 8200 port [tcp/*] succeeded!
Server has started and is listening on port 8200
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
         at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
         at lambda_method245(Closure, EndpointFilterInvocationContext)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)

[DataBitz]

I just created a new backup job to the same Storj storage, using the same advanced options but into a different folder and Duplicati also crashes.
Created another new backup job to the same Storj storage, using the default options (no advanced options) and Duplicati also crashes.
Seems to be something fundamentally broken with Duplicati backing up to Storj?
Backup jobs are using AES-256 encryption and StorJ authentication using an Access Grant.

Here is a config used in the 2nd (most simple) backup job, some values changed for privacy.

{
  "CreatedByVersion": "2.1.0.2",
  "Schedule": null,
  "Backup": {
    "ID": "7",
    "Name": "Media to Storj",
    "Description": "",
    "Tags": [],
    "TargetURL": "storj://storj.io/config?storj-auth-method=Access%20grant\u1234storj-satellite=us1.storj.io%3A7777\u1234storj-bucket=backup-duplicati\u5678storj-folder=Media",
    "DBPath": "/config/TEMEXWUCAK.sqlite",
    "Sources": [
      "/source/Media/"
    ],
    "Settings": [
      {
        "Filter": "",
        "Name": "encryption-module",
        "Value": "aes",
        "Argument": null
      },
      {
        "Filter": "",
        "Name": "compression-module",
        "Value": "zip",
        "Argument": null
      },
      {
        "Filter": "",
        "Name": "dblock-size",
        "Value": "50MB",
        "Argument": null
      },
      {
        "Filter": "",
        "Name": "keep-versions",
        "Value": "2",
        "Argument": null
      }
    ],
    "Filters": [],
    "Metadata": {},
    "IsTemporary": false
  },
  "DisplayNames": {
    "/source/Media/": "Media"
  }
}

[DataBitz]

I just deployed a new instance of Duplicati in docker and it has the same problem.

[kenkendk]

@DataBitz

Do you mean --http-readwrite-timeout=1h? Doesn’t seem to make any difference.

That option is not in 2.1.0.2, it is in 2.1.0.102, but it is broken there as well.
I see the original report says 2.0.1.2 is also having the issue.

@DataBitz

Here are the logs after starting the job and having the web ui stop straight after each. Did it twice to show the pattern of logs. These are from the docker terminal.
Uptime on the docker container is 28 hours.

The logs do not reveal anything useful for this problem unfortunately. My guess is that this is a native crash happening when calling the Storj library. If it is something like a segfault, the process will just be killed by the OS, and no logging will be shown.

For the uptime, is this the linuxserver image? If it is restarting the process on crash, then the container looks like it is “up” for a long time, but the process is not.

@DataBitz

I just deployed a new instance of Duplicati in docker and it has the same problem.

Thanks for testing, I hope @TopperDEL may have some input on possible solutions.

[DataBitz]

@kenkendk

linuxserver

Thanks,
Haven’t tried 2.1.0.102, it that worth trying?
Yes it is linuxserver.

[kenkendk]

@DataBitz

Haven’t tried 2.1.0.102, it that worth trying?

It has the same Storj library as 2.1.0.2, so it will not fix this issue.
You can see the list of changes:

Release: 2.1.0.102 (Canary) 2024-12-12

2.1.0.102_canary_2024-12-12 This release is a canary release intended to be used for testing. Note that this build is using .NET8, unlike the 2.0.8.1 build Note that this build has a redesigned restore flow Should issues arise, the previous restore engine can be enabled with --restore-legacy=true. Detailed list of changes: Upgraded FluentFTP to fix a crash on socket timeout Removed named channels to fix a rare exception Rewrote the restore process to be multithreaded and target optimal disk…

@DataBitz

Yes it is linuxserver.

Then it is likely handling the restart within the container.

[TopperDEL]

I just got the info through the github issue - I did not receive any info from this forum.

I just did some testing with my library on linux (ubuntu 24.04) and do not get any crash or info. I’ve created a small test program that creates files, upload them, fetches their info and downloads them again. That seems to work flawlessly.

But I do have a side-project of mine where I have an uplink.NET-using library within a docker container and that crashes sometime, too. So it may be docker leading to crashes here. Is the first backup you mentioned also within a docker container @DataBitz?

[TopperDEL]

I’ve just put my code into a docker container and it’s running without problems. If only I could reproduce the issue…

[kenkendk]

Would it help if you could reproduce with a Duplicati Docker container?

You can use the duplicati-backendtool to perform operations.

For example:

docker run duplicati/duplicati:2.1.0.2 /opt/duplicati/duplicati-backend-tool LIST 
  "storj://us1.storj.io:7777?storj-api-key=...&storj-secret=...&storj-auth-method=...&storj-satellite=..."

docker run duplicati/duplicati:2.1.0.2 /opt/duplicati/duplicati-backend-tool PUT 
  "storj://us1.storj.io:7777?storj-api-key=...&storj-secret=...&storj-auth-method=...&storj-satellite=..."
  ./testfile.bin

docker run duplicati/duplicati:2.1.0.2 /opt/duplicati/duplicati-backend-tool GET 
  "storj://us1.storj.io:7777?storj-api-key=...&storj-secret=...&storj-auth-method=...&storj-satellite=..."
  ./downloadedfile.bin

[TopperDEL]

Thanks! I have a small crashing application now and a core-dump, but I do not find the root cause. It seems that calling in parallel into my uplink.NET-library does not work on linux. One upload after the other works, though.

If anyone is a pro in debugging ubuntu core-dumps, let me know.

[TopperDEL]

Ok, so - I do not have a real repro. My test-code crashed because I’ve used new Random(), which is not thread safe. Damn it. Whole day searching for a bug at the wrong location…

[DataBitz]

Hi @TopperDEL, thank you for looking into this.
I did another test by using a folder with a small number of files (132) and the backups worked.
So it’s looks like it may only be a problem with large numbers of source files. When I watch the backup starting it looks like it is stopping at the “counting files” stage at around 7500 files.
To reproduce the problem you may need this many files.

[DataBitz]

Having trouble getting the duplicati storj syntax right for docker run with an access grant, but suspect this will work fine now we know its only a problem with a large number of files on the source.
FWIW when I used the duplicati/duplicati:2.1.0.2 docker image instead of lscr.io/linuxserver/duplicati it also crashes when starting the backup if there are a lot of files in the source.

[kenkendk]

@DataBitz

FWIW when I used the duplicati/duplicati:2.1.0.2 docker image instead of lscr.io/linuxserver/duplicati it also crashes when starting the backup if there are a lot of files in the source.

Is this unrelated to Storj? I mean does it just crash if there are many files in the source regardless of the destination?
(and how many is that?).

[DataBitz]

Hi @kenkendk,
Good point, I just setup a new backup job, of the files which I normally backup to Google Drive. This is a smaller backup drive with only about 200 files and it bombed out part way through sending the files after about 1 minute. It did start sending files though which.
As a further test I sent the main media backup of 70K files which normally goes to Storj to Google drive and it completed the counting files stage and started backing up files fine. I don’t have enough storage to send these to Google so cancelled the job after about 15 minutes.
So seems to be Storj specific.

[rarawls]

I tried limiting the backup to a subdirectory of only four files. Observed the same crash.

I’m using lscr.io/linuxserver/duplicati:2.1.0. Used an access access grant with global privilege.

duplicati-1  | Inside getter
duplicati-1  | Connection to localhost (::1) 8200 port [tcp/*] succeeded!
duplicati-1  | Server has started and is listening on port 8200
duplicati-1  | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
duplicati-1  |          at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
duplicati-1  |          at lambda_method245(Closure, EndpointFilterInvocationContext)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
duplicati-1  |       --- End of stack trace from previous location ---
duplicati-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
duplicati-1  | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
duplicati-1  |          at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
duplicati-1  |          at lambda_method245(Closure, EndpointFilterInvocationContext)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
duplicati-1  |       --- End of stack trace from previous location ---
duplicati-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
duplicati-1  | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
duplicati-1  |          at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
duplicati-1  |          at lambda_method245(Closure, EndpointFilterInvocationContext)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
duplicati-1  |       --- End of stack trace from previous location ---
duplicati-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
duplicati-1  | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
duplicati-1  |          at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
duplicati-1  |          at lambda_method245(Closure, EndpointFilterInvocationContext)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
duplicati-1  |       --- End of stack trace from previous location ---
duplicati-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
duplicati-1  | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
duplicati-1  |          at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
duplicati-1  |          at lambda_method245(Closure, EndpointFilterInvocationContext)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
duplicati-1  |       --- End of stack trace from previous location ---
duplicati-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
duplicati-1  | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
duplicati-1  |          at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
duplicati-1  |          at lambda_method245(Closure, EndpointFilterInvocationContext)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
duplicati-1  |       --- End of stack trace from previous location ---
duplicati-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
duplicati-1  | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup
duplicati-1  |          at Duplicati.WebserverCore.Endpoints.V1.ProgressState.Execute()
duplicati-1  |          at lambda_method245(Closure, EndpointFilterInvocationContext)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
duplicati-1  |          at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
duplicati-1  |          at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
duplicati-1  |       --- End of stack trace from previous location ---
duplicati-1  |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)

[kenkendk]

Hi @rarawls, welcome to the forum.

@rarawls

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
duplicati-1  |       An unhandled exception has occurred while executing the request.
duplicati-1  |       Duplicati.WebserverCore.Exceptions.NotFoundException: No active backup

This is unrelated to the crash itself. It happens because the webserver logs that the client gets a 404 response, which is the correct response. It has been fixed in source, so it no longer logs.

The issue in this thread seems to be related to Storj misbehaving on Linux. Is this the same you are seeing?