Release: 2.0.9.106 (Canary) 2024-09-03

[kenkendk]

@ts678

The 387 files that WinMerge shows Repair added are these:

Thanks! I can reproduce now.
I have 1038 files with a normal install and 652 files if I upgrade from a previous version.

edit: I have found a workaround for handling the missing files after an MSI upgrade.

I have no nice things to say about MSI at this point

[jdrch]

Thanks. Unfortunately even uninstalling and reinstalling didn’t help for me. I wound up having to downgrade to the 2024-08-29 canary build.

[Sami_Lehtinen]

Unhandled exception. System.TypeInitializationException: The type initializer for ‘Duplicati.Library.AutoUpdater.AutoUpdateSettings’ threw an exception.

@kenkendk Answer to the another question, yes. I’ve upgraded using the MSI package from previous canary on test system.

Here’s the full error message, if it’s helpful.

Unhandled exception. System.TypeInitializationException: The type initializer for 'Duplicati.Library.AutoUpdater.AutoUpdateSettings' threw an exception.
---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Memory, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. 
File name: 'System.Memory, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
   at System.Text.RegularExpressions.RegexParser.AddToConcatenate(Int32 pos, Int32 cch, Boolean isReplacement)
   at System.Text.RegularExpressions.RegexParser.ScanRegex()
   at System.Text.RegularExpressions.RegexParser.Parse(String pattern, RegexOptions options, CultureInfo culture)
   at System.Text.RegularExpressions.Regex..ctor(String pattern, RegexOptions options, TimeSpan matchTimeout, CultureInfo culture)
   at System.Text.RegularExpressions.Regex..ctor(String pattern, RegexOptions options)
   at Duplicati.Library.AutoUpdater.AutoUpdateSettings..cctor()
   --- End of inner exception stack trace ---
   at Duplicati.Library.AutoUpdater.AutoUpdateSettings.get_AppName()
   at Duplicati.Library.UsageReporter.Reporter.get_IsDisabledByEnvironment()
   at Duplicati.Library.UsageReporter.Reporter.get_IsDisabled()
   at Duplicati.Library.UsageReporter.Reporter.Initialize()
   at Duplicati.CommandLine.Program.Main(String[] args)
   at Duplicati.CommandLine.CLI.Net8.Program.Main(String[] args)

[Taomyn]

Something else to add to the “nice to have” list - show status of scripts in a job: I have a backup that runs a startup script which dumps an 8GB database for backup, and as this takes around 10-15min, all the GUI shows is:

And nothing in the live logs, so it makes it look like the job has hung.

[kenkendk]

@jdrch

Thanks. Unfortunately even uninstalling and reinstalling didn’t help for me. I wound up having to downgrade to the 2024-08-29 canary build.

That is unexpected.

The issues mentioned here are all related to files missing during upgrade.
The binaries from the previous build and this build should be the same (at least in terms of not showing the .NET8 installation message).

Did you keep getting the .NET8 install message, even with the uninstall+install cycle?

@Sami_Lehtinen

Here’s the full error message, if it’s helpful.

Thanks, super helpful! It shows that the file System.Memory.dll is missing. I think this is caused by the upgrade failure, so it should be fixed in the next build.

@Taomyn

Something else to add to the “nice to have” list - show status of scripts in a job: I have a backup that runs a startup script which dumps an 8GB database for backup, and as this takes around 10-15min, all the GUI shows is:

I have added an issue for showing the script phase.

[kenkendk]

@ts678

Port search seems to half-work now (better than before), but doesn’t back down as ports free up.
2.0.8.1 would search again, starting at 8200. Current behavior seems sticky even without a need.
I sort of recall seeing a port value in the database. Maybe it’s now used? Is the change intended?

Yes, this is an intended change. I think it was originally supposed to remember the port with the logic being that other things may point to it (shortcuts, bookmarks, cli-tools, etc). I realized that this was not working so I “fixed” it thinking it was broken during all the logic changes around the web server.

You can always force a port, but I think it makes sense to make it prefer the previously used port by default.

Is there a scenario where this breaks things?

@ts678

In a more obscure case, 2.0.8.1 handled the odd case of --webservice-interface=any “better” (depending on taste, I guess). If port 8200 was taken with --webservice-interface=loopback, second Duplicati would LISTEN on 0.0.0.0:8200 (but user might get confused). New one crashes:

I think this is happening because the logic is fairly odd in this case. If you have two processes listening on the same port, but for two different addresses, you can only reach the 0.0.0.0 one by using something other than localhost as that resolves to 127.0.0.1 which is bound. In this case you need --hosturl=http://192.0.0.x:8200 or similar, so you reach the network interface with an IP that is not 127.0.0.1.

If you use --hosturl=http://localhost:8200 --webservice-interface=any the server will bind to 0.0.0.0 but the connection will reach 127.0.0.1 and they do not share authentication (as they are different processes).

This problem only occurs if you are also running an instance with --webservice-interface=loopback because it makes a more specific binding that makes the connection reach the wrong adapter.

tl;dr: Don’t run two instances on the same port

[ts678]

@kenkendk

To upgrade from 2.0.9.105, start with --disable-db-encryption

I’m pretty sure this was aimed at TrayIcon and Server, but ServerUtil doesn’t accept this option:

C:\Duplicati\duplicati-2.0.9.106_canary_2024-09-03-win-x64-gui>Duplicati.CommandLine.ServerUtil.exe list-backups
Duplicati.Library.Interface.SettingsEncryptionKeyMissingException: Encryption key is missing.
   at Duplicati.Library.Encryption.EncryptedFieldHelper.Decrypt(String value, KeyInstance key)
   at Duplicati.Library.Encryption.EncryptedFieldHelper.Decrypt(String value)
   at Duplicati.CommandLine.ServerUtil.Settings.<>c.<LoadSettings>b__28_0(PersistedSettings x)
   at System.Linq.Enumerable.SelectListIterator`2.Fill(ReadOnlySpan`1 source, Span`1 destination, Func`2 func)
   at System.Linq.Enumerable.SelectListIterator`2.ToList()
   at Duplicati.CommandLine.ServerUtil.Settings.LoadSettings(String filename)
   at Duplicati.CommandLine.ServerUtil.Settings.Load(String password, Uri hostUrl, String serverDataFolder, String settingsFile)
   at Duplicati.CommandLine.ServerUtil.SettingsBinder.GetSettings(BindingContext bindingContext)
   at Duplicati.CommandLine.ServerUtil.Program.<>c__DisplayClass0_0.<Main>b__2(IServiceProvider _)
   at System.CommandLine.Binding.BindingContext.<>c__DisplayClass16_0`1.<AddService>b__0(IServiceProvider s)
   at System.CommandLine.Invocation.ServiceProvider.GetService(Type serviceType)
   at System.CommandLine.Binding.ServiceProviderValueSource.TryGetValue(IValueDescriptor valueDescriptor, BindingContext bindingContext, Object& boundValue)
   at System.CommandLine.Binding.BindingContext.TryBindToScalarValue(IValueDescriptor valueDescriptor, IValueSource valueSource, LocalizationResources localizationResources, Nullable`1& boundValue)
   at System.CommandLine.NamingConventionBinder.ModelBinder.GetBoundValue(IValueSource valueSource, BindingContext bindingContext, IValueDescriptor valueDescriptor, Boolean includeMissingValues, Type parentType)
   at System.CommandLine.NamingConventionBinder.ModelBinder.GetBoundValues(IDictionary`2 bindingSources, BindingContext bindingContext, IReadOnlyList`1 valueDescriptors, Boolean enforceExplicitBinding, Type parentType, Boolean includeMissingValues)
   at System.CommandLine.NamingConventionBinder.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Duplicati.CommandLine.ServerUtil.Program.<>c.<<Main>b__0_1>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

C:\Duplicati\duplicati-2.0.9.106_canary_2024-09-03-win-x64-gui>Duplicati.CommandLine.ServerUtil.exe list-backups --disable-db-encryption
Unrecognized command or argument '--disable-db-encryption'.

Description:
  List all backups

Usage:
  Duplicati.CommandLine.ServerUtil list-backups [options]

Options:
  --password <password>                    The password to use []
  --hosturl <hosturl>                      The host URL to use [default: http://localhost:8200/]
  --server-datafolder <server-datafolder>  The server datafolder to use for locating the database []
  --settings-file <settings-file>          The settings file to use []
  -?, -h, --help                           Show help and usage information



C:\Duplicati\duplicati-2.0.9.106_canary_2024-09-03-win-x64-gui>

As an unwanted workaround (anything nicer?) starting with fresh settings.json suggests that:

C:\Duplicati\duplicati-2.0.9.106_canary_2024-09-03-win-x64-gui>Duplicati.CommandLine.ServerUtil.exe list-backups
Connecting to http://localhost:8200/...
No database encryption key was found. The database will be stored unencrypted. Supply an encryption key via the environment variable SETTINGS_ENCRYPTION_KEY or disable database encryption with the option --disable-db-encryption

which suggests using an unsupported option, which gets rejected.

@kenkendk

I think it was originally supposed to remember the port

@kenkendk

Duplicati tries different ports, starting with 8200. The first user to log in will get that port, the next will get port 8300, next on 8400. The port number is recorded in the settings database, such that the next launch will attempt to keep this number.

was the idea in Oct 2017, but in addition to 2.0.8.1_beta_2024-05-07 starting fresh, it’s also in 2.0.2.1_beta_2017-08-01. I don’t think this memory is documented in the User’s Manual either.

There are pros and cons to memory. One con is that it’s harder to know what it got, and harder remedy if it didn’t go where desired. Restarting is easier than options or environment variables, especially for TrayIcon which is where the issue is, often is not CLI, and often loses messages.

The 2017 link later shows how memory doesn’t really solve the problem of chosen port moving:

@kenkendk

This can go wrong, say that Kees is 8200, Ken is 8300, but then the machine reboots and Jon logs in for the first time. He will then get 8200, and when Kees logs in, Duplicati will see that 8200 is taken and scan for a new one, finding 8300, etc.

If the users have bookmarks, then it all breaks.

Does it help a bit? Maybe for awhile. It can stop the port from going down, but not from going up, however I’m not sure that outweighs the increased port troubleshooting and return-to-usual pain.

The ServerUtil help is pretty brief, but help (and code) suggests that port 8200 is the default port.

Another comment in ServerUtil is there seem to be excess empty lines on list-backups output:

1: test 1


2: ftp


3: aftp

[kenkendk]

@ts678

I’m pretty sure this was aimed at TrayIcon and Server, but ServerUtil doesn’t accept this option:

True. I did not see a scenario where it would make sense to store an unencrypted JWT value, so you cannot disable encryption of the settings file. Only in the case where there is no key (2.0.9.106+) is it possible to store the token unencrypted.

@ts678

As an unwanted workaround (anything nicer?) starting with fresh settings.json suggests that:

That message is actually from opening Duplicati-server.sqlite but it is quite confusing in this context. I have registered an issue for suppressing these messages.

As a workaround, you can just delete the RefreshToken from the file. It will require you to run Duplicati.CommandLine.ServerUtil.exe login again though.

@ts678

was the idea in Oct 2017, but in addition to 2.0.8.1_beta_2024-05-07 starting fresh, it’s also in 2.0.2.1_beta_2017-08-01. I don’t think this memory is documented in the User’s Manual either.

I think the idea was the same from the beginning but it has just never been implemented correctly.

@ts678

There are pros and cons to memory. One con is that it’s harder to know what it got, and harder remedy if it didn’t go where desired. Restarting is easier than options or environment variables, especially for TrayIcon which is where the issue is, often is not CLI, and often loses messages.

The 2017 link later shows how memory doesn’t really solve the problem of chosen port moving:

My idea was that remembering the port would be a better way to fix a port conflict with another program. I think that comment illustrates an issue that occurs when running Duplicati for multiple users.

I think memory is actually better for multi-user setups as well. Ideally, each user starts in some order, as that would allocate a specific port for each user. Reboots would then keep the ports and bookmarks would work. It could still create some conflicts where the ports keeps getting taken in specific scenarios though.

I think for special setups like multi-users, there needs to be some management of ports anyway.

My use-case for this logic is not advanced users, it is non-technical users that just want something that works. The actual port has very little value to them, except if they choose to bookmark the page. These users are less likely to have a multi-user setup (my guess) so I would prefer to guard against the conflict with another program here.

Only the tray icon will probe for ports, so it is only users that are starting the via the tray that have the port probing issues.

My main argument is that if you rely on the port, make sure to set it directly.

@ts678

Another comment in ServerUtil is there seem to be excess empty lines on list-backups output:

That is the “description” field from the backup, but I have noticed it for my test backups as well and removed the empty lines for the next build.

[jdrch]

@kenkendk

Did you keep getting the .NET8 install message, even with the uninstall+install cycle?

No. Instead the tray icon never shows up on launch even though there’s a matching process in Task Manager, and based on the browser not being able to connect to the web UI I don’t think the web server ever actually starts.

[kenkendk]

@jdrch

No. Instead the tray icon never shows up on launch even though there’s a matching process in Task Manager, and based on the browser not being able to connect to the web UI I don’t think the web server ever actually starts.

In that case, I think you are seeing a different issue. Since it works with 2.0.9.105, my best guess is that this is related to the botched settings encryption that was introduced in 2.0.9.105 and removed again in 2.0.9.106.

With 2.0.9.105 installed, set either the commandline argument --disable-db-encryption or the environment variable DUPLICATI__DISABLE_DB_ENCRYPTION=true (whichever is easier for you). Then start 2.0.9.105 to have it disable the settings encryption, and you can now upgrade to 2.0.9.106.

I have tried to reproduce the scenario you describe where the server is crashed but the process does not exit. So far I have not been able to reproduce, but I will keep looking for a way.

[ts678]

2.0.9.107 is out, but before that, I got 2.0.9.106 Connection lost (with some logs) not at wake.

A tab got 401 Unauthorized to a refresh with a RefreshToken_8200 from a seemingly closed tab.

The set which I can't find in the two tabs with logging, so might be closed:

POST /api/v1/auth/refresh HTTP/1.1
Host: localhost:8200
Connection: keep-alive
Content-Length: 0
sec-ch-ua: "Not)A;Brand";v="99", "Microsoft Edge";v="127", "Chromium";v="127"
Accept: application/json, text/plain, */*
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
sec-ch-ua-platform: "Windows"
Origin: http://localhost:8200
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:8200/ngax/index.html
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cookie:
...

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 10 Sep 2024 11:40:30 GMT
Server: Kestrel
Set-Cookie: RefreshToken_8200=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXAiOiJSZWZyZXNoVG9rZW4iLCJzaWQiOiJ3ZWJzZXJ2ZXIiLCJmYW0iOiJEQ0VGODlCQzM5MTMwODE5IiwiY250IjoiMjE1IiwiaWF0IjoiLTE3MjU5Njg0MzEuMTc3Njk0IiwibmJmIjoxNzI1OTY4NDMxLCJleHAiOjE3Mjg1NjA0MzEsImlzcyI6Imh0dHBzOi8vZHVwbGljYXRpIiwiYXVkIjoiaHR0cHM6Ly9kdXBsaWNhdGkifQ.Jn3IubMLqUgyvqnl135zC2A_YdZ-H9-8wNGPI_xptWI; expires=Thu, 10 Oct 2024 11:40:31 GMT; domain=localhost; path=/api/v1/auth/refresh; samesite=strict; httponly
Transfer-Encoding: chunked

12b
{"AccessToken":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXAiOiJBY2Nlc3NUb2tlbiIsInNpZCI6IndlYnNlcnZlciIsImZhbSI6IkRDRUY4OUJDMzkxMzA4MTkiLCJuYmYiOjE3MjU5Njg0MzEsImV4cCI6MTcyNTk2OTMzMSwiaXNzIjoiaHR0cHM6Ly9kdXBsaWNhdGkiLCJhdWQiOiJodHRwczovL2R1cGxpY2F0aSJ9.jCuNLe8gf1re9UQgTazLA9HuLWMBEAUumYTfChNIjiY"}
0

The use:

POST /api/v1/auth/refresh HTTP/1.1
Host: localhost:8200
Connection: keep-alive
Content-Length: 0
sec-ch-ua: "Not)A;Brand";v="99", "Microsoft Edge";v="127", "Chromium";v="127"
Accept: application/json, text/plain, */*
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
sec-ch-ua-platform: "Windows"
Origin: http://localhost:8200
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: http://localhost:8200/ngax/index.html
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cookie: ...
RefreshToken_8200=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXAiOiJSZWZyZXNoVG9rZW4iLCJzaWQiOiJ3ZWJzZXJ2ZXIiLCJmYW0iOiJEQ0VGODlCQzM5MTMwODE5IiwiY250IjoiMjE1IiwiaWF0IjoiLTE3MjU5Njg0MzEuMTc3Njk0IiwibmJmIjoxNzI1OTY4NDMxLCJleHAiOjE3Mjg1NjA0MzEsImlzcyI6Imh0dHBzOi8vZHVwbGljYXRpIiwiYXVkIjoiaHR0cHM6Ly9kdXBsaWNhdGkifQ.Jn3IubMLqUgyvqnl135zC2A_YdZ-H9-8wNGPI_xptWI; default-theme=ngax

HTTP/1.1 401 Unauthorized
Content-Type: application/json
Date: Tue, 10 Sep 2024 13:09:25 GMT
Server: Kestrel
Cache-Control: no-cache,no-store
Expires: -1
Pragma: no-cache
Transfer-Encoding: chunked

{"Error":"Failed to refresh token","Code":401}

After playing around with the other tab with logging, which survived, I tried the Reload button on Duplicati’s blue Connection lost screen. This has sometimes worked before, and is less of a nuisance than the Log in button. It worked, but I wondered about how much tabs share a login.

Sharing is somewhat familiar from Internet web site logins, and it does seem that doing Log out from one tab logs out the other. It’s not always noticed, for example during a backup it makes the below, but the visual appearance is that the screen just stops updating without raising any errors:

and TrayIcon output (if one is making it visible)

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      Duplicati.WebserverCore.Exceptions.UnauthorizedException: Failed to refresh token
         at Duplicati.WebserverCore.Endpoints.V1.Auth.<>c.<<Map>b__3_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<TaskOfTToValueTaskOfObject>g__ExecuteAwaited|92_0[T](Task`1 task)
         at Duplicati.WebserverCore.Middlewares.HostnameFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Duplicati.WebserverCore.Middlewares.LanguageFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
         at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|129_0(ValueTask`1 valueTask, HttpContext httpContext, JsonTypeInfo`1 jsonTypeInfo)
         at Duplicati.WebserverCore.Middlewares.WebsocketExtensions.<>c__DisplayClass0_0.<<UseNotifications>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)