Release: 2.0.9.105 (Canary) 2024-08-29

Releases
1

2.0.9.105_canary_2024-08-29

This release is a canary release intended to be used for testing in preparation of a later stable release.

Unlike regular canary builds, this one has a major change in the build system, so it now runs on .NET8.

For that reason, the updater in previous canary builds does not detect this update yet, but this can be activated at a later time.

Important changes from last Beta

  • Updated to .NET8 with OS specific builds
  • Using Kestrel as the API/UI server
  • Mandatory password and new authentication scheme
  • Settings database version updated to v8
  • Encrypting data in Duplicati-server.sqlite with machine serial number

Please see list of known issues related to .NET8/Kestrel upgrade:

New tool to manage a running server

Due to incompatibility with duplicati_client a new tool is included, named Duplicati.CommandLine.ServerUtil.exe/duplicati-server-util.

The new tool can pause/resume a backup, run a backup, change the password and more:

Encrypting database fields

To reduce the risk of leaking encryption passphrases and credentials,
many fields in the Duplicati-server.sqlite file will be encrypted after running this version.
The key used to encrypt is derived from the machine serial number, so the database cannot be decrypted on another machine.

If your strategy relies on being able to read this database, you must take action.
These two setups are vulnerable:

  • If you do not store a copy of the passphrase elsewhere
  • If you make a copy of the settings database

We strongly recommend that you store a copy of the passphrase(s) securely, regardless of your setup.

If you want to choose the settings encryption key, you can set the environment variable SETTINGS_ENCRYPTION_KEY to a custom value.
If you want to never use the serial number as the passphrase, set the environment
variable DUPLICATI__REQUIRE_DB_ENCRYPTION_KEY=true, which will prevent Duplicati from starting without a user provided key.

If you need to change the key, you can temporarily decrypt the database by starting the server with --disable-db-encryption.
After starting, stop the instance again, set SETTINGS_ENCRYPTION_KEY and start again without the argument, to have it re-encrypted.

To downgrade from this version, run once with --disable-db-encryption, and change the version number to 7, then install the previous version.

As always, feedback is appreciated!

Detailed list of changes:

  • Simplified logic for finding the folder containing the settings database
  • Encrypting settings in database with machine serial number
  • Fixed issue with server not responding to CTRL+C or stop commands
  • Fixed issue with TrayIcon not trying multiple ports
  • Added utility program to control a running server
  • Improved the initial password setup experience
  • Added support for logging to Windows Event Log and added Description to Windows Service
  • Fixed an issue where the retention value could not be saved if it was a number
  • Fixed an upgrade issue where %HOME% would not resolve correctly on Linux
  • Fixed an issue with parsing --send-http-result-output-format
  • Updated Docker image to use environment variables and not use settings encryption by default
  • Added support for pre-loading default settings on a machine or installation
2

Should that env var name start with DUPLICATI__ like all the others?

3

Also, what are the specs for this key e.g. what is “machine serial number” and the maximum characters?

4

Good question, I was considering if a prefix was needed.

The convention for DUPLICATI__ is used for commandline argument mapping, and this is not a property that you can supply via the commandline. If think the usecase for this variable is special, so I do not see a need for having a commandline argument, but please enlighten me if I am wrong.

There are others, like DUPLICATI_HOME that uses a Duplicati prefix (with single-underscore), and still others like AUTH_USERNAME that have no prefix. There are even some that have Duplicati inside, like AUTOUPDATER_DUPLICATI_CHANNEL.

In all, the env names are quite sprawling and could use a standardization. Recommendations are appreciated :).

The library for this is DeviceId, and the specific code in Duplicati is:

Looking through that, it goes to /sys/class/dmi/id/board_serial on Linux.
I have tested it on a variety of machines, but right now I found a machine where it could not be read.

I will make a new build that includes a guard for this, as it could result in using an empty string as the encryption key, giving a false sense of security.

5

Ok, so a random 40 character code generated by my password manager should be ok?

6

Sorry, trying to wrap my head around this for when installing as a service, but for Linux and Windows, how do I set the environment variable so that when the service is started it uses the new key? I wouldn’t want the variable set permanently so I don’t want to add it to the operating system, so just for my current session so it’s only used until I log off.

7

I guess key sizes always comes down to potential adversaries. If you do not have state secrets, I would say it is sufficient (40 * ~6 bits = 240 bits key). The key protects IF the database is leaked.

I have investigated this a bit and the guards against empty passphrases were not being triggered because the DeviceId library will hash the result before returning it. In the case there was no information available, it returns the key WERC8GMRZGE196QVYK49JVXS4GKTWGF4CJDS6K54JPCHPY2JQ1AG which looks like a really secure key, but is essentially just the hash of an empty string. Because this string value is passed on as the deviceId all other checks pass, making everything appear encrypted.

If you have installed 2.0.9.105 with encryption enabled, the encryption key from the above is:

SETTINGS_ENCRYPTION_KEY=ECB47E9D8445E0A3F30A1435BE075C101F202FF5445BA01A9F9A8DBD4506F5F3

I will add this key to the blacklist so it cannot be used in future versions, but you may need it to decrypt your current settings.

8

I have not yet installed, but that path does not exist in my linux machine. Not in the host, and not in docker container.

9

Yeah, I found out now. The library is not super good for Linux use I see.

I have made a PR that handles no serial number.

For this release, the issue is not destructive, so you can update if you want to try it.

The only side effect is that the database will be encrypted with a fixed key (as listed above).
I have made the blacklist fix upgrade-enabled, such that if you use a blacklisted key, it will still decrypt your database on startup, but not use the blacklisted key.
In the next build, it will start up just fine, but decrypt the database and give a warning that the key is blacklisted, allowing you to set a key manually.

10

Meaning due to permissions, like this as a non-root user on Linux Mint 21.2 on VirtualBox?
This doesn’t look like a machine-specific issue (but keep reading…) but a user-specific one.

$ cat /sys/class/dmi/id/board_serial
cat: /sys/class/dmi/id/board_serial: Permission denied
$ ls -l /sys/class/dmi/id/board_serial
-r-------- 1 root root 4096 Aug 30 08:04 /sys/class/dmi/id/board_serial

There are also strings that are not technically empty, but not really useful as a serial number:

# cat /sys/class/dmi/id/board_serial | od -c
0000000   0  \n
0000002

Same result on AlmaLinux 9 on VirtualBox (just in case distro matters to the result somehow).

11

Exactly. That was how I discovered the issue with returning a useless ID.
I can see that many (all?) distros protect these, which is probably a good thing.

It is technically fine as it is hashed, but having a 3 byte key is also close to useless.
Beginning to look like we should just disable the deviceId feature for Linux :cry:

12

How is hashing a fixed string fine? This sounds to me like a very similar problem to this one:

which got blacklisted. Couldn’t blacklist also dodge non-empty fixed strings we can identify?

It might not just be Linux. Although I don’t know what the library looks at, here’s a Win11 VM:

C:\>wmic bios get serialnumber
SerialNumber
0


C:\>wmic baseboard get serialnumber
SerialNumber
0
13

I was not considering it fixed, but yes, if it is the same for multiple machines, it has little to no value.

That is a good angle I think. Maybe the real way to consider this is to be vigilant about the serial number data we actually get, instead of relying on the library to provide a good answer.

The data reported here makes me worry that this is a never-ending chase for weird setups that is impossible to replicate or get information on. In this thread alone, it looks like at least 3 different useless values, and only one is blacklisted.

14

I’m guessing it’s from the software providing the VM (not from VM guest). That’s lots to test…

15

Server help has disappeared compared to 2.0.9.104. This also affects TrayIcon display of that.

C:\Duplicati\duplicati-2.0.9.105_canary_2024-08-29-win-x64-gui>Duplicati.Server.exe --help
Supported commandline arguments:



C:\Duplicati\duplicati-2.0.9.105_canary_2024-08-29-win-x64-gui>Duplicati.GUI.TrayIcon.exe --help

C:\Duplicati\duplicati-2.0.9.105_canary_2024-08-29-win-x64-gui>Supported commandline arguments:

--hosturl: Supply the url that the TrayIcon will connect to and show status for
--no-hosted-server: Set this option to not spawn a local service, use if the TrayIcon should connect to a running service
--read-config-from-db: Set this option to read server connection info for running service from its database (only together with no-hosted-server)
--browser-command: Set this option to override the default browser detection
--detached-process: This option runs the tray-icon in detached mode, meaning that the process will exit immediately and not send output to the console of the caller
Additionally, these server options are also supported:
16

I found the issue with missing help text and fixed it.

I have made a PR that removes the DeviceId default keys.

17

ServerUtil helped to get my new installation password set more easily than before, although I did have to give it my database path which is non-standard. Maybe a normal setup would be easier?

The help text offers run <backup> and message like it should be able to take a name. It doesn’t:
Should run by name be supported?

C:\Duplicati\duplicati-2.0.9.105_canary_2024-08-29-win-x64-gui>Duplicati.CommandLine.ServerUtil.exe list-backups
Connecting to http://localhost:8200/...
1: test1



C:\Duplicati\duplicati-2.0.9.105_canary_2024-08-29-win-x64-gui>Duplicati.CommandLine.ServerUtil.exe run 1
Connecting to http://localhost:8200/...
Running backup test1 (1)

C:\Duplicati\duplicati-2.0.9.105_canary_2024-08-29-win-x64-gui>Duplicati.CommandLine.ServerUtil.exe run test1
Connecting to http://localhost:8200/...
No backup found with supplied ID or name

C:\Duplicati\duplicati-2.0.9.105_canary_2024-08-29-win-x64-gui>

EDIT 1:

An import seemed to work, and even prompted for password when I didn’t give one, and showed typing as *. Nice. What’s a bit crude compared to GUI is bad password stack instead of message.

The file is encrypted. Please provide the encryption password: *****
Connecting to http://localhost:8200/...
System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Duplicati.CommandLine.ServerUtil.Connection.ImportBackup(String file, String password, Boolean importMetadata)
   at Duplicati.CommandLine.ServerUtil.Commands.Import.<>c.<<Create>b__0_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.NamingConventionBinder.CommandHandler.GetExitCodeAsync(Object returnValue, InvocationContext context)
   at System.CommandLine.NamingConventionBinder.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Duplicati.CommandLine.ServerUtil.Program.<>c.<<Main>b__0_1>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

GUI says:

image

which is a more specific clue about what went wrong on the import, so makes the debug easier.

18

quote from the 2.0.9.104 thread

checked the machine, there is indeed 1 Duplicati-server.sqlite in the .config folder
the library folder only contains a folder with updates, no sqlite

downloaded the dmg file and installed it
rebooted, tried to access the url, that wasn’t working as you need to open it using the tray icon.
that wasn’t there, so started duplicati manually, then macos did its magic for downloaded applications, but still no tray icon
rebooted again, and there it was :slight_smile: a nice tray icon
needed to set the mandatory passphrase
and there the old database was, waiting for the next backup
ran a backup, everything went fine

still not sure why i had 2 databases then, i do try canary versions on that machine, but i am only going up in version

rebooted again

19

Hi, in this version the TrayIcon starts with the parameters: --no-hosted-server --hosturl=http://localhost:8200 --webservice-password=yourpassword (used to install the service)

The problem is that when i click on “Open” it freezes and i have to kill it from Task Manager.

From http://localhost:8200/ngax/index.html it still works, apart from the initial slowness in showing the translated UI and backups.

However, operation as a service remains very complex to install and manage.

20

Yes, it should! I will revisit, it worked in testing.

True! The server is quite protective of what information it will relay. Here it could have some more useful error. We could also test if the password works locally perhaps.

Super strange… There is a 2.0.9.106 out now, beware that it removes the encryption key that was applied with .105.

My best guess is some kind of security software that blocks the TrayIcon from making the connection?

Yes. That has not yet been addressed.

I want to split it out so the server (and TrayIcon) can use an elevated service that has VSS / Admin permissions, instead of the current setup where the HTTP server is running with Admin permissions. Once the split is made, I think the flow can be simplified by adding questions to the MSI package and installing the service directly.