I have just upgraded from canary 2.0.9.107-1 to stable 2.1.0.4-1
When I start the service, now it fails with the following error:
Duplicati.Library.Interface.SettingsEncryptionKeyMissingException
I presume the key I must have set up a while back (and I can’t find now) was stored somewhere else before the upgrade and now needs to be set as an environment variable.
Where can I find my previously stored key from 2.0.9.107-1 to use it now with the newer version?
I’m not sure you can, but it depends on where you left it. Some possibilities are described:
If you can’t find it, then maybe Import and export backup configurations can help to restart.
--settings-encryption-key=<key> is also allowed since 2.0.9.109_canary_2024-11-06:
--settings-encryption-key: Use this option to set the encryption key for the settings database. This option can also be set wit
h the environment variable SETTINGS_ENCRYPTION_KEY.
Encrypting database fields was 2.0.9.105_canary_2024-08-29 and either an environment variable or a default key based on the machine serial number, which unfortunately was not unique enough.
If you used the default key, I’m not sure if it’s saved anywhere, as it sounds like it’s just generated.
Possibly the developer can say more, but Canary is always subject to change as things are found.
If you have to, you can probably downgrade (might need special steps) to turn off encryption then rework it into the current plan. Make sure you store the secret key somewhere safe if you add one.
thank you, @ts678
I tried downgrading yesterday and that did not help. I will try going to an earlier version today.
I have all the passkeys for each backup, so, hopefully, I can just re-import all of them and not have to redo terabytes of backups all over again.
I will try looking for the preload.json file; hopefully the developer can shed some light on where else the encryption key may have been stored on previous version on my machine.
I usually store all keys in the password manager and I cannot find this one. Any way I can try and self-generate the default key from my machine?
2.0.9.106_canary_2024-09-03 is more explicit about the default key part of encryption design:
Note for users of 2.0.9.105: The method for extracting an encryption key from the machine seriail number did not produce secure results.
This feature has now been removed, and will prevent you from upgrading. To upgrade from 2.0.9.105, start with--disable-db-encryption, then exit, then upgrade to 2.0.9.106+.
Users from other versions than 2.0.9.105 will not have difficulties, and Docker users should not see any issues either.
and an attempt to look at source seems to confirm the brief use, which it seems wasn’t your case because you were at 2.0.9.107. Although things did change later, my idea of an old release being able to generate a default key from the machine deviceid seems wrong. The undo of that is here, with GitHub also claiming 2.0.9.106 change. Maybe you had set up a custom key and then lost it:
I’ve already gone through some possible ways to set it up. If it was an environment variable, I’d think it would still be hanging around (I worry about environment variable privacy though). If you added it on the service install, that’s a different privacy issue. I can’t say what you set up before.
So I just retracted that theory based on your 2.0.9.107 looking like it was past the default key era.
This is the most important one, otherwise the backup is gone. Losing configs is less of a disaster.
There is a chance that most of the config is still in the Duplicati-server.sqlite database, as the 2.1 encryption scheme only encrypts sensitive data such as destination credentials. I guess you can experiment with disabling encryption to see if you can make it show job with non-sensitive config, however that seems a little weird to me. You can look in Duplicati-server.sqlite with a DB browser such as DB Browser for SQLite to see what unencrypted config and options might help reconfigs.
thank you again, @ts678, really appreciate the help, it has been really frustrating.
I must have gone through this when I upgraded from 106 and completely forgot about it.
I suspect you may be right and the key may have been in the service file, however, not knowing that, I may have replaced it trying to get it to work yesterday. Not sure if systemd uses versioning for service files, I will try and check this weekend to see if I can recover. I’m sure it’s not in the higher env scope as I cannot find it. I will try and dig and let you know. Thanks again.
OK, I did a quick test and I downgraded to 106 and it works! So it must have been still using the device generated key when I was using 107. Not sure where it would have stored it as I did not pass it in any way after the upgrade.
Thank you again for the help. For now, I’ll stick to 106 and I will probably follow the process you described to decrypt/encrypt before the upgrade.
That is a bit strange. The auto-generated key was only in 2.0.9.105 so using 2.0.9.106 should not be able to use the auto-generated key.
Anyway, if it starts now, you can use --disable-db-encryption as mentioned by @ts678 to ditch the encryption.
thanks @kenkendk, that is my plan when I will try and update.
in the meantime, if you know a way to figure out the current key and where it comes from, I’d be curious to find out before wiping it out with --disable-db-encryption.
It can come from two places: commandline and environment variable.
To get the commandline from the process, try this SO suggestion.
To get the environment variables, try this.
If neither gives anything, then the input must be from preload.json but locations of this were not changed (except being expanded to more locations).
that’s easy to check:
$ ps -eo args | grep duplicati | head -n -1
/opt/duplicati/duplicati --webservice-port=8200 --auto-update=false
$ env | grep SETTINGS_ENCRYPTION_KEY
<empty line>
$ sudo strings /proc/<duplicati PID>/environ | grep SETTINGS_ENCRYPTION_KEY
<empty line>
$ cat /usr/share/Duplicati/preload.json
cat: /usr/share/Duplicati/preload.json: No such file or directory
$ sudo strings /proc/181883/environ | grep DUPLICATI_PRELOAD_SETTINGS
<empty line>
I’m out of ideas 
Hmm… only other thing is would be environment variable DUPLICATI__WEBSERVICE_PASSWORD ?
In any case, on Linux the autogenerated key did not work in most cases, so everyone was using the same password:
5A24149D0E630AF67CD3AE95CEAD6A618E240A105133F8CACEADF8111FFDB166
I checked also for that one, but no luck, the only Duplicati-specific variable in the systemd env is AUTOUPDATER_Duplicati_SKIP_UPDATE=1 (that confirms I am looking at the right process).
I just noticed something, though… If I look at the logs, I can see this:
Use the following link to sign in: http://localhost:8200/signin.html?token=<token>
That made me curious and I tried to go to http://localhost:8200 from an incognito window and… I got asked for authentication and to provide a password!!
Is that possible that the actual password is the token from the logs and it is cached in the browser’s cookies? that would explain where it is getting it.
In the cookies there is a “RefreshToken_8200”, the value matches the token from the logs.
I could try and upgrade and use that as encryption key; do you know if that is encoded and if any of this makes sense?
thanks.
That is a different password. There are a few in Duplicati now:
The link you found is posted in the log prior to setting an API access password. Once you have set one, it will no longer post the link, and the token expires shortly after being written.
And yes, the API access is cached with a token stored in a cookie.
This has no relation to the database encryption key.
But since it is running now, do you see values with enc: prefix in the database?
got it, thanks.
Yes, I read the Backup table from Duplicati-server.sqlite and all the values in the TargetURL column start with end-v1:
I’m staying out of the main chase, but (reasonably for security), signin access doesn’t last long.
For some external access, such as the one used by the TrayIcon and ServerUtil, a third token type is issued: Signin Token. The Signin Token is a placeholder for the password and can be used to obtain either a Refresh Token or an Access Token, but it is very short-lived.
I think it’s only 5 minutes, per an old forum statement and a peek at JWTProvider.cs.
thank you @ts678 and @kenkendk I truly appreciate all the help.
I had some time today and I ran the server with --disable-db-encryption
I confirmed the database was decrypted and no more enc-v1: cells
I then updated to the latest stable and it works!
We will never know where it was taking the encryption key, but, for now, I’m just glad I was able to upgrade and keep my backups.
P.S. I’m new to the forum, I’m not sure if I should tick the “Solution” box and if so, if you get any credit for helping; if so, I wouldn’t know who to “award” it as you have been both brilliant or if I should just pick my last message, open to suggestions.
From my point of view (but I don’t admin the forum), it’s mainly a clue to users fishing for solutions. Often people don’t do it, so searchers might also find solutions in unmarked topics. It’s also not an issue tracker. Duplicati GitHub Issues has managed Open/Closed, so it’s hard to lose track of stuff.
I don’t think credit direction matters much (unlike on some sites which seems to track that closely).