I am testing asymmetric gpg-keys and have removed the private key for security reasons. It will be stored somewhere else (in the safe). It will only be imported, when restore is needed. So if any hacker breaks into my backup-server, he cannot restore anything.
Nevertheless: My testresults shows that the restore of the files are working, even if the logfile says “No secret key found”. When I look in the restored folder all files are restored.
This shouldn’t work.
I am using the descriped advanced options:
"–gpg-program-path=/usr/bin/gpg
–gpg-encryption-command=–encrypt
–gpg-encryption-switches=–recipient “xyz@xxxjkhadslkh.com”
–gpg-decryption-command=–decrypt
The gnupg-key was created and trusted for xyz@xxxjkhadslkh.com.
More information:
apt list duplicati
Listing… Done
duplicati/now 2.0.7.1-1 all [installed,local]
On Linux 22.04.3
Hello @ts678 , your help is appreciated.
All the forum web search you did I found already and followed and tried to reproduce. The “gpg-encryption-switches” is also described there so.
Also I followed the approach, that the gpg-agent could be the problem and limited the max-cache-ttl to 30 seconds, default-cache-ttl to 30 seconds, but it doesn’t work.
Backup works from command-line, but restore does not work, even when I am importing the deleted secret key back.
What I am wondering is, that the GUI always warns me, that I want to use a weak passphrase.
That is true, because my passphrase was chosen, when I generated the gpg-pair from linux-command-line.
I assume, there is a bug in Duplicati with external gpg and asymmetric encryption. It seems that nobody has tested it.
It’s not much used, and unless automated test exists (I don’t know), that’s possible, but past test had
It’s context-dependent. Please describe how you are running. From a shell? If so, quoting is incorrect.
If you are doing it from the GUI and using the three-dot-menu Edit as text, it wouldn’t have dashes and also wouldn’t need any special quotes to protect from the shell because there isn’t a shell around.
Since you run gpg and I don’t, do you know if gpg itself requires any double quoting on --recipient? There are example on the Internet without it, so I’m not sure why the solution there decided to quote…
EDIT 1:
In GitHub issue, it sounds like it was from someone else, and I see a report it worked, and one it didn’t.
Asymmetric is not the usual way gpg is meant to run here, and seemingly the developer hadn’t used it.
EDIT 2:
Let me mention the GitHub post directly. Perhaps that will bring advice from somebody who’s tried this.
There’s quite a bit there, including the point that Duplicati’s backup normally needs to read data back, which makes me wonder if you turned that off. It reminds me of the problem that cold storage gets in.
Is this all from the command line? The GitHub issue people were using the GUI, which possibly didn’t remove the leading double dashes at the time, but does now. So you might have made a quotes error.
EDIT 4:
so if you just toss this into a shell, --gpg-encryption-switches=–recipient "xyz@xxxjkhadslkh.com", possibly Duplicati treats as --gpg-encryption-switches=–recipient
which will break decryption, right?
EDIT 5:
Having embedded spaces in a string that should be kept together is a risk in any command line usage.
Are you determined to use the command line? The GUI doesn’t have a shell, so option entry is simpler.
EDIT 6:
A counter-argument is The gnupg-key was created and trusted for xyz@xxxjkhadslkh.com.
When I test something on the command line like the input you show, it splits wrong, and tells me that:
System.IO.IOException: Backup aborted since the source path “xyz@xxxjkhadslkh.com” does not exist. Please verify that the source path exists, or remove the source path from the backup configuration, or set the allow-missing-source option.
so there must be something different in the invocation. Mine was similar to yours, but on one line and adding --encryption-module=gpg. Maybe you should try manually decrypting files to see if it works.
If you can get it by hand, and also get the right arguments into Duplicati, it seems like it ought to work.
If you want to see the invocation of /usr/bin/gpg by Duplicati argument by argument, you can try script
#!/usr/bin/bash
echo Command and arguments
for word in "$@"; do echo "$word"; done
prog="/usr/bin/gpg"
echo Running "$prog" "$@"
exec "$prog" "$@"
or something similar. Make it executable, have --gpg-program-path point to it, and watch the output.
Hello @ts678
why is it not much used? Assume, that a hacker hacks every server, even the backup server: Wouldn’t it be nice, that the secret key is not on this server and only the public key for encryption? Our goal is to store the whole backup asymmetric encrypted in the cloud.
To your Edit 1: Yes, that links in Github I also found already. And no: GPG was made for asymmetric enryption. It was intentionally made to store public keys on public servers so someone could encrypt files or mails specific for someone, who has the private/secret key for this public key. This symmetric encryption was new for me and I heard it the first time related here with duplicati. Always using GPG with asymmetric encryption.
To your Edit 2: I read this already. Somewhere he said he did it successfull, but the ticket is still “open” and I found no documentation on the readocs of duplicati for gpg and their parameters.
To your Edit 3: I did both. I tested it from GUI and from CLI. It is not working also with the double dashes.
To Edit 4: It was --gpg-encryption-switches=–recipient \"xyz@xxxjkhadslkh.com\" with backslashes.
Edit 5 and 6: I will see if I can reproduce it, but i t m I tent to not test duplicati furthermore as it does not meet our goals with simple asymmetric encryption.
I don’t think a user poll has ever been conducted as to “why”, but you can see the actual usage at https://usage-reporter.duplicati.com/ where the backups yesterday break down their encryption as
None 60067
AES 113601
GPG 276
and there’s no breakdown of symmetric versus asymmetric. I’ve also noted how Duplicati backup can read files, however that can be worked around with some reliability penalty, and maybe asymmetric is also workable, as some people have gotten it working, haven’t said it stopped, but aren’t helping us…
Some people worry about ransomware which deletes backups, even asymmetric encrypted ones. Typically Duplicati is recommended to run on the system being backed up, so if that gets hacked, asymmetric encrypted backups don’t keep hackers out of source files which are directly readable.
Asymmetric encryption is slower too. Web search asymmetric symmetric encryption performance. Reports include some pretty extreme differences such as 1000 times. OK for mail, bad for backup.
If you can get it working at least from command line, you can do your own benchmark on big files.
Some people may also want to restore from backups. I’m assuming private-key-in-safe stops that.
GUI should preferably use the dropdown, with no extra quoting in text field unless there is a reason.
Manual page for gpg doesn’t give me a clear reason, just --recipient name, so quotes seem odd.
Similarly here, escaped quotes seem odd. Why was it done? Maybe there’s a reason I haven’t seen.
Your choice. I suspect it can work if you put in the effort to set it up. If you look elsewhere, good luck.
Checking some competing backups (not including commercial ones), this looks like a rare capability. Support asymmetric backups #187 was opened to collect use cases, but I don’t think feature exists, unlike Duplicati where the feature partially exists, but seems to require setup which so far is elusive.
You’ll see some of the same arguments I gave, such as encryption not preventing a backup deletion. Backup programs are definitely starting (barely) to pick up object lock which helps block such attack.
There’s probably not much emphasis on preventing reading backups, as one can just read sources.
The purpose of the script I gave was to serve as a wrapper around /usr/bin/gpg to see how it’s used, meaning that if it looks contrary to its own documentation. fix the Duplicati setup to use it as needed.
It works fine here on my first test. I didn’t use all the mentioned setup steps. Some defaults seemed OK. Some others I just thought I’d try omitting. Possibly there’s a problem with that which I’ll discover later…
I ran gpg gen-key, and answered prompts. In Duplicati, I set Encryption: GNU Privacy Guard, external. Advanced options got gpg-encryption-switches of --recipient<space><email> without quoting, also:
was override of the default, and interestingly it made the password entry fields on first screen vanish. When restoring, I typed in the password when GUI (possibly gpg directly?) requested it. Without that password, or without the private keys in ~/.gnupg/private-keys-v1.d, the restore failed after its retries.
Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.
default of false might be making you think blocks were restored from gpg, but they weren’t. Testing restore while Source is accessible can do this, just as an attacker can get to Source.
I think there’s always some remote data used at the end to fix the file attributes, but by then, casual observation would see all the files there. Closer look might find dates etc. not correct.
As I see the above worry has come up on GitHub, let me strengthen case for source data reads again (as if the cited documentation isn’t enough). I’ll move the actual backup then try a restore. Because Duplicati complains if backup files vanish, I turn off check with no-backend-verification:
If this flag is set, the local database is not compared to the remote filelist on startup. The intended usage for this option is to work correctly in cases where the filelisting is broken or unavailable.
Restore result is my test file was restored, and the backup log suggests the reason how it came:
2024-03-15 22:08:40 -04 - [Verbose-Duplicati.Library.Main.Operation.RestoreHandler-MissingTargetFile]: Target file does not exist: /tmp/short.txt
2024-03-15 22:08:40 -04 - [Verbose-Duplicati.Library.Main.Operation.RestoreHandler-FilePatchedWithLocal]: Target file is patched with some local data: /tmp/short.txt
and this was before the first attempt at any destination file access, which of course did not work. Restore using local data is what happened here. Nothing weird in gpg, which wasn’t even used.
Another possible reason is (I think) it’s more likely to be on Linux (my Linux Mint included it).
Linux itself is not much used compared to Windows. The usage reporter has those stats too.
This wasn’t soon done, so I read the man page and took a try. First I got an opinion from file:
so far so good, but this is only a rough identifier, and isn’t decrypting. So try gpg decrypt next:
$ gpg --decrypt duplicati-ba45614824b5148259acebd8f2b9447d3.dblock.zip.gpg | hexdump -C | head
gpg: encrypted with 3072-bit RSA key, ID 95B21BB849A2735A, created 2024-03-15
"<name and address>"
00000000 50 4b 03 04 14 00 00 00 08 00 9d 7e 6f 58 56 ab |PK.........~oXV.|
00000010 64 db 74 00 00 00 92 00 00 00 08 00 00 00 6d 61 |d.t...........ma|
00000020 6e 69 66 65 73 74 7b bf 7b 7f b5 52 58 6a 51 71 |nifest{.{..RXjQq|
00000030 66 7e 9e 92 95 91 8e 92 73 51 6a 62 49 6a 8a 92 |f~......sQjbIj..|
00000040 95 92 91 81 91 89 81 b1 a1 69 88 a1 a5 a9 91 a9 |.........i......|
00000050 65 94 92 8e 92 6b 5e 72 7e 4a 66 5e 3a 50 b6 b4 |e....k^r~Jf^:P..|
00000060 24 cd 02 28 e2 94 93 9f 9c 5d 9c 59 95 aa 64 65 |$..(.....].Y..de|
00000070 08 52 6f 00 15 f2 48 2c ce 00 aa 0a f6 70 34 32 |.Ro...H,.....p42|
00000080 35 03 aa 73 cb cc 49 45 17 73 2c 28 80 db ac 64 |5..s..IE.s,(...d|
00000090 a4 67 a0 67 ae 67 a4 54 0b 00 50 4b 03 04 14 00 |.g.g.g.T..PK....|
Looks like gpg was happy, and turned it into a .zip file, just as it should have done. Grab it:
$ file dblock.zip
dblock.zip: Zip archive data, at least v2.0 to extract, compression method=deflate
$ unzip -v dblock.zip
Archive: dblock.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
146 Defl:N 116 21% 2024-03-15 15:52 db64ab56 manifest
193 Defl:N 127 34% 2024-03-15 15:52 dc24559d KNwaScU6PPXCJ-qqiZEVCh8vT_DnYFeM-jz1TKecZk8=
5 Defl:N 7 -40% 2024-03-15 15:52 3bb935c6 8sobtsfpB9Btr-Roflefznazfk6Tt2BQItpS5szCb9I=
-------- ------- --- -------
344 250 27% 3 files
which looks just right. The source file is 5 bytes, and I’m surprised it got compressed, but it did.
$ od -c /tmp/short.txt
0000000 t e s t \n
0000005
$ ls -ln /tmp/short.txt
-rw-rw-r-- 1 1000 1000 5 Mar 15 22:08 /tmp/short.txt
Timestamp is still at the restore time because the restore didn’t run long enough to set file dates.
When Duplicati fails with an error, be quite distrustful that all restored well, even if it said it did…
Hi @ts678 ,
you have worked a lot on the weekend with my issue with very interesting results. Most interesting is the local restore, when “No secret key found” and local datafiles are still available. I think, it should be obvious, that I don’t want the “copied” file from original source to the chosen restore-folder (e.g. /tmp/…).
When I have created the my backup, I have copy and paste this and added my prior gpg full-gen key in there with my mailaddress.
When I click on “Export into command line” I get this (beside other commands):
When I have these Restore Options in the GUI, I don’t expect, that it will copy them from the original files. I expect, that it will take them from the backup-source by default.
Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.
The main reason, why I don’t test duplicati furthermore is, that the options related to gpg are not described in the advanced-options-page of Duplicati. see Link here
There is only one option mentioned: encryption-module with only aes.
I cannot use a software, which is not well documented in that point.
Documentation is not something I directly work in, but maybe you can find a better alternative.
Consider my other suggestions that gpg is slow, and solves neither direct access nor deletion.
