--list-broken-files question

Support
1

I’m running the cli to test the list-broken-files command. The command looks like this (S3-compatible storage)

/Applications/Duplicati.app/Contents/MacOS/duplicati-cli list-broken-files \ ${S3BUCKET} --auth-username="${S3S3ACCESSID}" --auth-password="${S3S3ACCESSKEY}" \ --s3-ext-forcepathstyle="True" --accept-specified-ssl-hash="${S3S3SSLHASH}" --dbpath="${LOCALSQLITEDATABASE}" \ --passphrase="${ENCRYPTIONPASSPHRASE}" --full-result"

Three questions:

  1. I’m still getting the Enter encryption passphrase: prompt, why?
  2. I’m now looking at a mono process that has been taking up 100% of a CPU core for 30 minutes. This is a rather small test backup set (1.4GB). I’m also seeing no network traffic at all for this mono process.

Is this normal?

Note: I do not expect to find any broken file in this particular backup

2

I’d try without quotes

Edit: Use --disable-module=console-password-input in your command. That fixes #1

3

It doesn’t. I still get the prompt. This is my command in the shell, now. Note: those quotes do not matter, these are removed by the shell.

/Applications/Duplicati.app/Contents/MacOS/duplicati-cli list-broken-files \
  ${S3BUCKET} --auth-username="${S3S3ACCESSID}" --auth-password="${S3S3ACCESSKEY}" \
  --s3-ext-forcepathstyle="True" --accept-specified-ssl-hash="${S3S3SSLHASH}"
  --dbpath="${LOCALSQLITEDATABASE}" \
  --passphrase="${ENCRYPTIONPASSPHRASE}" --disable-module=console-password-input \
  --full-result
4

What Duplicati version is this?

If you put an echo to the left of the whole current line, does the resulting command line look OK to you?

Have you tried any other commands in duplicati-cli on this backup? Do they all have the same problem?

The original post had an extra double quote at its end which isn’t shown in the latest post). What was it?

Technically ${S3BUCKET} would be better off double-quoted, because word splitting follows substitution.

5

Ahum. Notice those S3S3 strings? They should have been just S3 of course. In the example above, there is one continuation \ missing. This is the one I have now:

/Applications/Duplicati.app/Contents/MacOS/duplicati-cli list-broken-files \
  "${S3BUCKET}" --auth-username="${S3ACCESSID}" --auth-password="${S3ACCESSKEY}" \
  --s3-ext-forcepathstyle="True" --accept-specified-ssl-hash="${S3SSLHASH}" \
  --dbpath="${LOCALSQLITEDATABASE}" \
  --passphrase="${ENCRYPTIONPASSPHRASE}" \
  --full-result

The output now is:

No certificates found, you can install some with one of these commands:
    cert-sync /etc/ssl/certs/ca-certificates.crt #for Debian based systems
    cert-sync /etc/pki/tls/certs/ca-bundle.crt #for RedHat derivatives
Read more: http://www.mono-project.com/docs/about-mono/releases/3.12.0/#cert-sync
The supplied option --auth-username is not supported and will be ignored
The supplied option --auth-password is not supported and will be ignored
The supplied option --s3-ext-forcepathstyle is not supported and will be ignored
No broken filesets found in database, checking for missing remote files

Backend not supported: https

Hmm. Why not?

6

What would be a good command to use for a test that only touches the local computer?

7

Hmm, protocol s3 instead of https of course.

S3BUCKET="s3://storage-fqdn:port/bucket/"
STORAGE="${S3BUCKET} --auth-username=${S3ACCESSID} --auth-password=${S3ACCESSKEY} --s3-ext-forcepathstyle=true --accept-specified-ssl-hash=${S3SSLHASH}"

/Applications/Duplicati.app/Contents/MacOS/duplicati-cli list-broken-files \
  ${STORAGE} \
  --dbpath="${LOCALSQLITEDATABASE}" \
  --passphrase="${ENCRYPTIONPASSPHRASE}" \
  --full-result
8

I’ve found out there is an issue here: duplicati somehow wants URL-translated input on the command line. E.g. a + in a pass phrase has to be given as %2B on the command line when part of the s3 URL. And s3:storage-fqdn:port has to be given as storage-fqdn%3Aport because it is given as an argument of the URL.

I found out that the command line export for a backup set puts everything in the s3 URL as ?-subcommands and does this translation. If I use that command line construct, it works. But using --auth-username etc., it does not. I tried doing the URL-style translation as passphrase etc, but it doesn’t work.

So with an ACCESSKEY of SOMEPASSWORDWITHA+ and the S3 server being storage-fqdn:9000 I need this as command line element for the storage:

STORAGE=“s3s://bucketname/?s3-server-name=storage-fqdn%3A9000&s3-location-constraint=&s3-storage-class=&auth-username= SOMEUSERNAME&auth-password=SOMEPASSWORDWITHA%2B&s3-ext-forcepathstyle=true&accept-specified-ssl-hash=SOMEHASH”

So, it seems --auth-username and such do not work properly if there is something in it that needs to to escaped for in an URL. That feels like a bug to me.

I would expect
duplicati-cli command s3://storage-fqdn:9000/bucketname --auth-username="SOMEUSERNAME" --auth-password="SOMEPASSWORDWITHA+"
to work on the command line, but it doesn’t. Putting everything as it is done as the command line export works (with URL escapes for special characters), though.

9

What version of Duplicati is this? updated AWSSDK to newest version #3472 might affect this because it looks like auth-password just flows into Amazon.S3.AmazonS3Client whose documentation doesn’t call for encoding.

If the updated library (e.g. in 2.0.4.5 beta) doesn’t take care of it, a Duplicati solution needs to make sure that encoding won’t be done twice by mistake for some use case. I don’t know the design or code well, or have S3.

To avoid this issue getting lost among old support requests, you could file it in Features category, or in Issues.

10

Version 2.0.3.3_beta_2018-04-02

Not updating clients yet because of the problems mentioned with speed. Added an issue.