I’ve recently tried using duplicati with minio (S3 compatible storage). I want to backup multiple systems to the same bucket, so need to use a folder per system. I’ve configured my first system with a folder path and started a backup. This correctly creates the folder in my bucket and backed up the files. However when duplicati got to the verification step it is adding an extra slash to the request for the file to read. This results in the following error from minio
Jun 10 06:34:20 jps-offsite minio[3812]: Error: Object name invalid: jon-backup#jon-2019_2020-06-08//duplicati-20200609T000849Z.dlist.zip.aes
Jun 10 06:34:20 jps-offsite minio[3812]: 6: /home/jpschewe/.gocode/pkg/mod/github.com/minio/minio@v0.0.0-20200608193042-c1382c09d994/cmd/object-api-input-checks.go:74:cmd.checkListObjsArgs()
Jun 10 06:34:20 jps-offsite minio[3812]: 5: /home/jpschewe/.gocode/pkg/mod/github.com/minio/minio@v0.0.0-20200608193042-c1382c09d994/cmd/object-api-common.go:279:cmd.listObjects()
Jun 10 06:34:20 jps-offsite minio[3812]: 4: /home/jpschewe/.gocode/pkg/mod/github.com/minio/minio@v0.0.0-20200608193042-c1382c09d994/cmd/fs-v1.go:1268:cmd.(*FSObjects).ListObjects()
Jun 10 06:34:20 jps-offsite minio[3812]: 3: /home/jpschewe/.gocode/pkg/mod/github.com/minio/minio@v0.0.0-20200608193042-c1382c09d994/cmd/fs-v1.go:1379:cmd.(*FSObjects).ListObjectsV2()
Jun 10 06:34:20 jps-offsite minio[3812]: 2: /home/jpschewe/.gocode/pkg/mod/github.com/minio/minio@v0.0.0-20200608193042-c1382c09d994/cmd/bucket-listobjects-handlers.go:240:cmd.objectAPIHandlers.ListObjectsV2Handler()
Jun 10 06:34:20 jps-offsite minio[3812]: 1: net/http/server.go:2012:http.HandlerFunc.ServeHTTP()
How do I convince duplicati to not send this extra slash?
Edit your backup job and look at configuration page 2. Do you have a trailing slash on the “folder path” field?
There is no trailing slash. Although I just noticed it was using the Amazon SDK instead of the Minio SDK.
I figured out why I have the Amazon AWS SDK selected. Changing the value to Minio doesn’t save. Each time I change it, go directly to save or through the next buttons to save. I then go back and edit the backup again and it’s back to Amazon.
Interesting… what version of Duplicati are you using?
Hopefully someone with Minio experience can help…
I did a quick test and created a dummy backup job. I set the client library to Minio and saved it, and it seemed to stick just fine. When I went to edit the job it was still set to Minio SDK.
I changed it back and forth a few times, saving each time, and it always stuck. This is with the current development branch of Duplicati.
I’m using Duplicati version 2.0.5.1_beta_2020-01-18
I don’t see any errors in the web console. When I export the backup job to a JSON file I don’t see any reference to either SDK.
As far as the double slash goes. I went back and created a new backup and made sure to name the bucket the same as my username so that I don’t get the warning about the bucket not starting with my username. This time the backup did not produce the error on the minio server about file not found. However the backup has been sitting in “Verifying backend data” for over an hour.
I have logging to a file enabled at the information level. Here is the end of the log file. The current time is 6:33 on 6/11.
2020-06-10 21:38:38 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20200610T221926Z.dlist.zip.aes (38.12 MB)
2020-06-10 21:38:38 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed
2020-06-10 21:38:38 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00
2020-06-10 21:38:38 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider:
2020-06-10 21:38:38 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted:
2020-06-10 21:38:38 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete:
2020-06-10 21:38:49 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted
2020-06-10 21:38:49 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()
According to strace, it’s sitting in a mutex. So it appears to be a deadlock.
I got past the deadlock by recreating the backup.
However I’m still seeing the double slash. This is using the Amazon SDK. When I set the Minio SDK I get access denied when testing the connection. I have tried with s3-ext-forcepathstyle set and unset. So 2 issues:
- why am I getting the double slashes when Duplicati does a get operation?
- Why doesn’t the minio SDK connect?
I have just verified that the double slashes problem exists with canary release 2.0.5.107_canary_2020-05-26
Given that I’ve been able to reproduce this on 2 systems with the beta version and the canary version. I’ve filed a bug at S3 backend adds double slashes when verifying backend files · Issue #4225 · duplicati/duplicati · GitHub.
Thanks for doing that, it will help get some more eyes on the issue.
I haven’t had time yet but hopefully someone can help test.
