Amazon S3 storage class GLACIER_IR ignored by WebUI

[yscialom]

(note: this thread is a duplication of an issue I opened on github: #4882 as I’ve been told the forum is the preferred way of communication for Duplicati’s team)

Unwanted Behaviour Description

Environment info

• Duplicati version: 2.0.6.3_beta_2021-06-17
• Operating system: Ubuntu 20.04
• Backend: Amazon S3
• Misc: Running from Docker image linuxserver/duplicati:2.0.6

Description

Trying to backup files to Amazon S3 storage class GLACIER_IR is:

• successful using the command line;
• defaulting to standard storage class using the web UI.

Steps to reproduce

Using Duplicati’s Web UI, on freshly installed instance

Steps

1. Click “+ Add backup”
2. Select “Configure a new backup”, click “Next >”
3. Set “1. General” as pleases you, click “Next >”
4. Set:

• Storage Type: S3 Compatible
• Storage class: Custom storage class and enter GLACIER_IR
• Set other fields accordingly to your Amazon setup
• Click “Next >”

5. Set “3. Source Data”, “4. Schedule” and “5. Options” as pleases you, click “Save”.
6. Go back to the web UI “Home” page, and “Run now” your newly created backup.
   See issue-4882-webui-backup.json for an export of a backup created with the web UI and with GLACIER_IR as custom storage class.

Actual result

1. Objects on Amazon’s servers are created with the “Standard” storage class.
2. Backup exports do not mention the custom storage class, neither in JSON exports nor when trying to edit the backup using the web UI (defaults to default).

Screenshots

Expected result

1. Objects on Amazon’s servers are created with the “Glacier Instant Retrieval” storage class.

Using Duplicati’s command line, on freshly installed instance

Steps

Run

mono /app/duplicati/Duplicati.CommandLine.exe backup \
     "s3s://(redacted)/test-06?s3-server-name=s3.eu-west-3.amazonaws.com&s3-location-constraint=eu-west-3&s3-storage-class=GLACIER_IR&s3-client=aws&auth-username=(redacted)&auth-password=(redacted)" \
     /source/media/ \
     --backup-name="GLACIER_IR test6" \
     --dbpath=/data/glacier-ir-test6.sqlite \
     --encryption-module=aes \
     --compression-module=zip \
     --dblock-size=50mb \
     --passphrase=(redacted) \
     --keep-versions=2 \
     --disable-module=console-password-input

Screenshots

Debug log

No relevant log found.

Analysis

It appears to be a purely web UI bug: its backend does not list GLACIER_IR as a possible storage class to its front-end:

HTTP/2 POST /api/v1/webmodule/s3-getconfig 
(headers removed for simplicity)

s3-config: "StorageClasses"
---
HTTP 200
(headers removed for simplicity)
{
	"Status": "OK",
	"Result": {
		"(default)": "",
		"Standard": "STANDARD",
		"Infrequent Access (IA)": "STANDARD_IA",
		"One Zone Infrequent Access (One Zone IA)": "ONEZONE_IA",
		"Glacier": "GLACIER",
		"Deep Archive": "DEEP_ARCHIVE",
		"Reduced Redundancy Storage (RRS)": "REDUCED_REDUNDANCY",
		"Intelligent Tiering": "INTELLIGENT_TIERING"
	}
}

[gpatel-fr]

Hello

maybe it’s because the project manager is not really interested ? quote: “I generally recommend not using Glacier with Duplicati”, see here:

Problems getting started with duplicati

That is a “first install” issue that has not been fixed yet. Once you get a real error (i.e. force stop a running backup to trigger it), you will get a popup with the error, and clicking “Dismiss” will make it work like normal again. Glacier is a bit problematic, due to the restore process and it has not been fully automated (nor integrated in Duplicati). It works by using the S3 life-cycle rules to move files to Glacier. This means that Duplicati can no longer verify that all is working as e…

[yscialom]

Hello fpatel-fr, thank you for your response.

GLACIER_IR is kind of an hybrid storage class; in my opinion it is a glacier only in name. Duplicati works really well with it… when using the command line.

[gpatel-fr]

Yes you are right, it’s not as if Glacier is not included in the hard-coded list, so it’s probably because it’s a rather recent addition to Amazon storage and Duplicati has not been updated to follow the change.

[ts678]

Welcome to the forum @yscialom

Introduce Amazon S3 Glacier Instant Retrieval storage class and a new setting in S3 Object Ownership to disable ACLs for bucket and the objects in it. was 3.7.172.0 commit. Duplicati’s .dll looks like 3.3.104.25. Searching in a hex editor found the Unicode strings for a sample of what you caught, but not the new one.

@yscialom

Duplicati works really well with it… when using the command line.

This is good to hear although it’d be better if someone could update the .dll and we could do a release. Google Cloud Storage has what sounds like a similar option which is hot storage that simply sets up the price to steer the usage, for example Glacier Instant Retrieval has a minimum storage charge of 90 days.

Cost optimisation on wasabi is a similar situation of at least trying to get some benefits back from pricing. They have a few other catches, but I’m not going to promise I found all the subtleties to Amazon’s prices.

@yscialom

I’ve been told the forum is the preferred way of communication for Duplicati’s team

It’s more active for commentary, but it’s not a bug tracker whereas GitHub Issues is. Now we have both…

What’s very low recently are developers and (for now at least) volunteers to commit PRs and do releases. Sometimes people work on issues with personal relevance. Care to see if you can build with a new SDK?

Duplicati only exists and improves thanks to the efforts of volunteers in all areas. More are most welcome.