Couldn't connect to minio server when proxy configured in linux

ts678 · December 8, 2020, 12:44am

I’m going to guess that proxies that require authentication aren’t supported, but I have no proxy so I played with nc -l -p <various> to first see that s3-ext-proxyport could make it do proxy-like things there. I saw this:

GET http://192.168.0.102:9000/bucket?max-keys=1000&prefix=folder%2F HTTP/1.1
User-Agent: aws-sdk-dotnet-45/3.3.104.25 aws-sdk-dotnet-core/3.3.103.37 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_10.0.19042.0 ClientSync
host: 192.168.0.102:9000
X-Amz-Date: 20201207T213137Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=id/20201207/us-east-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=cf877c5d589e74898832c58a6e2c52859f4e4d26348994ddef202949a07a4763
Proxy-Connection: Keep-Alive

on fake proxy port 9001, which I guess is how it asks the proxy to go connect to port 9000 for actual get.
I’m not particularly familiar with proxies at the protocol level, but I’d guess you got a 401 error+challenge.

You’d have to run a network trace as suggested previously, if you want to see what it actually sent back. That information might be useful if you want to try the workaround I’ll give. It authenticates but with limits.

Are you a developer or highly technical? You posted this in Developer category, so I’ll go talk about code.
I didn’t write this, its author is rarely available, and developers tend to do more on Issues, which is where this might have to go as an enhancement request. There are lots of those in line, along with other issues.

Duplicati seems to query the third-party SDK from Amazon in order to find out what options it can accept:

github.com

duplicati/duplicati/blob/7d958adb8a71b334834ce42ecc9571c78e8a259e/Duplicati/Library/Backend/S3/S3Backend.cs#L381-L390


var exts =
    typeof(Amazon.S3.AmazonS3Config).GetProperties().Where(x => x.CanRead && x.CanWrite && (x.PropertyType == typeof(string) || x.PropertyType == typeof(bool) || x.PropertyType == typeof(int) || x.PropertyType == typeof(long) || x.PropertyType.IsEnum))
        .Select(x => (ICommandLineArgument)new CommandLineArgument(
            "s3-ext-" + x.Name.ToLowerInvariant(),
            x.PropertyType == typeof(bool) ? CommandLineArgument.ArgumentType.Boolean : x.PropertyType.IsEnum ? CommandLineArgument.ArgumentType.Enumeration : CommandLineArgument.ArgumentType.String,
            x.Name,
            string.Format("Extended option {0}", x.Name),
            string.Format("{0}", x.GetValue(defaults)),
            null,
            x.PropertyType.IsEnum ? Enum.GetNames(x.PropertyType) : null));

Most options have pretty simple types that are accepted by the filter. You might say you just need a string for username, and a string for password. That was once the case, but it was deprecated, then obsoleted.

AmazonS3Config now has ProxyCredentials which means it takes a System.Net.ICredentials which isn’t one of the simple types that most things use and which Duplicati can deal with. I think the setting code is:

github.com

duplicati/duplicati/blob/7d958adb8a71b334834ce42ecc9571c78e8a259e/Duplicati/Library/Backend/S3/S3AwsClient.cs#L58-L74


foreach (var opt in options.Keys.Where(x => x.StartsWith("s3-ext-", StringComparison.OrdinalIgnoreCase)))
{
    var prop = cfg.GetType().GetProperties().FirstOrDefault(x =>
        string.Equals(x.Name, opt.Substring("s3-ext-".Length), StringComparison.OrdinalIgnoreCase));
    if (prop != null && prop.CanWrite)
    {
        if (prop.PropertyType == typeof(bool))
            prop.SetValue(cfg, Utility.Utility.ParseBoolOption(options, opt));
        else if (prop.PropertyType.IsEnum)
            prop.SetValue(cfg, Enum.Parse(prop.PropertyType, options[opt], true));
        else if (prop.PropertyType == typeof(int))
            prop.SetValue(cfg, int.Parse(options[opt]));
        else if (prop.PropertyType == typeof(long))
            prop.SetValue(cfg, long.Parse(options[opt]));
        else if (prop.PropertyType == typeof(string))
            prop.SetValue(cfg, options[opt]);
    }

If you follow the code differently, or wish to do the enhancement yourself, pull requests can be submitted.

A workaround might be to configure Duplicati to use an Rclone destination, which appears to understand standard environment variables such as curl might use. I did see a 2019 note that it only does basic auth, however I have no idea what proxy server is in use and what sorts of authentication it could be looking for.