Duplicati to Minio through proxy

Hi!

I’ve been testing Duplicati+Minio a while and it really looks very good! But I have a question. I’m testing to backup from one site to another over the internet through a proxy ssh tunnel.

Creating an outgoing tunnel that goes to the Minio machine (Minio and sshd on the same machine) and hitting test connection in Duplicati backup job goes well. I see the requests in ssh log. But when I run the actual job itself and not only “test connection” it doesn’t seem to go through the tunnel. And with a dynamic tunnel I can’t even get that far.

Is there anything I can do to make Duplicati run over the proxy ssh tunnel? I noticed som advanced option regarding s3 and proxy, how can these options be used and is this solvable?

Thanks in advance!

/Magnus

  1. Is there a reason you’re not using the TLS support built into Minio?

  2. It sounds like you’re already using SSH, is there a reason you’re using SFTP?

  3. If it were me I’d likely end up creating a backup job that does what I want such as:
    a. Select “S3 Compatible” Storage Type
    b. Set a Server of "locahost:12345
    c. CHECK the Advanced Option of --s3-ext-forcepathstyle to disable the AWS DNS lookup (not needed if using Duplicati 2.0.2.6 or higher)
    d. Export the backup “As Command-line”

  4. Make a script in which I do something like:
    a. run ssh -D 12345 me@myserver (I’d use & to put it in the background, if running in Linux)
    b. use the command line from above to execute the backup
    c. close SSH session…somewhow…

  5. Schedule the script with my favorite scheduler.

You could probably do something similar directly in the Duplicati GUI with --run-script-before and --run-script-after parameters, but I’m not really sure how the SSH tunnel would stay open or get closed in that scenario…

Thank you for your reply!

The reason for looking at an ssh proxy tunnel is that I will only have one single port open to the remote computer. So I thought I’d do everything I need, like remote administration of the computer and so on, through that port with a tunnel. The other option is setting up a “proper” VPN instead.

I have a local test backup server with Minio which I can run client backup jobs to with no problem when connecting to it “normally” without proxy. I’m setting up the jobs like you describe. I do everything through the Duplicati web page, didn’t know I could export jobs to command line, that’s cool!

With a bit more testing I get the job running through a normal forward ssh tunnel (-L). I was confused earlier since it seemed like the actual job wasn’t going through the tunnel but rather connecting directly to the test server. But then I started Minio to only accept connections from localhost and the job still ran fine so I was obviously just wrong and confused :smile: So that works.

On the server that is running sshd I started minio:

/opt/minio/minio server --address 127.0.0.1:9000 --config-dir /opt/minio/configs/main /backupstorage

And on the client:

ssh -v -L 9000:localhost:9000 minio@192.168.0.101

And run a backup job exactly as you describe with localhost:9000 as server.

So all is good, I can start additional tunnels to sshd on the server to do other things at the same time and of course run ssh shells.

But curious that I am I started looking at starting my local ssh tunnel as dynamic/proxy (-D). And I could without a problem run that at the same time as the normal forward tunnel was running as above and both work fine, for example testing firefox running through the proxy just for testing at the same time a backup job runs through the static tunnel. BUT I don’t understand how to set up the Duplicati backup job to run throught the proxy instead of the static tunnel. If it’s even possible?

Good job getting the static tunnel working just from my sketchy theoretical!
Though I’mm still a little confused how Minio via SSH tunnel is easier than SFTP (except maybe user setup) as they both work over the same single port.

Once you’re happy with your setup you might want to consider posting a #howto for other users. :slight_smile:

As far as static tunnel vs proxy I’m not really sure - I haven’t done much with SSH proxies but my GUESS is there nothing built into Duplicati to use them.

1 Like

No, Duplicati does not support proxies unfortunately:


But there is a workaround mentioned here:

@magnust, so in looking at those links it sounds like until it’s included in the code (if ever) adding the following to Duplication.CommandLine.exe.config or Duplicati.GUI.TrayIcon.exe.config (as appropriate) might do what you need.

Please let us know if you try this out. If it works we can put it in a #howto. :slight_smile:

<system.net>
  <defaultProxy>
    <proxy
      proxyaddress="http://[your proxy address and port number]"
      bypassonlocal="false"
    />
  </defaultProxy>
</system.net>

I’ve set up a static ssh tunnel which seem to work perfectly with Duplicati/Minio. When thinking about it I realized that a proxy was overkill for what I wanted to accomplish :slight_smile: Yeah, I needed to polish up my skills on ssh a little, hehe.

Currently I’m preloading the to be remote server locally before I ship it over to the of site location. Fancy name for “my friends house” :smiley:

Just a quickly copied raw todo list collected from different places and tested succesfully on Debian 9. If anyone is interested. Note: This is with an encrypted ssh tunnel and not an ssh proxy.

On Linux server with Minio:

sudo mkdir -p /opt/minio/configs/main
sudo mkdir -p /backupstorage/main   or somewhere else with lots of storage space
sudo chown -R <your backup user>:<your backup user group> /opt/minio/
sudo chown -R <your backup user>:<your backup user group> /backupstorage/
cd /opt/minio
wget https://dl.minio.io/server/minio/release/linux-amd64/minio
chmod +x minio

sudo nano /etc/systemd/system/minio-main.service
	[Unit]
	Description=Minio Main
	[Service]
	User=<your backupuser>
	ExecStart=/opt/minio/minio server --address 0.0.0.0:9000 --config-dir /opt/minio/configs/main /backupstorage/main
	[Install]
	WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable minio-main.service
sudo systemctl start minio-main.service	
sudo systemctl status minio-main.service
sudo tail -n 3000 /var/log/syslog | grep "minio"   NOTE: copy and save the two keys you see!

Now you need to create a “bucket” (storage area) in Minio for Duplicati to use. Surf to your Minio Linux box:
<ip number or fqdn for your backup server>:9000
and add a bucket. Easiest is to surf to this address while being on the same LAN so you don’t have to hassle with reaching it through a firewall. To reach it AT ALL from remote using ssh you MUST have port 22 open in the firewall at it’s location. To surf to the Minio web interface you need to have port 9000 open (you can choose another port when setting up Minio as above). If you can’t have any ports open whatsoever to the Linux/Minio server read my paragraph at the bottom of this post.

On windows computer with Duplicati to run backup job

  • Download and install 64x cygwin from www.cygwin.com
  • During install you choose and add two extra non default packages by searching for “ssh” and add:
    • autossh
    • openssh

As Administrator account, in cygwin terminal/shell write:

ssh-keygen -t rsa -b 4096
	choose blank passphrase
	save in /home/Administrator/.ssh/id_rsa

ssh-copy-id -i /home/Administrator/.ssh/id_rsa.pub <your servers backup user>@<ip number or fqdn for your backup server>

TEST 1. You should now auto login without password using your local secret key:
ssh -i /home/Administrator/.ssh/id_rsa.pub <your servers backup user>@<ip number or fqdn for your backup server>
exit

TEST 2:
autossh -M 0 -i /home/Administrator/.ssh/id_rsa -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -N -L 9000:localhost:9000 <your servers backup user>@<ip number or fqdn for your backup server> -v
ctrl-c

CREATE AS SERVICE so the tunnel is always up:
cygrunsrv -I Autossh-MINIO-tunnel -p /usr/bin/autossh -a "-M 0 -i /home/Administrator/.ssh/id_rsa -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -N -L 9000:localhost:9000 <your servers backup user>@<ip number or fqdn for your backup server>" -e AUTOSSH_GATETIME=0

In windows services edit the Autossh-MINIO-tunnel service:
	Run as user Administrator + it's password

In Duplicati when setting up your backup jobs

On the destination page:
    Storage type: S3 compatible
    Server: Custom server url
    <your servers backup user>@localhost -p 9000
    Don't enable Use SSL
    Use the two keys you copied earlier from the server as AWS Access keys
    Enable s3-ext-forcepathstyle under advanced settings

When all is running fine you can secure your ssh server a bit by editing this ON THE BACKUP LINUX server:

sudo nano /etc/ssh/sshd_config
	PermitRootLogin no
	PasswordAuthentication no

.

Oh, one more thing. If you want to backup many different computers to your remote Linux/Minioserver you don’t have to set up and run a ssh tunnel on each computer! You just set up one computer to run the tunnel and have all the others use that tunnel. You just need to add the option to share the tunnel by adding this to the command line starting autossh “-o GatewayPorts=yes” like this:

TEST:
autossh -M 0 -i /home/Administrator/.ssh/id_rsa -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o GatewayPorts=yes -N -L 9000:localhost:9000 <your servers backup user>@<ip number or fqdn for your backup server> -v
CREATE AS SERVICE so the tunnel is always up:
cygrunsrv -I Autossh-MINIO-tunnel -p /usr/bin/autossh -a "-M 0 -i /home/Administrator/.ssh/id_rsa -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o GatewayPorts=yes -N -L 9000:localhost:9000 <your servers backup user>@<ip number or fqdn for your backup server>" -e AUTOSSH_GATETIME=0

.

Oh right, just another thing: If you have no option whatsoever to have ANY open ports in the firewall at the remote site where you placed your Linux/Minio server, you can let it “call home” to you instead. This will not be stopped by any normally set up firewall (outgoing traffic allowed, incoming restricted). You will however have to open a port in your local firewall. To get the remote computer to “call home” you just set up one more ssh tunnel (a “reverse” tunnel) quite similar to above but this time you start the client ssh tunnel on your Linux/Minio server and have the ssh server deamon running on your local computer. This is how I run it actually. If anyone is interested just give me a holler.

.

As someone said, it might be easier to set up with just sftp instead of Minio. But where is the fun in that?

.

/Magnus

1 Like

Great! Once you’ve got it all up and running through the tunnel please stop by the #howto section and share with others what steps it took to get going. :+1:

I edited in a long todo install list while you posted. See above :smiley:

Do you want me to copy paste this into a new thread in howto?

1 Like