I use a 1GB/s connection with 16ms of ping time. This is very common in my region, because the providers offer internet connection over G-PON and DS-Lite, which drive the latency high.
sftp is a very latency critical protocol. With high latency the throughput falls to very small numbers. To overcome throughput issues, there were a series of patches in openSSH called “HPN-SSH”. With those patches in place, you can reach full speed in sftp transfers also with high latency lines. Those patches are active in most linux distribution out of the box. The patches need to be active on the file recipient side (its a buffer of incoming packets) to speed up the transfer.
With tools using openSSH and this extension (winSCP) upload and download reach about 450 Mbit/s on my line per connection.
In Duplicati without those patches the upload reach 400 Mbit/s per connection, but the download (restore backup) only get 35 Mbit/s.
So its joy to do the backups and I love duplicati, but when it comes to restoring the pain begins. Think most people don’t test this beforehand.
So my Questions and Request is:
Is it possible to enable this HPN buffer also in ssh.net in Duplicati?
Is it possible to use openSSH in Duplicati?
And I want also to vote for the multiple parallel download feature in Restore as mentioned in forum and Issuetracker a lot of times ;).
Yes, I try it there as well.
I think ssh .net is used to keep things simpler in cross platform compatibility.
But using openSSH would be more standard and faster.
How much effort would it be to implement an openSSH backend? For example for FTP there are also multiple backends included.
Would a pull request be accepted if I implement an openSSH backend and send a pull request?
Of course an experienced .net developer could do this much quicker as I as an c++ developer who never touched .net.
Parallel download on restore would speed up all backends.
That would be a great feature!
Edit: This post got flagged as “Spam”, but I have absolutely no idea why. Please contanct me if you think I have written something wrong and tell me what. Maybe it was because of the missing space in ssh .net what made a link out of this?
I really don’t know (I’m just a volunteer on the forum and have only submitted minor pull requests, not one of the main developers). I do think there’s an effort to avoid platform-specific libraries and APIs as much as possible and stick with .NET.
Thats why Im asking beforehand.
Would be a shame all that work and then get the pull rejected.
Can you please ask the developers if:
A Backend only for windows would be accepted?
Maybe, because there is a generic all platform ssh .net solution that would be ok, because its “only” a performance boost on windows then?
openSSH with binary’s for the different platform’s would be accepted?
I can only test Windows & Linux binary’s, would it be possible for the devs to do the rest work for mac and the other supported plattforms?
And sorry to ask this, Im new in this forum.
Why my original post got hidden by community flag? Have I done something wrong?
No, the system automatically flagged them because it thought the behavior was suspicious (new user posting many URLs). I have restored your messages since there’s nothing wrong with them!
Good to hear there is already a pull request in on the sshnet project. I think we’ll need to wait until it’s merged and a new release is made available. I could be wrong but I don’t think we use the sshnet source directly in Duplicati, but rather their released library.
Very interesting, thank you for this tipp!
I will test this tomorrow, because it needs a bit of research how to setup rclone in duplicati. Its only a single line and no user/url/port/… stuff.
I created a config “remote” with the sftp target in rclone.
Also i created a config “local”, but i have no idea what this is used for or what it shall include. Local config ist only this:
[local]
type = local
Then i create a duplicati job. I enter the “remote” and “local” and set the path to rclone binary in advanced options. Check connection works and he create the new directory for my backup.
But then when i run the job, its about 1 minute with status “Verify backend-data”. Then a error message appear: “Error while running rclone Backend test” without any further information.
Any idea whats wrong?
What is the reason of this “local” configuration? Maybe know about why I need it would make it easier to create one with propper content.
Need more error output. I don’t know why it’s quiet. Try watching About → Show log → Live → Retry.
Author’s explanation was earlier under Rclone local repository link (was wrong but is now fixed).
Basically author didn’t know rclone could just use a local path. You might have seen how copies work:
put is copyto {local_repo}:{filename} {remote_repo}:{remote_path}/{remotename}
get is copyto {remote_repo}:{Path.Combine(this.remote_path, remotename)} {local_repo}:{filename}
The local folder used is typically a temporary folder, and temporary files mostly begin with a dup- prefix. tempdir will let you set the folder as you like, but what puzzles me is why put works but get does not…
If you don’t want to do another backup to get a get, the Verify files button will try to download some.
First for all who search – because there are a lot of threads about it.
The local config does not need to include a path at all. Config file should just look like this:
[local]
type = local
It was not working, because duplicati does not save the “rclone-executable” value. It is used when you check connection, but after saving, the field is just empty. However often I enter the path there, it gets cleared on each save. So only way is, to include rclone binary in your system path. Don’t forget to restart duplicati service afterwards for this change to take effect.
Now the speed Results.
Backup works, but is half the speed of normal sFTP. This is because there are huge gaps between the uploads of the files. It looks like it is not running in parallel like the sftp backend.
Restore is much faster. Instead of 35 Mbit/s read from ssh . net it can read with about 170 Mbit/s. But also file by file with huge gaps in between. So best would be backup with sFTP client and restore with rclone.
Next I will look into compiling ssh . net with the already submitted changes for speed….
Here it seems to save it (check Commandline GUI Target URL box), and even use it.
BUT on re-entry to job Edit it seems to not read it back, therefore the Save loses it…
This reminds me of another bug on Destination Advanced options. I can’t find now…
I think the read-back-into-GUI is JavaScript. I don’t do JavaScript. Do you happen to?
Gaps may also occur if uploading gets ahead of packaging for uploads in temp folder.
If uploading gets behind, you can see roughly 50 MB dup-* files waiting for an upload.
You know the rhythm of your system though. It’s too bad there’s no GUI queue status.
Sometimes gaps-activity can be looked at with About → Show log → Live → Profiling
If it’s doing prep work for next upload, that would show. Also look for rclone processes.
It’s not built into each backend, as far as I know. I thought it worked on everything. The help has:
--asynchronous-concurrent-upload-limit (Integer): The number of concurrent
uploads allowed
When performing asynchronous uploads, the maximum number of concurrent
uploads allowed. Set to zero to disable the limit.
* default value: 4
I did a local build and copied the “Release/net40/Renci.SshNet.dll” binary to the duplicati2 install directory where the other binary was (Make a backup of this file first!).
That’s it – now sFTP downloads are as fast as in WinSCP .
I can’t upload the file here.
If someone want to use it as well – write me a message.
But be careful – it’s completely without any warranty and only tested a few times by myself.
I used it with Duplicati 2.0.6.3_beta_2021-06-17 – but it’s a unchanged file since 2021_01_24.
.