# SFTP issue with high bandwidth high latency connections
**URL:** https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405
**Category:** Developer
**Created:** [November 17, 2021, 7:18am UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405 "2021-11-17T07:18:43Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 17, 2021, 7:18am UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/1 "2021-11-17T07:18:43Z")
I use a 1GB/s connection with 16ms of ping time. This is very common in my region, because the providers offer internet connection over G-PON and DS-Lite, which drive the latency high.
sftp is a very latency critical protocol. With high latency the throughput falls to very small numbers. To overcome throughput issues, there were a series of patches in openSSH called “HPN-SSH”. With those patches in place, you can reach full speed in sftp transfers also with high latency lines. Those patches are active in most linux distribution out of the box. The patches need to be active on the file recipient side (its a buffer of incoming packets) to speed up the transfer.
With tools using openSSH and this extension (winSCP) upload and download reach about 450 Mbit/s on my line per connection.
In Duplicati without those patches the upload reach 400 Mbit/s per connection, but the download (restore backup) only get 35 Mbit/s.
So its joy to do the backups and I love duplicati, but when it comes to restoring the pain begins. Think most people don’t test this beforehand.
**So my Questions and Request is:**
- Is it possible to enable this HPN buffer also in [ssh.net](http://ssh.net) in Duplicati?
- Is it possible to use openSSH in Duplicati?
- And I want also to vote for the multiple parallel download feature in Restore as mentioned in forum and Issuetracker a lot of times ;).
**Related links:**
HPN FaQ: [HPN-SSH FAQ | PSC](https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq/)
Think this is the SSH from Duplicati: [GitHub - sshnet/SSH.NET: SSH.NET is a Secure Shell (SSH) library for .NET, optimized for parallelism.](https://github.com/sshnet/SSH.NET)
---
### Author:  [@drwtsn32](https://forum.duplicati.com/u/drwtsn32)
#### Post date: [November 17, 2021, 1:45pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/2 "2021-11-17T13:45:41Z")
Duplicati uses [GitHub - sshnet/SSH.NET: SSH.NET is a Secure Shell (SSH) library for .NET, optimized for parallelism.](https://github.com/sshnet/SSH.NET) for SSH, so I think that’s where your idea would need to be implemented. I see that you already opened an issue for this over on that project. Will be interesting to see what developers come back with.
---
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 17, 2021, 2:03pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/3 "2021-11-17T14:03:45Z")
Yes, I try it there as well.
I think ssh .net is used to keep things simpler in cross platform compatibility.
But using openSSH would be more standard and faster.
How much effort would it be to implement an openSSH backend? For example for FTP there are also multiple backends included.
Would a pull request be accepted if I implement an openSSH backend and send a pull request?
Of course an experienced .net developer could do this much quicker as I as an c++ developer who never touched .net.
Parallel download on restore would speed up all backends.
That would be a great feature!
_Edit: This post got flagged as “Spam”, but I have absolutely no idea why. Please contanct me if you think I have written something wrong and tell me what. Maybe it was because of the missing space in ssh .net what made a link out of this?_
---
### Author:  [@drwtsn32](https://forum.duplicati.com/u/drwtsn32)
#### Post date: [November 17, 2021, 3:51pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/4 "2021-11-17T15:51:13Z")
> [@deduplicatigmx](#):
>
> Would a pull request be accepted if I implement an openSSH backend and send a pull request?
I really don’t know (I’m just a volunteer on the forum and have only submitted minor pull requests, not one of the main developers). I do think there’s an effort to avoid platform-specific libraries and APIs as much as possible and stick with .NET.
---
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 17, 2021, 4:12pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/5 "2021-11-17T16:12:22Z")
Thank you for your contribution .
Thats why Im asking beforehand.
Would be a shame all that work and then get the pull rejected.
Can you please ask the developers if:
A Backend only for windows would be accepted?
Maybe, because there is a generic all platform ssh .net solution that would be ok, because its “only” a performance boost on windows then?
openSSH with binary’s for the different platform’s would be accepted?
I can only test Windows & Linux binary’s, would it be possible for the devs to do the rest work for mac and the other supported plattforms?
And sorry to ask this, Im new in this forum.
Why my original post got hidden by community flag? Have I done something wrong?
---
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 17, 2021, 6:09pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/6 "2021-11-17T18:09:28Z")
I just found out there is already a Pull request in ssh . net for this feature!
>
>
> This PR changes just 2 constants:
> \- Increases maxPendingReads from 10 to 100
> \-… Increases socket send/receive buffer to 10 SSH packets (320K)
>
> This results in huge improvements in transfer speeds when using sftpClient.DownloadFile/UploadFile, in particular with fast local internet connections and remote servers with high latency. The results are even better when used together with PR #865 - speeds are now comparable with Filezilla.
>
> I tested on two servers: a relatively close one (Frankfurt) and a distant one (AWS-hosted SFTP Server). Here are the results:
>
> \`\`\`
> Server Ping Pending PR865 Iterations Average
> \------ ----- ------- ----- ------------------- ----------
> FRA 30ms 10 No 29.59 28.90 26.55 =\> 28.35 MB/s
> FRA 30ms 100 No 43.28 37.74 41.54 =\> 40.85 MB/s (144%)
> FRA 30ms 100 YES 65.38 67.76 66.97 =\> 66.70 MB/s (235%) \[link limit 500Mbps\]
>
> AWS 100ms 10 No 7.01 6.69 7.45 =\> 7.05 MB/s
> AWS 100ms 100 No 19.97 21.31 20.96 =\> 20.74 MB/s (294%)
> AWS 100ms 100 YES 25.50 26.36 26.22 =\> 26.02 MB/s (369%)
> \`\`\`
>
> The 500Mbps link is satisfyingly saturated when combining these 2 PRs! 😎
>
> \### Reasoning
> \*\*1. Increasing the socket send/receive buffer size\*\*
> This change has low impact. This buffer holds packets that have been received and are waiting to be processed by the application. In cases where the machine is temporarily bogged down, it may happen that the incoming packets are not processed quickly enough and the previous 2-packet buffer gets full, and a 3rd incoming packet then causes the connection to stall. Increasing the buffer to 10 packets reduces the probability of this scenario, and makes the transfer more smooth and resilient to hiccups in CPU usage.
>
> \*\*2. Increasing the maxPendingReads\*\*
> This setting controls the maximum number of SFTP Read Requests that are sent to the server and are yet unanswered. When a packet arrives at the client, it sends another request for the next packet. In other words, it's the maximum number of \_in-flight\_ data packets.
>
> Each SFTP data packet is 32KB in size, so maxPendingReads=10 means that there's a maximum of 320 KB in flight before the transfer stalls. With 100, this is raised to 3.2MB in flight.
>
> Modern internet connections are FAST. Transferring 320KB at 100Mbps takes about 30ms, and at 500Mbps it takes just 6.4ms (plus latency, ignoring details). This means that when talking to a server that is 100ms away this happens:
> \- SSH.Net sends 10 read requests
> \- connection is idle for 100ms, while the requests and replies are on their way
> \- ~100ms later, the 10 packets arrive: this takes about 10ms on a fast connection
> \- as the 10 packets arrive, SSH.Net sends 10 more requests
> \- the connection is now idle for ~90ms, waiting for the 2nd batch
> \- the loop repeats - the connection is idle up to 90% of the time!
>
> The numbers are not accurate and I'm ignoring lots of detail but you get the idea. 320KB of in-flight data is just not suitable for today's fast connections anymore. Raising it to ~3.2MB fixes this problem (for now), and restores performance on distant servers.
You just need to pull this to duplicati to dramatically increase performance.
---
### Author:  [@drwtsn32](https://forum.duplicati.com/u/drwtsn32)
#### Post date: [November 17, 2021, 6:40pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/7 "2021-11-17T18:40:54Z")
> [@deduplicatigmx](#):
>
> Why my original post got hidden by community flag? Have I done something wrong?
No, the system automatically flagged them because it thought the behavior was suspicious (new user posting many URLs). I have restored your messages since there’s nothing wrong with them!
Good to hear there is already a pull request in on the sshnet project. I think we’ll need to wait until it’s merged and a new release is made available. I could be wrong but I don’t think we use the sshnet source directly in Duplicati, but rather their released library.
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [November 17, 2021, 7:06pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/8 "2021-11-17T19:06:07Z")
> [@deduplicatigmx](#):
>
> the download (restore backup) only get 35 Mbit/s.
Maybe you can time an [rclone sftp](https://rclone.org/sftp/) download of a file. If fast, then test Duplicati [rclone](https://duplicati.readthedocs.io/en/latest/05-storage-providers/#rclone) storage type.
[How to avoid upload speed degrading with latency? #158](https://github.com/pkg/sftp/issues/158#issuecomment-391721905)
[Rclone 1.42 release](https://forum.rclone.org/t/rclone-1-42-release/5902)
> Fix slow downloads for long latency connections
---
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 17, 2021, 7:48pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/9 "2021-11-17T19:48:18Z")
Very interesting, thank you for this tipp!
I will test this tomorrow, because it needs a bit of research how to setup rclone in duplicati. Its only a single line and no user/url/port/… stuff.
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [November 17, 2021, 8:45pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/10 "2021-11-17T20:45:01Z")
> [@deduplicatigmx](#):
>
> Its only a single line and no user/url/port/… stuff.
All of that is configured into the rclone config in advance. Some notes on the setup for Duplicati:
[Rclone backend documentation](https://forum.duplicati.com/t/rclone-backend-documentation/2930)
[Rclone local repository](https://forum.duplicati.com/t/rclone-local-repository/8858)
and if you want code:
[https://github.com/duplicati/duplicati/tree/master/Duplicati/Library/Backend/Rclone](https://github.com/duplicati/duplicati/tree/master/Duplicati/Library/Backend/Rclone)
Note that it implements [IBackend](https://github.com/duplicati/duplicati/blob/a4a02ff884d59a964a0a9d9cdc3945db0560c8e6/Duplicati/Library/Interface/IBackend.cs#L28-L37) not [IStreamingBackend](https://github.com/duplicati/duplicati/blob/a4a02ff884d59a964a0a9d9cdc3945db0560c8e6/Duplicati/Library/Interface/IStreamingBackend.cs#L25-L30), so UI might be a little bit less featured.
>
If you feel current Duplicati manual could use clarification, [pull requests can be submitted to it](https://duplicati.readthedocs.io/en/latest/).
Actually, pull requests can be submitted to Duplicati too, and volunteers are very much sought.
---
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 18, 2021, 3:02pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/11 "2021-11-18T15:02:03Z")
Cant get rclone to work…
I created a config “remote” with the sftp target in rclone.
Also i created a config “local”, but i have no idea what this is used for or what it shall include. Local config ist only this:
```
[local]
type = local
```
Then i create a duplicati job. I enter the “remote” and “local” and set the path to rclone binary in advanced options. Check connection works and he create the new directory for my backup.
But then when i run the job, its about 1 minute with status “Verify backend-data”. Then a error message appear: “Error while running rclone Backend test” without any further information.
Any idea whats wrong?
What is the reason of this “local” configuration? Maybe know about why I need it would make it easier to create one with propper content.
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [November 18, 2021, 3:48pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/12 "2021-11-18T15:48:41Z")
> [@deduplicatigmx](#):
>
> Any idea whats wrong?
Need more error output. I don’t know why it’s quiet. Try watching About → Show log → Live → Retry.
> [@deduplicatigmx](#):
>
> What is the reason of this “local” configuration?
Author’s explanation was [earlier](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/10) under `Rclone local repository` link (was wrong but is now fixed).
Basically author didn’t know rclone could just use a local path. You might have seen how copies work:
`put` is `copyto {local_repo}:{filename} {remote_repo}:{remote_path}/{remotename}`
`get` is `copyto {remote_repo}:{Path.Combine(this.remote_path, remotename)} {local_repo}:{filename}`
The local folder used is typically a temporary folder, and temporary files mostly begin with a `dup-` prefix.
[tempdir](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#tempdir) will let you set the folder as you like, but what puzzles me is why `put` works but `get` does not…
If you don’t want to do another backup to get a `get`, the `Verify files` button will try to download some.
---
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 18, 2021, 3:48pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/13 "2021-11-18T15:48:47Z")
Ok I found the issue. It is working now.
First for all who search – because there are a lot of threads about it.
The local config does not need to include a path at all. Config file should just look like this:
```
[local]
type = local
```
It was not working, because duplicati does not save the “rclone-executable” value. It is used when you check connection, but after saving, the field is just empty. However often I enter the path there, it gets cleared on each save. So only way is, to include rclone binary in your system path. Don’t forget to restart duplicati service afterwards for this change to take effect.
**Now the speed Results.**
Backup works, but is half the speed of normal sFTP. This is because there are huge gaps between the uploads of the files. It looks like it is not running in parallel like the sftp backend.
Restore is much faster. Instead of 35 Mbit/s read from ssh . net it can read with about 170 Mbit/s. But also file by file with huge gaps in between. So best would be backup with sFTP client and restore with rclone.
Next I will look into compiling ssh . net with the already submitted changes for speed….
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [November 18, 2021, 4:32pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/14 "2021-11-18T16:32:16Z")
> [@deduplicatigmx](#):
>
> It was not working, because duplicati does not save the “rclone-executable” value.
Here it seems to save it (check Commandline GUI Target URL box), and even use it.
BUT on re-entry to job Edit it seems to not read it back, therefore the Save loses it…
This reminds me of another bug on Destination Advanced options. I can’t find now…
I think the read-back-into-GUI is JavaScript. I don’t do JavaScript. Do you happen to?
> [@deduplicatigmx](#):
>
> there are huge gaps between the uploads of the files.
Gaps may also occur if uploading gets ahead of packaging for uploads in temp folder.
If uploading gets behind, you can see roughly 50 MB `dup-*` files waiting for an upload.
You know the rhythm of your system though. It’s too bad there’s no GUI [queue](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#asynchronous-upload-limit) status.
Sometimes gaps-activity can be looked at with About → Show log → Live → Profiling
If it’s doing prep work for next upload, that would show. Also look for rclone processes.
> [@deduplicatigmx](#):
>
> It looks like it is not running in parallel like the sftp backend.
It’s not built into each backend, as far as I know. I thought it worked on everything. The help has:
```plaintext
--asynchronous-concurrent-upload-limit (Integer): The number of concurrent
uploads allowed
When performing asynchronous uploads, the maximum number of concurrent
uploads allowed. Set to zero to disable the limit.
* default value: 4
```
[https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/Operation/Backup/BackendUploader.cs](https://github.com/duplicati/duplicati/blob/master/Duplicati/Library/Main/Operation/Backup/BackendUploader.cs)
---
### Author:  [@deduplicatigmx](https://forum.duplicati.com/u/deduplicatigmx)
#### Post date: [November 19, 2021, 2:14pm UTC](https://forum.duplicati.com/t/sftp-issue-with-high-bandwidth-high-latency-connections/13405/16 "2021-11-19T14:14:16Z")
**YEAA! Succes. Now I can Restore with 350 Mbit/s instead of 35 Mbit…**
I Downloaded [GitHub - sshnet/SSH.NET: SSH.NET is a Secure Shell (SSH) library for .NET, optimized for parallelism.](https://github.com/sshnet/SSH.NET) on “master” branch Tag **2020.0.01** like it is used in Duplicati.
Then I modified only this patch – it changes only 3 valuesProcessing: Renci.SshNet.dll…
: [Improve SFTP performance on medium/high latency connections by zybexXL · Pull Request #866 · sshnet/SSH.NET · GitHub](https://github.com/sshnet/SSH.NET/pull/866/commits)
Be careful – the first commit includes an error which is fixed in the second commit.
No need to also include the testcase comits.
I did a local build and copied the “Release/net40/Renci.SshNet.dll” binary to the duplicati2 install directory where the other binary was (Make a backup of this file first!).
**That’s it – now sFTP downloads are as fast as in WinSCP 😀.**
I can’t upload the file here.
If someone want to use it as well – write me a message.
But be careful – it’s completely without any warranty and only tested a few times by myself.
I used it with Duplicati 2.0.6.3\_beta\_2021-06-17 – but it’s a unchanged file since 2021\_01\_24.