Backups to SFTP Not Closing Connection

#1

I recently upgraded from 2.0.4.5 to 2.0.4.22 to test some of the fixes to an issue I was seeing. Since then I’ve noticed that connections to my SFTP server are not closing after the backup completes. My assumption is that this has something to do with the changes for multiple connections to the target backup.

These open connections will stay open indefinitely (days) until I run out of allowed connections on my SFTP server and backups start failing. I’ve reduced asynchronous-concurrent-upload-limit down to 2 which seems to halve how many connections are left open after each backup.

From what I’ve observed, the backup starts and two connections are opened. As the backup continues two more connections are opened and there are 4 concurrent connections (despite being set at 2), and ultimately two of the 4 are closed and the remaining two are left open indefinitely.

I can do some more troubleshooting and try to figure out at exactly which stage things are happening if that helps but thought I’d see if anyone else has noticed the issue.

#2

I hadn’t noticed it (I don’t use SFTP generally), but I can reproduce it. Also confirming 2.0.4.5 doesn’t do it.

C:\>netstat -ano | findstr :22 | findstr ESTABLISHED
TCP [::1]:22 [::1]:65411 ESTABLISHED 3584
TCP [::1]:65411 [::1]:22 ESTABLISHED 18476

is a way to count connections (if going to localhost as above, expect connections to show mirror-images). Limited tests did seem to show it as upload related, e.g. test of downloading all files had no open leftovers.

This seems like a serious limit, correct? If (not confirmed) it was in v2.0.4.16-2.0.4.16_canary_2019-03-28 then I guess nobody picked up canary and noticed it. There’s a wish-it-was-seen-sooner backend-specific bug in “FTP (Alternative)” in Problems after upgrade to 2.0.4.21 which requires a UNIX-style FTP server…

Although I hope SFTP can be fixed, there’s some time pressure from Amazon Cloud Drive access ending. There are other SFTP servers, but free ones sometimes have limits like yours, or only allow personal use.

Characterizing the problem further will help figure out where it lies. Even better would be if you read code…

#3

This might affect all backends. See the issue below:

#4

I was worried about that possibility, but I’d tested “FTP” (not “FTP (Alternative)” and everything closed OK. Monitoring was with Sysinternals Process Explorer keeping watch on the TCP/IP status of FileZilla Server. Your very recent finding (thanks for finding it!) raises my worry level again though. Some of the backends seem difficult to test (FTP was easy, so I tested…). Using rclone (which I don’t) might be highly obvious…

#5

Proposed fix here:

#6

Thanks @ts678 and @warwickmm!

I’ll be sure to test the fix out as soon as it’s available. As of right now I can get around the issue by restarting the service. Appreciate the quick response and action.

#7

The fix has been merged and should appear in the next canary release.

1 Like