Operation has timed out

No NAT? I’m surprised Residential even sells public IPs. OK, so NAT’s not an issue, but a firewall might still be.

For isolating the problem, I was thinking maybe a file transfer from friend to you via another means might help.

https://en.wikipedia.org/wiki/Comparison_of_WebDAV_software will let you copy a specific file at a known time.

I’m not sure how you remote-access friend’s house, but if through ssh, you could probably also try scp or sftp.

Duplicati does have a throttling option which I hope isn’t on, and if on, I would hope it can never trip over itself.

Although it’s harder, if you wish to debug at the TCP level, note that sequence numbers on the wire are often shown as relative sequence numbers (values are far smaller, and easier to inspect). Whatever tool you used probably has a switch to turn this feature off, and then the view should be the same seen from both the sides.

Once you have a capture, quite often the capture format is portable or convertible for use with a different tool. For example, tcpdump is often available, small, but old. Wireshark adds lots of advanced analysis capabilities.

7.5. TCP Analysis might help, but notice it’s in Chapter 7. Advanced Topics. Internet has resources too…

If you’re doing this as a learning opportunity, great! There’s a chance it might wind up at this level regardless.

I am using Wireshark to debug. I did change to absolute sequence numbers to check. Using relative sequence numbers things line up.

The test last night was between friend’s house and my server. I did another test this morning between my house and friend’s house. The sequence numbers line up. There is a SonicWall firewall in front of my server, I’m guessing that is what is changing the sequence numbers.

My house has dynamic IPs, but I use a Linux box as my router, so I have the ability to see everything without NAT.

I do have the duplicati throttling on for my server, but not for the other hosts. Given that I have the problem on other hosts that don’t have the throttling, I’m fairly certain that isn’t the issue.

Where you able to test with --no-auto-compact=true to see if the timeout errors went away?

Also, it was a bit unclear to me what happened when you took the computer to your friend’s house - did it work OK there? Note that depending on his router, if you were inside his network but still referencing his external address your requests might have gone out of this router then right back in thus potentially suffering the same issue you’re seeing from outside the network.

Is it possible his router itself is blocking or throttling things for some reason?

The computer that is at my friends house started at my house. There on the local LAN it worked. Now that it’s at my friends house I’m getting timeouts. Looking at the packet trace with wireshark it appears there are some retransmissions going on. I would expect that TCP would handle the retries or duplicati would.

I have not tried with no-auto-compact, however if I turn off verification the timeouts go away.

I haven’t inspected all of the settings of my friends router, but he hasn’t made changes to it to do any throttling. I’ve run the speed tests, results above, at his house and the speeds are what we expect for the service that he is paying for.

The details of TCP are almost always an OS thing. The app sees reliable transport (moving as fast as it can). Retransmissions, like duplicate ACKs, can be normal. It depends on how many. “netstat -s” can give the stats.

I’m not sure if your systems support it, but one can sometimes find bottlenecks from where the backlogs land.

Limiting factors might include the data sender, the network, and the data receiver. If, for example, a backlog is visible at the server on a transfer of a big file from friend’s house to Duplicati, the “netstat -a” (change to suit) “Send-Q” column would show data trying to leave sender system. On the client (at your house), see “Recv-Q”. Backlogs at the receiver create back pressure on TCP, which can backlog sender TCP, and then sender app.

If you have a netstat that can show the TCP queues, it’s possibly easier than Wireshark, however Wireshark does seem to have some throughput analysis tools. Or just get a WebDAV client, try a download, and time it.

It’s a shame there’s no issue yet to take to the ISP, however I’m far from convinced Duplicati is the bottleneck for transfer rate. Transfer spacing perhaps, but that should be visible in the log files or the live server logs…

I don’t think I’m hearing actual timings, but is it correct that a full 50MB dblock cannot transfer in 10 minutes? While larger files might be possible (e.g. a big dlist file with millions of files named), I don’t think you use any.

Example references:

Application Analysis Using TCP Retransmissions, Part 1

Network Packet Loss, Retransmissions, and Duplicate Acknowledgements

A full 50MB dblock can transfer in less than 10 minutes. I just tested downloading a 50MB dblock file with my browser and found that the first 40MB came down in about 2 minutes. The last 10MB seems to have gotten stuck. Looking at netstat -t on the duplicati server I see:
tcp6 0 994776 XXX.XXX.XXX.XXX:443 XXX.XXX.XXX.XXX:40564 ESTABLISHED

However I’m not connected via ipv6, rather I’m connected by ipv4. The client side, doing the download, shows zero queue and shows “tcp” as the first column.

When a duplicati test is done, I see more than 1 file being downloaded. Typically one of the files gets the 5 retries and then duplicati gives the warning.

Something else I just tried was to change the retry time to be 1 minute, instead of 10 seconds. However that doesn’t appear to be used here as I just saw the retry messages still every 10 seconds.

I just copied the same file via ssh and it completed without issue. When looking at the netstat output I see “tcp” in the first column. So it seems that apache is marking itself as a tcp6 connection even though the connection is ipv4. The server doesn’t have any IPv6 addresses, I’ve turned it off at this point due to firewall issues.

Seeing this tcp6 connection and the difference between ssh and apache I modified the apache configuration to force it to only try and talk ipv4 (using Listen 0.0.0.0:port). This didn’t fix things. I can see the change in the output of netstat, however I still see the send queue getting large and the connection getting stuck.

One more reply before going to work.

I did some more testing and found that downloading a dblock file directly from apache (no php in the mix) works with http, but times out with https. So it appears to be related to https somehow.

More testing this morning. I tried putting apache on port 444. No change. http on port 444 works, https on port 444 times out.
I setup an ssh tunnel to the localhost:443 on the server and pulled a file via https over the tunnel and that works. I ran the same tunnel to the external IP of the server and that worked too.