Saw an error in duplicati about the updater failing which prompted me to check the website. Appears to be down at the moment.
duplicati.com appears to be down.
Log:
System.Net.WebException: The remote server returned an error: (502) Bad Gateway.
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
at Duplicati.Library.AutoUpdater.UpdaterManager.CheckForUpdate(ReleaseType channel)
Sent a PM to the administrator. Oddly, updater is not complaining for me, and web site is slightly there however not up enough to do anything useful. Digital Ocean is not reporting any general outages now.
still down. I had seen this a few days ago but it did come back after 1 or 2 hours but now it has already been longer.
The doc site is gone as well. Thankfully the OAuth site (https://duplicati-oauth-handler.appspot.com) is distinct and is still working.
Thanks for reporting this. It has been up/down for a week or so, but now it collapsed.
I will migrate to a new server ASAP.
Yes, that was designed with high availability in mind, where the others are more cost-focused solutions.
Restarted the server, and now it responds again. Will do the migration more slowly now.
That’s weird. I think it’s an entirely different large-scale hosting infrastructure, per readthedocs.io.
Regardless, seems fine here now, and I’m not sure what it was doing before.
Sure but probably the web server on duplicati.com is doing the redirection to readthedocs.io, so accessing the canonical URL docs.duplicati.com is not working. If I had used
it would have worked obviously.
Yes, there is a small redirect there to readthedocs.
Migration to a new server is now completed. Will be a bit wonky until the DNS is flushed everywhere and then I close the old server.
it seems to point to a new IP but it don’t work (although it answers to ping and it seems to have a SSH service running). Last web archive dates from 5 days ago.
It works for me, responding to IP 134.122.74.64
Indeed now it does. But when I posted it did not - it was at this IP address, I confirmed with wget.
It was half-down for me, although working now.
C:\>nslookup www.duplicati.com
Server: UnKnown
Address: 192.168.86.1
Non-authoritative answer:
Name: www.duplicati.com
Address: 134.122.74.64
C:\>ping duplicati.com
Pinging duplicati.com [134.122.74.64] with 32 bytes of data:
Reply from 134.122.74.64: bytes=32 time=103ms TTL=49
Reply from 134.122.74.64: bytes=32 time=106ms TTL=49
Ping statistics for 134.122.74.64:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 103ms, Maximum = 106ms, Average = 104ms
Control-C
^C
C:\>curl http://www.duplicati.com
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.23.3</center>
</body>
</html>
C:\>curl https://www.duplicati.com
curl: (28) Failed to connect to www.duplicati.com port 443 after 21152 ms: Timed out
date -u
dim. 12 févr. 2023 17:34:30 UTC
(py311) gerard@j5005:~$ LANG=en_EN wget https://duplicati.com
--2023-02-12 18:34:31-- https://duplicati.com/
Resolving duplicati.com (duplicati.com)... 134.122.74.64
Connecting to duplicati.com (duplicati.com)|134.122.74.64|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12644 (12K) [text/html]
Saving to: 'index.html'
index.html 100%[========================================================================================================================================>] 12.35K --.-KB/s in 0.03s
2023-02-12 18:35:29 (357 KB/s) - 'index.html' saved [12644/12644]
(py311) gerard@j5005:~$ LANG=en_EN wget https://duplicati.com
--2023-02-12 18:35:40-- https://duplicati.com/
Resolving duplicati.com (duplicati.com)... 134.122.74.64
Connecting to duplicati.com (duplicati.com)|134.122.74.64|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12644 (12K) [text/html]
Saving to: 'index.html.1'
index.html.1 100%[========================================================================================================================================>] 12.35K --.-KB/s in 0.08s
2023-02-12 18:38:21 (152 KB/s) - 'index.html.1' saved [12644/12644]
it seems that the main page is timing out on subsequent loadings and finally displaying the data but too late (1 minute first time, 3 second time) for a typical browser so all I get with it is a failure page.
Port 443 again. Below is from Linux comparing to port 80:
$ while true; do nc -v -w 5 134.122.74.64 443; done
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
nc: connect to 134.122.74.64 port 443 (tcp) timed out: Operation now in progress
^C
$ while true; do nc -v -w 5 134.122.74.64 80; done
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
Connection to 134.122.74.64 80 port [tcp/http] succeeded!
^C
$
A Windows Wireshark trace showed port 443 not even responding to the initial TCP connection SYN.
EDIT:
Before switching to nc, I did get Linux telnet to connect to port 443 – once, but it’s downhill from there.
EDIT:
Port 80 responded at least once like it did the other day, although more slowly. Look at the times here:
$ curl --trace-ascii - --trace-time http://www.duplicati.com
13:25:02.361311 == Info: Rebuilt URL to: http://www.duplicati.com/
13:25:02.381870 == Info: Trying 134.122.74.64...
13:25:02.381933 == Info: TCP_NODELAY set
13:25:02.482837 == Info: Connected to www.duplicati.com (134.122.74.64) port 80 (#0)
13:25:02.482956 => Send header, 81 bytes (0x51)
0000: GET / HTTP/1.1
0010: Host: www.duplicati.com
0029: User-Agent: curl/7.58.0
0042: Accept: */*
004f:
13:25:30.009802 <= Recv header, 32 bytes (0x20)
0000: HTTP/1.1 301 Moved Permanently
13:25:30.009926 <= Recv header, 22 bytes (0x16)
0000: Server: nginx/1.23.3
13:25:30.009983 <= Recv header, 37 bytes (0x25)
0000: Date: Sun, 12 Feb 2023 18:25:27 GMT
13:25:30.010047 <= Recv header, 25 bytes (0x19)
0000: Content-Type: text/html
13:25:30.010205 <= Recv header, 21 bytes (0x15)
0000: Content-Length: 169
13:25:30.010263 <= Recv header, 24 bytes (0x18)
0000: Connection: keep-alive
13:25:30.010322 <= Recv header, 38 bytes (0x26)
0000: Location: https://www.duplicati.com/
13:25:30.010385 <= Recv header, 2 bytes (0x2)
0000:
13:25:30.010424 <= Recv data, 169 bytes (0xa9)
0000: <html>
0008: <head><title>301 Moved Permanently</title></head>
003b: <body>
0043: <center><h1>301 Moved Permanently</h1></center>
0074: <hr><center>nginx/1.23.3</center>
0097: </body>
00a0: </html>
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.23.3</center>
</body>
</html>
13:25:30.010654 == Info: Connection #0 to host www.duplicati.com left intact
$
@kenkendk Still down, though behavior has been varying. Could you please investigate the situation?
I don’t know of any good ways to remotely check system activity and actual versus allowed processing.
FWIW time test in last post took 8 minutes to respond around 90 minutes ago, and 11 seconds recently.
The nc test continues to work on port 80 and fail on port 443. Maybe there’s some TCP queue problem.
TCP SYN Queue and Accept Queue Overflow Explained
Scaling Linux Services: Before accepting connections
a possibility could be that the server is under attack (denial of service).
If it’s the case, your tests are not doing much good 
You mean those that finished like this? 
Since we’re on this again, I’ll post more links in case this is a shared CPU. I found top shows “steal”.
How To Monitor CPU Use on DigitalOcean Droplets
What percentage of CPU steal is consider normal/acceptable?
Basically I can’t tell if server is underpowered (maybe needs upgrade) or overloaded, including DoS.
Or maybe something just broke in the software, and I’m far from expert in what sort of thing to check.
I’m thinking it might be load > power. An ssh to it was very slow to quiz me about the new fingerprint.
BTW without packet capture, look for SYN_SENT in netstat -an or lsof -i @www.duplicati.com
One would like the TCP connection to go to ESTABLISHED, but it needs the server to give a reply…
At least the check for updates seems to be working, although there’s more redundancy to that.
C:\ProgramData\Duplicati\duplicati-2.0.6.104_canary_2022-06-15>Duplicati.Library.AutoUpdater.exe check
No updates found
Anyone who wants a latest-Beta 2.0.6.3 download can use this GitHub site until server is back:
Trying PM to server admin. From another diagnostic view method, here’s Linux showing a successful connection and disconnection to port 80 from telnet, and one to port 443 that isn’t getting a response.
# tcpdump -i any -n host 134.122.74.64
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
07:32:06.432880 IP 10.0.2.15.43430 > 134.122.74.64.80: Flags [S], seq 3804073452, win 64240, options [mss 1460,sackOK,TS val 3994389539 ecr 0,nop,wscale 7], length 0
07:32:06.537325 IP 134.122.74.64.80 > 10.0.2.15.43430: Flags [S.], seq 564685825, ack 3804073453, win 65535, options [mss 1460], length 0
07:32:06.537374 IP 10.0.2.15.43430 > 134.122.74.64.80: Flags [.], ack 1, win 64240, length 0
07:32:12.193371 IP 134.122.74.64.80 > 10.0.2.15.35396: Flags [R.], seq 549453826, ack 4174870670, win 65535, length 0
07:32:19.993352 IP 10.0.2.15.43430 > 134.122.74.64.80: Flags [F.], seq 1, ack 1, win 64240, length 0
07:32:19.993864 IP 134.122.74.64.80 > 10.0.2.15.43430: Flags [.], ack 2, win 65535, length 0
07:33:06.359771 IP 10.0.2.15.35608 > 134.122.74.64.443: Flags [S], seq 3092628054, win 64240, options [mss 1460,sackOK,TS val 3994449466 ecr 0,nop,wscale 7], length 0
07:33:07.360510 IP 10.0.2.15.35608 > 134.122.74.64.443: Flags [S], seq 3092628054, win 64240, options [mss 1460,sackOK,TS val 3994450466 ecr 0,nop,wscale 7], length 0
07:33:09.376523 IP 10.0.2.15.35608 > 134.122.74.64.443: Flags [S], seq 3092628054, win 64240, options [mss 1460,sackOK,TS val 3994452482 ecr 0,nop,wscale 7], length 0
07:33:13.568719 IP 10.0.2.15.35608 > 134.122.74.64.443: Flags [S], seq 3092628054, win 64240, options [mss 1460,sackOK,TS val 3994456675 ecr 0,nop,wscale 7], length 0
07:33:21.760818 IP 10.0.2.15.35608 > 134.122.74.64.443: Flags [S], seq 3092628054, win 64240, options [mss 1460,sackOK,TS val 3994464867 ecr 0,nop,wscale 7], length 0
07:33:37.889129 IP 10.0.2.15.35608 > 134.122.74.64.443: Flags [S], seq 3092628054, win 64240, options [mss 1460,sackOK,TS val 3994480995 ecr 0,nop,wscale 7], length 0
07:34:11.936443 IP 10.0.2.15.35608 > 134.122.74.64.443: Flags [S], seq 3092628054, win 64240, options [mss 1460,sackOK,TS val 3994515042 ecr 0,nop,wscale 7], length 0
^C
13 packets captured
13 packets received by filter
0 packets dropped by kernel
#
Thanks for reporting. I do get a notice appx. every 12h if the site is not responding.
It looks like something was leaking memory and that caused it to be unstable.
I have expanded a bit, removed some processes and rebooted.