Duplicati ERROR and restarting: [ERROR] FATAL UNHANDLED EXCEPTION: System.UriFormatException: Invalid URI: Invalid port specified

Support
1

Hello,

I am having an issue where Duplicati has a fatal error and then restarts itself. I mainly notice this when I try to preform initial backups that are larger than 100GB. I am unsure if this is just something that is eventually happening to me given enough time or if larger sizes are to blame.

Once Duplicati restarts itself I can press the Run Now button in the WebUI and it continues with the backup successfully. If the backup is larger it may crash again at which point I need to start the backup again but it does finish successfully.

Future backups seem to work without this issue, I am not sure if that’s because they don’t take nearly as long and therefore this issue doesn’t come up or if its because they are also a lot smaller only backing up incrementally.

I’ve read the other thread that mentioned this issue but the solution isn’t anything concrete; checking my zabbix graphs during the backup I can see that the machine is not running out of resources.

Looking for suggestions,
Thanks

My environment:
Rocky Linux release 8.9
Mono JIT compiler version 6.12.0.107
Duplicati - 2.0.7.100_canary_2023-12-27
Remote SFTP Backup Storage
Using the duplicati web UI

The error:

Feb 08 03:47:51 systemd[1]: Started Duplicati web-server.
Feb 08 08:02:12 duplicati-server[2052957]: [ERROR] FATAL UNHANDLED EXCEPTION: System.UriFormatException: Invalid URI: Invalid port specified.
Feb 08 08:02:12 duplicati-server[2052957]:   at System.Uri.CreateThis (System.String uri, System.Boolean dontEscape, System.UriKind uriKind) [0x0007b] in <5ac5513a1ab7495cab9a4cbe1e35d78f>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at System.Uri..ctor (System.String uriString) [0x00014] in <5ac5513a1ab7495cab9a4cbe1e35d78f>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at HttpServer.HttpClientContext.OnRequestLine (System.Object sender, HttpServer.Parser.RequestLineEventArgs e) [0x00095] in <bed89f1655ee48029f6d6812f54c58ad>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at (wrapper delegate-invoke) System.EventHandler`1[HttpServer.Parser.RequestLineEventArgs].invoke_void_object_TEventArgs(object,HttpServer.Parser.RequestLineEventArgs)
Feb 08 08:02:12 duplicati-server[2052957]:   at HttpServer.Parser.HttpRequestParser.OnFirstLine (System.String value) [0x00172] in <bed89f1655ee48029f6d6812f54c58ad>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at HttpServer.Parser.HttpRequestParser.Parse (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x0010f] in <bed89f1655ee48029f6d6812f54c58ad>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at HttpServer.HttpClientContext.OnReceive (System.IAsyncResult ar) [0x00059] in <bed89f1655ee48029f6d6812f54c58ad>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at System.Net.Sockets.SocketAsyncResult+<>c.<Complete>b__27_0 (System.Object state) [0x0000b] in <5ac5513a1ab7495cab9a4cbe1e35d78f>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <5dfd69ae4e3b402db546d8ded6fc755e>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <5dfd69ae4e3b402db546d8ded6fc755e>:0
Feb 08 08:02:12 duplicati-server[2052957]:   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <5dfd69ae4e3b402db546d8ded6fc755e>:0
Feb 08 08:02:12 systemd[1]: duplicati.service: Main process exited, code=exited, status=255/n/a
Feb 08 08:02:12 systemd[1]: duplicati.service: Failed with result 'exit-code'.
Feb 08 08:02:12 systemd[1]: duplicati.service: Service RestartSec=100ms expired, scheduling restart.
Feb 08 08:02:12 systemd[1]: duplicati.service: Scheduled restart job, restart counter is at 568.
Feb 08 08:02:12 systemd[1]: Stopped Duplicati web-server.
Feb 08 08:02:12 systemd[1]: Started Duplicati web-server.
2

Is this going up faster than you expect? The original post sounded like it was occasional.
Is there anything interesting in journalctl -u duplicati or such (I’m not expert in it)?

Not hugely out of date, and I’m not sure about RPM suppliers. Where is this mono from?

In case it matters, what browser, and is it on localhost or remote? IPv4? IPv6 potential?

Did you also manage to have a TrayIcon talking to Duplicati server. It uses HTTP API too.

Generically speaking, after you get a handle on the crash rate and maybe the causes, try
removing HTTP traffic to see if a source of the offending HTTP can be isolated somehow.

Which port do you mean to use? The default is 8200, but one can set it up as one prefers.
Running lsof -i :8200 or similar will show what’s connected so you can test using less.

Are there any proxies or other fancy networking gear in here, or is it direct HTTP connect?
If unencrypted, you could probably use tcpdump or wireshark to get a really good view…

Unfortunately I don’t think the HTTP server itself does logging, and expertise is too scarce.
Even an expert would probably want to know what was going in that might be causing this.

3

So far from what I can tell this only happens when I try to start new backups or have a reason to recreate current backups that are over 100GB. Given that this crash happens in waves during these initial backups and I have to keep starting the initial backup until it is complete and how long the server has been currnetly on for I would say that it sounds resonable it has restarted over 500 times at this point.

journcalctl just shows a lot of repeats of the same error i originally posted, you can see it here: journalctl -u duplicati-- Logs begin at Mon 2024-01-29 08:00:32 UTC, end at Fr - Pastebin.com

Ill try updating mono and see if it fixes my issue, at the time I installed it using the following (but as you have mentioned I is a few versions out of date):

	curl https://download.mono-project.com/repo/centos8-stable.repo | tee /etc/yum.repos.d/mono-centos8-stable.repo
	yum install mono-devel -y

Im using the latest version of Firefox 122
I am accessing the duplicati webui remotely with IPv4
I do not have a trayicon or anything duplicati related installed on the machine I am using to interact with the duplicati webui, just my browser.

I connect to the duplicati webui which is a remote machine in a different location and it is backing up over SFTP to another remote machine in a another location.

I have not changed the port and it is still 8200

There are no proxies setup, I can connect directly via the IP:8200 or a domain I have setup a record for duplicati.domain.com:8200 (the hostname is set in the duplicati webui).

4

You’re the only one who could try to spot a correlation. To me, the time to failures just look variable.

Not every last test has been tried, e.g. monitoring network, or closing all GUI connections to server.

So far, this does not ring any bells for me. I’d wonder if it was resources, but you checked that a bit.

Maybe somebody else has an idea. These intermittent problems can be challenging to troubleshoot.

5

This looks like a problem in the connection between web UI and server, so I don’t think the backup size or backend has anything to do with it.

Do the crashes only occur when you actively use the UI? In that case:

The invalid port error might be caused by an incorrectly formatted URL where colons are not escaped properly, or it might miss a forward slash after the port number. That would be an error in the JavaScript of the UI. This would be a request visible in your browser, so you could check in your browser debugging tools (in the network tab) what requests are made right before the error occurs.

If you can reproduce the error somewhat often, you should open the browser debugger before you run the backup that might trigger the problem. Either way, there shouldn’t be any problems for scheduled backups since they don’t involve the UI.

Do these errors also occur while you are not using the UI (there are 500+ restarts in the log):

Another reason might be that your server is publicly accessible, so anyone might make some requests to test out the limits of the system. It seems there is some request that can crash the simple http server used by Duplicati.

In that case, I would strongly suggest that you use a VPN and isolate the duplicati port from the public Internet, or use SSH port forwarding when you need to access the UI. I would not trust the security of the Webserver on a public port, it is pretty outdated.

6

might be crashing on the last line below. If so, I wonder if bad input was from outside, or maybe an occasional inside accident, e.g. if what looks like the server address serializes to something invalid.
Although the web server reportedly doesn’t listen on an IPv6 address, colons in one would be fatal.

7

Yes that code looks very fragile. The UriPath has to start with a slash, otherwise it crashes. Also I don’t know whether ipv6 is printed properly using [] by the address to string.

8

Similar but worse might include netstat -a | grep :8200 to see who else might be connected. Hopefully there isn’t an active attack too, but one definitely doesn’t want to be right on the Internet.
Duplicati.Server.exe shows --webservice-port option that you could put in /etc/default/duplicati.
Moving to a different obscure port would lose any forgotten leftovers that might still be connecting.

9

Thank you for all of the suggestions.

Looking now it does appear to also happen when no backups are being run and I am not connected to the webui

Looking at the timestamps of the restarts I do see that it is happening even with I am not using the UI

I will isolate the duplicati port from the public internet and monitor is this continues to happen.

Will post my results in the future.

1 Like
10

This seems to be the cause of the issue I was facing. While I didn’t manage to catch the exact time or connection that was causing duplicati to crash and restarts I have since changed it so that duplicati only listens on localhost and use a password protected revese proxy to access it from the public internet.

Since doing this it has not crashed and restarted; going forward I guess we’ll see but so far so good. I have also ran one of my backups that takes a longer amount of time and it finished without stopping so that didn’t appear to be the cause.

Happy and hopeful that this is solved for me now :ok_hand:

1 Like