Kestrel is coming

Started work on integrating the Kestrel webserver onto my .net 6 branch (Fork? Starting to feel like a fork…)

Ended up replicating a large number of static properties into a “FIXMEGlobal” class that hopefully someday can be fixed with dependency injection.

As an aside I still can’t get over how nice it is to do work in .netCore where I click build and the whole thing just works. Run publish and it is basically shippable. VSCode understands it, so much nicer than the .net 4 headache.

2 Likes

Perhaps Duplicati will stay one backup at a time lol. But, I heart your post for static even though it may destroy any hope of spinning off multiple backups :slight_smile:

Or, saying it differently, to be more precise that tons of heavy blocking isn’t worth doing for concurrency. Static is meant to be project wide so it may not do concurrency well. Instanced variables tend to work better and offer less traps.

Just something to keep in mind if not thinking about it. As if its not thought about then it could be a future problem.

It’s already static in “Program.cs” I just factored a bunch into a separate static class to try and separate it from the entrypoint.

My end goal is for it to be properly done with instances and dependency injection. But one step at a time :sweat_smile:

It would be interesting to know whether Kestrel avoids the current roughly 4K header limit from:

Web UI is not shown after the installation (400 error) #4833 where big other-party cookie failed.

Web interface not accessible via IPv6 #4177 is another lack that this new server solves, I hope.

meaning those whose issues I cited may need to wait, but there’s an add to your pre-announce.
Interesting style on that BTW. :wink:

The default header maximum for kestrel is 32k so should definitely help.

I’m secretly just trying to encourage new blood to help on my branch. Especially because if it is just me I don’t see getting around to building a migration path.

Talking about encouraging fresh blood …

I am new to duplicati - actually I just started using it actively a few weeks ago.
But I have some background in c#/.net development and might help and contribute a bit…

I have already cloned the codebase and setup a vs code & devcontainer environment to build it. I had started to set it up like this, because I am more on .net core /.net 6 and did not want to disturb my environment with the .net framework/mono stuff. I was quite pleased to finally find the .net 5 / .net 6 branches (or forks?) and would love to help modernizing the codebase. I think this could help to encourage even more “new blood” for the project :wink:

1 Like

Compiling / running the .net framework version is such a PITA. Reducing the barrier to entry should help, though I think open source in general is becoming broad enough that it is hard to keep sufficient volunteer interest. I’m the “maintainer” of a Scala library and I don’t do scala these days so it is pretty much also in a “pull requests welcome” state.

I don’t routinely get time to work on Duplicati, so depending on your level of experience you could try to take a stab at making the existing rest api classes accept requests through kestrel. I’ve been taking an approach of trying to keep the existing code working while making the new stuff work so I can always compare and see if I broke something that I changed vs broke something that I added. But YOLO is also a valid strategy to get it working.

Other work includes trying to get dependency injection in a good state. Ideally we would be able to use the dependency injection framework with or without kestrel so the command line executable doesn’t have to start up the webserver if it doesn’t want to. And we could share 99% of the code that way.

The tray icon also needs a little love so it doesn’t have a tooltip of “Test” and if you have a Mac, getting the DMG to play nice also would be good.

Longer term I want to abstract out the management interface so that the REST API is simply implementing it. I want it to be able to coexist with other implementations (my greedy experiment is to make an IPFS based decentralized management GUI so I can securely manage backups on many systems remotely behind firewalls).

I’m also always open to chatting about software design.

Cool. Sounds interesting.

I have quite some ideas and would love to exchange and discuss them. What would the best way of doing this?
Is there some .net 6 migration thread or github issue/feature for these kind of generic discussions? Or shall we stay on this thread for now?

For now, let me just dump my ideas here:
From what I understand from the current code base, the backend for the angular web page is basically a homegrown httpserver which offers a bunch of endpoints like /acknowledgements, /backup, /systeminfo etc.
To move towards a kestrel based backend, I would actually propose a step-wise approach using something like the strangler fig pattern. This allows to gradually replace one endpoint after another. The approach is pretty well described e.g. here or here. It’s basically just about setting up a reverse proxy (like nginx or yarp) in front of the existing webserver. In the beginning the reverse proxy just forwards all requests to the existing implementation. Then we setup a new kestrel-based webservice in parallel and start implementing some endpoints (e.g. /acknowledgements or /systeminfo to start small). Once we have some endpoints (re-)implemented in the new service, we can re-configure the reverse proxy to route requests for these endpoints to the new implementation while all other requests will still be handled by the old webserver. Then we can iterate until all endpoints are migrated and finally remove the reverse-proxy and directly hit the new webserver.

What do you think about this approach?

This thread is probably fine. There have been a couple threads on the forum and on GitHub but they haven’t really seen traffic in probably 6 months or more

There is also the GitHub milestone

I was pursuing a kind of hybrid with that pattern. I was working to wholesale get Kestrel to invoke the existing rest API classes which will require translating Kestrel request objects to the legacy object types. Then using the pattern you described to update one API at a time to use dependency injection and asp.net types.

If that translation isn’t too painful it saves having to proxy web requests

Great - so let’s use this thread for the time being.

Your approach of invoking the existing rest API classes from a new kestrel based setup is also interesting; but to be honest: I was curious how much effort it really is to setup the proxy and just gave it a try on top of your net5-kestrel branch. It was actually quite straight forward and works smoothly.

The proxy is basically an asp.net 6 core webapi project which uses yarp as a reverse proxy while also offering it’s own controller and middleware based endpoint(s). So it’s kind of a hybrid between a kestrel webserver and a reverse proxy in one web app. Currently there is just one controller defined for api/v1/acknowledgements to demonstrate the concept.

The “old” webserver is moved to port 8300, and the reverse proxy is now listening on 8200, forwarding all requests that it can’t fulfill itself to the old webserver.

Currently, only the api/v1/acknowledgements endpoint is directly served by the new implementation, providing some dummy test message, while all other requests are served by the existing implementation of the “old” webserver.

It seems to be a nice way to gradually move one rest endpoint after another to the new setup until we can completely remove the old webserver.

It is (for now) just a proof of concept. It’s working, but it’s not yet fully tested for any unexpected side-effects and the REST endpoint served by the new webserver is is still fully unauthenticated and missing all other security features of the existing implementation like the xsrf protection etc. But it should be quite easy to add these things using available asp.net 6 features …

I took the liberty to create a pull-request to your branch/fork – but only to show you what I did and probably as a location for a discussion on code level. No need to merge it if you prefer any other approach - it’s just an idea/proposal :wink:

What do you think?

I think I am most interested in the MapToApiVersion annotation that I didn’t know about.

I managed to find some time this weekend to look at this again (before seeing your reply).

I got it to where I think the only thing to try and get all the existing routes working is implementing shims for the old webserver request/response/session objects. (Currently everything throws not implemented)

I’m curious to see how your work goes directly porting endpoints. That is probably the better state to be in.

I’m also still dabbling with remembering the generic host dependency injection since I want to get all the executables using DI. Though initially I don’t expect much of anything to actually leverage it.