Started work on integrating the Kestrel webserver onto my .net 6 branch (Fork? Starting to feel like a fork…)
Ended up replicating a large number of static properties into a “FIXMEGlobal” class that hopefully someday can be fixed with dependency injection.
As an aside I still can’t get over how nice it is to do work in .netCore where I click build and the whole thing just works. Run publish and it is basically shippable. VSCode understands it, so much nicer than the .net 4 headache.
Perhaps Duplicati will stay one backup at a time lol. But, I heart your post for static even though it may destroy any hope of spinning off multiple backups 
Or, saying it differently, to be more precise that tons of heavy blocking isn’t worth doing for concurrency. Static is meant to be project wide so it may not do concurrency well. Instanced variables tend to work better and offer less traps.
Just something to keep in mind if not thinking about it. As if its not thought about then it could be a future problem.
It’s already static in “Program.cs” I just factored a bunch into a separate static class to try and separate it from the entrypoint.
My end goal is for it to be properly done with instances and dependency injection. But one step at a time 
It would be interesting to know whether Kestrel avoids the current roughly 4K header limit from:
Web UI is not shown after the installation (400 error) #4833 where big other-party cookie failed.
Web interface not accessible via IPv6 #4177 is another lack that this new server solves, I hope.
meaning those whose issues I cited may need to wait, but there’s an add to your pre-announce.
Interesting style on that BTW. 
The default header maximum for kestrel is 32k so should definitely help.
I’m secretly just trying to encourage new blood to help on my branch. Especially because if it is just me I don’t see getting around to building a migration path.
Talking about encouraging fresh blood …
I am new to duplicati - actually I just started using it actively a few weeks ago.
But I have some background in c#/.net development and might help and contribute a bit…
I have already cloned the codebase and setup a vs code & devcontainer environment to build it. I had started to set it up like this, because I am more on .net core /.net 6 and did not want to disturb my environment with the .net framework/mono stuff. I was quite pleased to finally find the .net 5 / .net 6 branches (or forks?) and would love to help modernizing the codebase. I think this could help to encourage even more “new blood” for the project
Compiling / running the .net framework version is such a PITA. Reducing the barrier to entry should help, though I think open source in general is becoming broad enough that it is hard to keep sufficient volunteer interest. I’m the “maintainer” of a Scala library and I don’t do scala these days so it is pretty much also in a “pull requests welcome” state.
I don’t routinely get time to work on Duplicati, so depending on your level of experience you could try to take a stab at making the existing rest api classes accept requests through kestrel. I’ve been taking an approach of trying to keep the existing code working while making the new stuff work so I can always compare and see if I broke something that I changed vs broke something that I added. But YOLO is also a valid strategy to get it working.
Other work includes trying to get dependency injection in a good state. Ideally we would be able to use the dependency injection framework with or without kestrel so the command line executable doesn’t have to start up the webserver if it doesn’t want to. And we could share 99% of the code that way.
The tray icon also needs a little love so it doesn’t have a tooltip of “Test” and if you have a Mac, getting the DMG to play nice also would be good.
Longer term I want to abstract out the management interface so that the REST API is simply implementing it. I want it to be able to coexist with other implementations (my greedy experiment is to make an IPFS based decentralized management GUI so I can securely manage backups on many systems remotely behind firewalls).
I’m also always open to chatting about software design.
Cool. Sounds interesting.
I have quite some ideas and would love to exchange and discuss them. What would the best way of doing this?
Is there some .net 6 migration thread or github issue/feature for these kind of generic discussions? Or shall we stay on this thread for now?
For now, let me just dump my ideas here:
From what I understand from the current code base, the backend for the angular web page is basically a homegrown httpserver which offers a bunch of endpoints like /acknowledgements, /backup, /systeminfo etc.
To move towards a kestrel based backend, I would actually propose a step-wise approach using something like the strangler fig pattern. This allows to gradually replace one endpoint after another. The approach is pretty well described e.g. here or here. It’s basically just about setting up a reverse proxy (like nginx or yarp) in front of the existing webserver. In the beginning the reverse proxy just forwards all requests to the existing implementation. Then we setup a new kestrel-based webservice in parallel and start implementing some endpoints (e.g. /acknowledgements or /systeminfo to start small). Once we have some endpoints (re-)implemented in the new service, we can re-configure the reverse proxy to route requests for these endpoints to the new implementation while all other requests will still be handled by the old webserver. Then we can iterate until all endpoints are migrated and finally remove the reverse-proxy and directly hit the new webserver.
What do you think about this approach?
This thread is probably fine. There have been a couple threads on the forum and on GitHub but they haven’t really seen traffic in probably 6 months or more
There is also the GitHub milestone
I was pursuing a kind of hybrid with that pattern. I was working to wholesale get Kestrel to invoke the existing rest API classes which will require translating Kestrel request objects to the legacy object types. Then using the pattern you described to update one API at a time to use dependency injection and asp.net types.
If that translation isn’t too painful it saves having to proxy web requests
Great - so let’s use this thread for the time being.
Your approach of invoking the existing rest API classes from a new kestrel based setup is also interesting; but to be honest: I was curious how much effort it really is to setup the proxy and just gave it a try on top of your net5-kestrel branch. It was actually quite straight forward and works smoothly.
The proxy is basically an asp.net 6 core webapi project which uses yarp as a reverse proxy while also offering it’s own controller and middleware based endpoint(s). So it’s kind of a hybrid between a kestrel webserver and a reverse proxy in one web app. Currently there is just one controller defined for api/v1/acknowledgements to demonstrate the concept.
The “old” webserver is moved to port 8300, and the reverse proxy is now listening on 8200, forwarding all requests that it can’t fulfill itself to the old webserver.
Currently, only the api/v1/acknowledgements endpoint is directly served by the new implementation, providing some dummy test message, while all other requests are served by the existing implementation of the “old” webserver.
It seems to be a nice way to gradually move one rest endpoint after another to the new setup until we can completely remove the old webserver.
It is (for now) just a proof of concept. It’s working, but it’s not yet fully tested for any unexpected side-effects and the REST endpoint served by the new webserver is is still fully unauthenticated and missing all other security features of the existing implementation like the xsrf protection etc. But it should be quite easy to add these things using available asp.net 6 features …
I took the liberty to create a pull-request to your branch/fork – but only to show you what I did and probably as a location for a discussion on code level. No need to merge it if you prefer any other approach - it’s just an idea/proposal
What do you think?
I think I am most interested in the MapToApiVersion annotation that I didn’t know about.
I managed to find some time this weekend to look at this again (before seeing your reply).
I got it to where I think the only thing to try and get all the existing routes working is implementing shims for the old webserver request/response/session objects. (Currently everything throws not implemented)
I’m curious to see how your work goes directly porting endpoints. That is probably the better state to be in.
I’m also still dabbling with remembering the generic host dependency injection since I want to get all the executables using DI. Though initially I don’t expect much of anything to actually leverage it.
I’ve managed to shim quite a bit of the webserver in Kestrel.
I am able to load the UI and create a new backup job
TODO:
Hi @tsuckow,
I’ve been looking at duplicati and your port. I’ve also been thinking about you “fork” comment.
How hard would it be to create some smaller pull requests that move as many libraries as possible to netstandard20? Lowering the scope of change is always helpful. Is the commandline in a position to be .net6?
I think to be able to have native binary for linux, mac and Windows if only for command line would be a big project move and it seems like most of that work you’ve done.
What are your thoughts on that?
It’s been a while but if I remember correctly the native libraries are the problem. Because Duplicati doesn’t use inversion of control, the file system libraries are at the very bottom of the stack and those are problematic. I’ll try to refresh my memory the next time I’ve got some time. The honey do list has been long lately.
I tracked down when I gave up on coexisting. Net5 split/more bugfixes by tsuckow · Pull Request #4483 · duplicati/duplicati · GitHub
@warwickmm Had made a net5-split branch that the goal was to maintain building for .net Framework while also targeting .net 5.
I personally was never able to get the framework builds to function, but I also started down the .net Core path because I had trouble with Framework even from the master branch. Some libraries in my branch have ended up needing full .net so don’t depend on .netstandard which is a problem for dual targeting. In theory much tinkering could be done to make that work but I would argue it would need someone going back to the old net5-split branch who can get the framework stuff functioning sufficiently. Then we could work through merging.
It is also sufficiently frustrating that the duplicati repository does not have a gitattributes file normalizing line endings, the whole repo is a mix of crlf and lf. Some changes in net5-split resulted in changing the line ending format of numerous files.
AFAIK the command line works flawlessly in my branch. Also be aware that my branch does not support encryption/password on sqllite for reasons of nebulous licensing.
Thanks.
I do have framework working. I have a strong preference for small pull requests that are merged quickly rather than what you were forced to do.
I’d love to be able to merge one project at a time moving to netstandard20, before even targeting .net6. That would mean manually adjusting you branch and pulling bits out. The difficult part of that is the attribution to your work would be removed. If I’m able to setup a way for that to happen, would you be interested in either me merging projects, or making pull requests that do that?
I don’t particularly care about attribution. So go nuts, i’d love to incrementally get there as well.
If you can get a foothold I’ll also try to surgically port changes. One low hanging fruit is Thread.abort → thread. Interrupt.
Thanks for the history. It’s taken many turns, I wasn’t there, and branch plan is now being discussed.
Historically, it seems like we had branches for features and large changes, ideally merged to master.
There’s a proposal to flip that, but I’m not really convinced. As a big change, your input is valuable…
Refreshing a repository after changing line endings sounds severe. Is there a better path to “better”?
Sorry I’m not a GitHub guru, but I do expect we’ll have to support devs on different line-end systems.
Can you expand on that? I don’t think this is SEE. The problem is that they removed old code that we use, meaning we’re stuck on an old System.Data.SQLite until we can find how to get encryption back.
I’m not sure that it’s a real problem. I think that there is some kind of feature called editorconfig that allow to set an editor for any line ending.
example:
cat .editorconfig
root=true
[*.cs]
indent_style = space
indent_size = 4
end_of_line=lf
(yes I know I should have used it myself but fact is I only found this very recently…and I have not even tried to find if there is some standard defined for the project in some Wiki corner. Shame on me)
IMO you should really write /how/how NOT/ and /back/hack/
the proper way to protect the database from prying eyes is to install Duplicati as service. Upgrading Sqlite to recent version could also be useful for other reasons (performance maybe).
Thereby raising other issues which arguably we should fix (Issues are filed). It’s less easy.
There’s also the question of multiple users, e.g. on shared system running as themselves.
As I understand it, the weak scrambling is also meant to help protect from malware scans.
They might not be polite enough to stay out of SYSTEM. Anyway, agree other paths exist.
Solidify SQLite dependencies #4024 points out that Linux uses the distro-installed SQLite,
and this gums up both the currently scrambling and what we can count on being available.
Kind of off-topic for Kestrel, but it’d be nice to start thinking about how to protect secrets…
Current scheme isn’t great, and there are Issues and talk of better ways to do this anyway.
Point taken.