Battle plan for migrating to .Net8

Developer
61

A small update:
I have started the big task of getting all installers working with the new build system in PR #5135.

It is time consuming work, but so far I have managed to get builds for:

  • Windows x64+x86+arm64, msi & zip packages
  • Linux (generic) x64+x86+arm64, zip
  • MacOS x64+arm64, zip+dmg+pkg packages
  • Debian x64+arm64, deb

My next step is looking at the Fedora RPM packages and finally the Docker images.
I expect the Docker images to fairly easy, but the RPM build to be challenging.

The updater changes for the current 2.0.x line needs to be updated to handle the switch.
I have done some of this work in PR #5129 but it still needs the UI to also support this change. I think I will use a the notification mechanism, so clicking the ā€œdownloadā€ button will create an error message saying ā€œplease visit ā€¦ for a manual upgradeā€.

There is some comment in the Avalonia docs that mentions you need to install:

libice6, libsm6 and libfontconfig1

I assume Ubuntu has these installed by default.

I al all for changing the prefix. I think it was the recommended style back then, but it is no longer.
I would advise not going all-in on the change now, because it will delay the real work and cause massive merge conflicts. I would prefer having .Net8 in the master branch and then doing these cleanups in individual PRs. So for now, I recommend sticking with the m_prefix (unless you are writing new code).

62

Cool I was not changing much of old code. My plan for the old code change was to remove it after we rewrite it with new style API.

I did not had the time to much last couple of days, I just wrote Websocket endpoint and tested if this works, I planned to work on the FE first to make sure that it is actually connecting to WS and consuming messages.

So for now:

  • make WS works
  • change FE to use WS for notifications instead of polling
  • make sure new serverstate endpoint works
  • make sure that Duplicati still works with some backup/restore process
  • but I am not sure if I want to do it manully everytime so I will probably write some Selenium tests for that
  • if new endpoint works remove the old one

just repeat it for every API endpoint afterwards and it is done. :slight_smile:

63

My progress is now:

  • Docker builds are working
  • Fedora RPM builds are working
  • The .Net4 builds can now detect a ā€œmanual update requiredā€ and has a custom UI for it

Synology packages will initially not be provided as they depend on the old HTTP server in DSM which is no longer present. All newer stations should be able to run Docker and use it that way, but later it would be nice with a real .spk package.

I need to go through testing on each platform and make sure everything is working as expected. The manifest files and signing parts are also missing some smaller tweaks.

My immediate plan is then to move on with getting the manual updater fixes into a new canary build.

I will hold back the 1MiB block size update and CACHEDIR.TAG to avoid introducing anything new that could break.

If it goes as planned, that canary can be promoted to beta and will be the last .Net4 build :rocket:

2 Likes
64

with a good release note, I hope. We can probably anticipate some questions, but will get others. Potentially a How-To would avoid some repetitious queries, or at least give something to point to.

RuntimeInformation.ProcessArchitecture Property might possibly be helpful to help guide choice.

Encrypted server database will have to be dealt with while thereā€™s still an SQLite that can do that.

Install-over-old versus uninstall-old-and-install-new needs advice, and possibly in both directions.

All the potential variations of Service, Server, TrayIcon, and CommandLine need to be explained.

Iā€™m not sure what the custom UI works in. Web UI is rather obvious, but thatā€™s not the only place.

I guess the old folders full of updates become obsolete in the new design. Do they get removed?

We should try to stop people from full manual deletion of base install, if they used portable-mode.

Despite all our efforts, somebody will have migration pains, but at least we should make an effort.

Are we going to start version numbers with 2.1 or something so we can tell versions apart easily?

1 Like
65

It took me sometime. Still quite messy but it works. FE connects via Websockets and receives serverstate info.

obraz
obraz804Ɨ265 48.6 KB

my propososal for endpoints are small, simple methods like that
obraz

Would that be something you would be interested to merge into net8 branch (not sure if this is master or still the kestrel-avalonia-update)?

66

Yes, there is a place for putting in a link that explains all the things that needs to be handled manually.

I think this is only part of the selection, but hopefully we can write a good guide on how to update.

Yes, that is a bit of an annoying issue. I will make a tool that can remove the encryption. The tool will only work on .Net4 or Mono, similar to the current Duplicati. Depending on the roll-out, we can consider if this tool should be downloaded or bundled with the initial releases.

Something like this: Different ways to make a Duplicati backup ā€¢ Duplicati ?

I was thinking of the WebUI. The CLI parts are also able to display the message.

Not currently. They are just left lingering so it is possible to downgrade.
We can of course explain how to remove the update folder.
If there is a need for it, we can provide a removal tool.

I do not see special considerations for portable mode? Installing the new version instead of the current one should do the same?

Yes, that is no doubt going to happen. The Synology issue has already been brought up.
I will start a new topic keeping track of known migration issues.

Yes, I was thinking that this warrants a minor version upgrade.

I have not yet merged into master. I would like to make the final canary build before I merge. I hope I can complete this very soon.

The current head for me is #5135 but it mostly just adds things around the installer. I still need to fix the build pipeline and tests.

I would very much like the websocket fixes in, as it removes the old http server, but a separate PR would make it easier to include.

67

Presumably English only, like so much of Duplicati outside of some fortunate program strings.

Still being in a forum battle with the oddities of the AUR (Arch) package and its unusual setup reminds me that there are third party packages and automatic updaters that may break as we change things or require manual steps.

Thereā€™s also RuntimeIdentifier (has a mention in Issues) and OSDescription, if theyā€™d help any.

Might work. User would have to know their system somehow, so might need to run OS queries.

Thatā€™s the challenge of high flexibility I meant. They can also use various combinations of those.
I donā€™t know if the migration document has to follow the same format, but it must cover all cases. Installation itself is simpler, now that there arenā€™t updates everywhere, but stop everything firstā€¦

I actually wasnā€™t sure the CLI used to check each run (update server hammer). It used to be that:

  --auto-update (Boolean): Toggle automatic updates
    Set this option if you prefer to have the commandline version
    automatically update
    * default value: false

I agree itā€™d be good to allow that. I donā€™t know if the DB version bump is still in the plans though.
Preventing accidental runs of old code has merit, but needs an escape path, if run is intentional.

ā€œfull manual deletionā€ meant remove the install folder, which is where portable mode database is.

68

Its cool to me.
I would just hate to see my work done around this going to waste because of something.
I can continue work on this to make it cleaner, and test it a bit more.

As for merge process, I do not care if this will be master or not, now or in 2 months. Just do not want to waste time on this; I have other projects I am working on currently.

69

Knock on wood, based on what I have read they would likely see it update to the last .net4 version and then think there are no updates. Which is probably reasonable. I would consider .net core migration to be a major version change in duplicati. So requiring a manual intervention in my opinion could be expected.

PS: Excited seeing all the work going into this, and thinking through all the nuance.

70

I see, that would be terrible :confused:

If it goes into the kestrel branch I foresee no major headaches with merging :crossed_fingers:

Yes, in any case, since the base framework is now .Net, any dependencies would need to change for packagers. My current builds are using self-contained so there are actually no additional dependencies for the binary packages, except for the GUI parts on Linux.

1 Like
71

ok cool! I will cleanup the code a bit, test it and rebase my branch on top of origin.

72

Hi, finally testing final version of the code but I am experiencing bugs that I do not think are caused by my changes, but they are pretty basic functionalities that should be working. I guess I am working on wrong branch?

I.e. when I am trying to delete backup configuration I am getting exception:

System.NotImplementedException: The method or operation is not implemented.
at Duplicati.WebserverCore.LegacyHttpRequestShim.get_Param() in /home/natan/git/FOOS/duplicati/Duplicati/WebserverCore/LegacyHttpRequestShim.cs:line 89

Where is the current working code for .net8?

Hmm I looked at history of feature/kestrel-avalonia-upgrade and it seems to be updated very recently by you @kenkendk

Is it possible that such basic functionality does not work? I did not touch LegacyHttpRequestShim beside fixing bugs myself but wellā€¦ HTTP DELETE should work right?

73

Not sure. I have only looked at the code path through the HttpServer. Looking at LegacyHttpRequestShim it looks like there is quite some work to be done before it is complete :frowning:

74

We do not need to ā€˜finishā€™ this in way to implement all members. Most of them are not really needed. If we move entirely to kestrell, we can got rid of this class and IHttpResponse/Request interfaces they implement.
But those that are actually being used should not throw not implemented exception.
Oh right I will fix those errors and we can merge my changes IMHO.
What branch I should target with my PR?

75

The master branch should be the target now :sunglasses:

1 Like
76

So I fixing some bugs connected to new API and WebSocket and encountered one big issue:

System.Text.Json does not provide a way to have equivalent of Newtosoft DefaultNamingStrategy that is 1-1 with C# names.
On the other hand minimal APIs (which I was planning to use) does not provide integration with Newtonsoft.Json.

So:

  1. We can drop Minimal API and leave serialization strategy as is with Newtosoft. But I personally would not go that way. Controllers are not go to solution anymore with .NET, so we would still be migrating to something that may be dropped in newer versions of .NET
  2. We can change JSON serialization rules for camelCase to be the same as in minimal API but this required basically touching whole FE, all the access to the state.XXX variables all the templates etc. I would personally go this way but since this require a lot of changes I am not sure if this is what you wantā€¦ BUT! For a long time I did not saw JS code having PascalCase so we would inline with industry standard I think.
  3. Do some hacking to glue all of it together (i.e. renaming all properties from camelCase to PascalCase before setting state etc. Probably the quickest route but seems like ugly hack.
77

I would go for System.Text.Json. If you do not provide a naming strategy (or use null) it should leave the names unchanged.

I think this will be the fastest and most direct path.

I know that camelCase is the default web format, and we should migrate to that standard. But rewriting major parts of the FE seems error prone. If we can have a single place where the BE naming strategy is set, we can change it when we migrate away from AngularJS.

78

Hmmm I am pretty sure I checked that but it was using camelCaseā€¦
But I checked that again today and you are right and setting PropertyNamingPolicy = null explicitly to null it actually works with PascalCase.

Ok no problem then. Thanks.