Battle plan for migrating to .Net8

[kenkendk]

Background

The change from .Net4 to .Net Core and later the combined .Net5 has happend, but this has not been adopted by Duplicati. The work was first started by @mnaiman in 2018, but was never merged. This was partly because of lack of action on my part and partly because the scope is much larger than originally expected.

One part is the actual merge of the code, which has been attempted through PRs (initial, and then later).

This change has some issues because Duplicati currently relies on isolated AppDomains which is no longer supported in .Net and has no direct successor. The updater mechanism and the service runner depends on this feature.

Another, more complicated issue, is that Duplicati was written with the concept of using .Net byte-code, where the same binaries can run on multiple platforms. The “write once, run everywhere” approach that is common with Java. The .Net framework now favors a building published applications, that are tailored to each operating system and processor pair, similar to how Rust and Go works.

This change has implications for the build process as the current setup is to create one “golden zip file” for the release, and then repackage this zip file to formats suitable for each operating system.

Current state

To better understand the challenges, this section introduces the current design, and the the issues that are with this design and the changes described above.

The updater system

With the current code, the system will actually use the downloaded version as a “stub”, only used to launch the correct version. In case there is no update, it will launch itself. If there is an update, this new version is checked to ensure it is pristine, and then executed.

The design idea behind this was to make it easy to update, requiring no administrative access, in the hope that this would cause users to update more often. It was also a system that could be implemented cross-platform, and as the zip update is also platform-independent this works.

There are a few downsides to this approach:

• Speed: The check for tampering takes time, so starting the CLI takes a bit of time
• Loss of control: Any changes to the files, sometimes caused by anti-virus, will cause an unexpected downgrade.
• Monitoring: The installed version will display something different than the actual running version
• Usage: To allow the updater to change executables, it will spawn multiple process, causing up to 4 Duplicati processes running
• Limited update: Since a new version is not installed, new requirements cannot be introduced. This is relevant when there are native library changes, that cannot be shipped with the update
• Oversize: As the combined zip has everything, it is larger than needed for the given operating system

The Tray Icon

The UI entry of the project is based on a single executable, with some delay-loading to support starting the correct UI for the platform. This is slightly hard to support, because build support is often only present on the platform where it is being build (i.e., MacOS support is not present on Windows and vice-versa).

Since the new .Net approach is to build specific versions for each platform, this decision is better made at compile time, which ensures only the relevant libraries are included. In turn this means that each version can only be built on some platforms, and no platform can build all versions.

On top of this, there are some issues with the way both Windows and MacOS encode the type of executable (GUI or CLI application) which makes it hard to implement this switch. For MacOS, there is a workaround with a native loader application for starting the app, and another workaround for showing the tray icon, based on Python and RUMPS.

Considerations for a solution

I have a desire to fix multiple problems while everything is messy anyway, but there is also a large risk that the update gets stuck in details and drags on forever, and eventually is difficult to merge back in.

Ideally, the updates should be possible to split into multiple PRs.

To assist in developer growth, I would really like any developer to be able to check out the source on their own system, and simply execute dotnet run to get a debuggable version running. This should work on any operating system without the need for additional downloads or fixes.

Also, since the project is mostly written in C#, I would like as much as possible to be debuggable from C# (excluding any talk about the web-UI in this post).

I think an easy update solution is an absolute must for any serious project, so this needs to be included.

Suggested solutions

The key change is the need to have a single codebase, but to distribute separate native builds. This translates directly into a need for having multiple builds, instead of a single “golden zip”. This means that the build process will need to create a matrix of different supported processor and operating system builds. This directly translates into also creating multiple installers for the different operating systems.

New packages to create

To figure out how many different packages that are required, I would start by figuring out what builds are needed:

• Windows: x64, Arm64?
• MacOS: x64, Arm64
• Linux: x32, x64, Arm64, Arm7, Arm6?

In terms of GUI support, we need at least:

• Windows, Cocoa, GTK

I think we need at least a zip/tar build of each version, and then installers for the relevant platforms:

• .msi, .pkg, .dmg, .deb, .rpm

I think we can discontinue the Synology version, in favor of a Docker build, as all supported Synology systems are now x86 based and have Docker.

So the list would be something like:

• Zip: Win-x64-Win, Win-Arm64-Win, Mac-x64-Cocoa, Mac-Arm64-Cocoa
• Tar: Linux-x32-GTK, Linux-x64-GTK, Linux-Arm64-GTK, Linux-Arm7-GTK
• .msi: Win-x64-Win, Win-Arm64-Win
• .pkg: Mac-x64-Cocoa, Mac-Arm64-Cocoa
• .dmg: Mac-x64-Cocoa, Mac-Arm64-Cocoa
• .deb: Linux-x32-GTK, Linux-x64-GTK, Linux-Arm64-GTK, Linux-Arm7-GTK
• .rpm: Linux-x32-GTK, Linux-x64-GTK, Linux-Arm64-GTK, Linux-Arm7-GTK
• Docker (no UI): Linux-x32, Linux-x64, Linux-Arm64, Linux-Arm7

So, already, this is 26 different packages. It would be nice to split the .deb and .rpm packages into non-UI versions and UI versions, which would increase the count to 34 versions, but perhaps we can reuse these additional packages to base the Docker images on, so we end on 30 packages.

This seems super excessive, but looking at other projects, it looks like this is the common way to do it. Examples: VLC, Firefox.

The updater

Once it is established that multiple installers are needed, it becomes clear that using the current update process will be difficult to support. I think further on, it might be possible to make a smooth update experience, but in the interrest of moving forward, I suggest a very simple solution for the update.

Each package will be build with a small identifier (i.e., a text file) that describes what package is the source. When checking for updates, this identifier (e.g., MacOS-Arm64-Cocoa.pkg) can be used to figure out which package format was originally used, and then download the relevant file updated package. The installation will then require the user to activate the installer, but this will play nice with the OS installation process and show up correctly in the list of installed applications.

While this is less smooth than the current process, it addresses all issues mentioned above without adding much in terms of development.

The tray-icon and GUI framework

Since the packages and/or builds are framework targeted anyway, it makes sense to make native/individual “loaders” for the GUI. To avoid needing to change logic, like the menu texts, in each loader, the loaders should be empty shells that simply work on data sent from the main applications, and send events back.

This design is already implemented in the RUMPS application, and can be implemented by sending the desired menu, including base-64 encoded images in JSON to the shell application. The shell application can then send events back, such as “click” and obtain an updated menu.

For Windows, the executable needs to be marked as a GUI application, as it will otherwise show a GUI window. The GUI shell application can be written in C# and simply load the framework-free tray-icon assembly and activate it. This prevents additional processes from starting, as the a .Net application is already running.

The same approach can be used by the GTK based application, but there is no requirement to mark the executable, as this is distinction does not exist in Linux.

For Cocoa there should be a separate application in Objective-C which should be stored as a compiled binary, to allow builds on non-MacOS platforms. This process is already partially implemented with the current launcher application, so the work here is limited to porting the RUMPS code to Cocoa.

For all three platforms, the compiled binary can be checked in and is not expected to change, as the menu and logic is in C# and the shell applications just shows the contents.

Migration of existing installs

To support migration of the existing users, we need to release at least a single update where the support for choosing installers is present. This could be an additional tag or similar, that is picked up, so the user is directed to the download page, instead of attempting to download the (missing) zip file. We need to handle the case where the user sees the upgrade with an old client and upgrades to the latest pre-.net8 release as well. The description should clearly show that the previous Duplicati should be manually uninstalled, and we can prevent the previous version from accessing the database by injecting a database upgrade with the .net8 version.

Suggested work plan

Based on the above considerations, it should be possible to get from the current state, to .Net8 and back to a working state. I have identified these sub-tasks and an order I think they should be done in:

• Update to SDK style projects
• Update to .Net8
• Build tray-icon shell applications
• Update build script to generate multi-arch releases
• Update installers
• Update build script to generate builds
• Update “check updates” code for new and previous logic

Comments, questions, objections, alternatives, etc are much appreciated.

[TopperDEL]

Wow, this is much work - but it needs to be tackled, if Duplicati should become future-proof. Thanks for the insights!
I can only provide help with the Storj DCS backend. And I only have minor knowledge about Duplicati itself. I’m not sure if it is easier to rebuild it from scratch (and use e.g. Uno Platform for the GUI) or to try and bring everything to .Net8. But I appreciate your thoughts, your work and your commitment to this awesome project!

[ts678]

I like the aim of a smaller first approach over a huge wait, but I’m not expert enough to know all details. There’s currently a GitHub .NET 5 milestone with a lot of things that were once in mind for that project.

The .NET Framework and mono platforms are pretty stagnant, as intended. Mono appears a bit worse. Performance should also be better on the new platforms, except maybe where SQLite is the slow spot.

Single-file deployment talks about Native libraries, and we use a few. This may factor into questions of platform support, what is extracted at startup, where to, and whether antivirus will interfere with extract.

I’ve also found a number of ways where Duplicati itself (sometimes with user help, sometimes not) will break its own update folders (some of this is by writes to current directory), so that problem will vanish.

Doing self-contained instead of framework-dependent puts more keep-up-with-security-fixes on us, but possibly some people would prefer frequent releases anyway, while some may like current slower plan.

Easing the collect-all-the-parts work that Linux users see a little and Mac users even more, will be nice. There will probably still be some OS dependencies for the tools (e.g. apt on Linux) to go off to obtain…

mono has already had to drop their rpm builds due to toolchain issues. FreeBSD almost has a .NET port.
The world is moving, and seeing Duplicati start to get with the times is a very good thing to see. Thanks!

[ts678]

@kenkendk

Oversize: As the combined zip has everything, it is larger than needed for the given operating system

Doesn’t ability to share the framework and all its/our .dll files among all our many .exe files offset that?

@kenkendk

To allow the updater to change executables, it will spawn multiple process

The current updater plan requires a whole lot of explanation, and gets even more confusing if a TrayIcon manages to be on a different update than the Windows service, which has its own autoupdate fail issues.

@kenkendk

Speed: The check for tampering takes time, so starting the CLI takes a bit of time

Not just the CLI. My Windows service won’t start with a few updates, as it hits default 30 second timeout. TrayIcon activation is also failing sometimes, and the prior attempt to fix it worsened the update situation.

@kenkendk

we can prevent the previous version from accessing the database by injecting a database upgrade with the .net8 version.

Is that necessary? If so, is it reasonably possible if system has a service and multiple non-service uses?

[kenkendk]

@TopperDEL

I’m not sure if it is easier to rebuild it from scratch

I doubt it

I rebuilt the core for fun in an hour or so, but all the small things will take significant effort to recreate.

@ts678

Doesn’t ability to share the framework and all its/our .dll files among all our many .exe files offset that?

I think it is possible to do some manual combination of the final builds, so we can still re-use the .dll files across multiple executables.

@ts678

Is that necessary? If so, is it reasonably possible if system has a service and multiple non-service uses?

I was only worried that some users would neglect the uninstall and have two versions running.

[TopperDEL]

My idea with rebuilding was: What if we start with a clean core and add the backends one after another. Or maybe use a kind of a plugin system, where every backend is a plugin and has a kind of a “Feature Set”/“supported OS”-flag. The idea is to get the first New version out quickly and expand it afterwards.
Updating would basically mean to uninstall the previous one and Kind of “start from scratch”.

But this is just my idea, without knowing the full picture. I just Remember that adding the Storj DCS backend Was quite a journey due to the HTML-stuff that evolved during time.

Just my two Cents.

[kenkendk]

@TopperDEL

What if we start with a clean core and add the backends one after another.

I think the backends are actually quite straightforward to upgrade. There is some work in HTTP libraries, but they are mostly backwards compatible. My idea was to basically remove anything that does not work in .Net8 and with that accept some loss of functionality in exchange for getting a working version faster. Then we can add features later. I think the two approaches share a common theme: minimum viable product, but the path is slightly different.

@TopperDEL

due to the HTML-stuff that evolved during time.

Yeah, AngularJS is another project that needs addressing. Fortunately, @Jojo-1000 has started this work, so we can hopefully have something more modern in the HTML area.

[ts678]

@kenkendk

This translates directly into a need for having multiple builds, instead of a single “golden zip”.

Developers need a way to build subsets that suit their personal needs, without building the whole set.

There is a secondary occasional use where they build test builds for others who run various systems. Generally I see them provided via a GitHub area having multiple OS types and a .zip file inside one. Sorry I’m not familiar with how cross-builds and cross-OS-package is done, but .zip needs that less.

@kenkendk

I think it is possible to do some manual combination of the final builds, so we can still re-use the .dll files across multiple executables.

I suspect it’s something we can play with, measure, and change, so it’s not essential to decide it now.

@kenkendk

I was only worried that some users would neglect the uninstall and have two versions running.

Good point. Even without a version change, having two Duplicati at work will lead to potential messes.

[tsuckow]

Other than changing the updater, significant work was done on all of the listed items.

The biggest gotchas:

• BSD support is probably impractical

• Encryption of the database is legally encumbered

• Trying to update to SDK projects while remaining on .net framework is not for the faint at heart. The dependency tree is nuts. The project files are certainly the biggest source of merge conflicts between the branch and mainline.

Edit: This branch is probably the most complete (note it’s .net6 not 5 as the branch name implies). GitHub - tsuckow/duplicati at feature/net5-kestrel

[tsuckow]

I went ahead and merged GitHub - duplicati/duplicati at feature/upgrade-to-sdk-style-projects into my branch. Biggest difference is the power api for windows which another contributor had a different implementation.

It builds (except on macos, should be easy fix), but I didn’t test it. Fix release build dependancy problem · tsuckow/duplicati@65e907d · GitHub

[kenkendk]

@tsuckow

Edit: This branch is probably the most complete (note it’s .net6 not 5 as the branch name implies). GitHub - tsuckow/duplicati at feature/net5-kestrel

Ok, I will look at that and see what is ready there. I have used upgrade-assistant to upgrade all projects to .Net8 and I am currently picking up on work from @mnaiman 's PR to fill in the actual code changes.

Once I have it building, I will look at your changes and incorporate that into a PR.

@tsuckow

Other than changing the updater, significant work was done on all of the listed items.

I am not aware of work on the TrayIcon applications either, have I missed something?

[Wim_Jansen]

@kenkendk

I think we can discontinue the Synology version, in favor of a Docker build, as all supported Synology systems are now x86 based and have Docker.

Not all Synology systems can run docker. Eg. DS216j does not run docker AFAIK.

update - did find a thread online where it does seem possible to run docker on DS216j. Will revert back here if it was succesfull for me.

[kenkendk]

@Wim_Jansen

Not all Synology systems can run docker. Eg. DS216j does not run docker AFAIK.

Yes, I used to have an Arm based system which could not run Docker, but Synology is ending support for them in October, so after that I cannot see any Synology products that does not support Docker.

[Wim_Jansen]

DS216j is ARMHF based and still supported AFAIK.
Docker is not officially supported on my synology ds216j. I managed to get it working, but running duplicati on this armhf-based platform did not work, as there are no builds for this HW.

So - either armhf needs to be added, so I can continue to use duplicati. Or I stick with older releases once this migration starts.

[kenkendk]

@Wim_Jansen

So - either armhf needs to be added, so I can continue to use duplicati. Or I stick with older releases once this migration starts.

Hmm, it looks like even Arm7 is not going to be supported: net7/net8 run on arm9 or arm11 · Issue #83975 · dotnet/runtime · GitHub

[Wim_Jansen]

I guess I need to build a business case to move away from the DS216j in that case…

[tsuckow]

@kenkendk

I am not aware of work on the TrayIcon applications either, have I missed something?

My branch has an Avalonia based cross platform tray icon implementation. There was a quirk on macos where the magic bar or whatever its called still showed the app but it may just need a setting or an update may have fixed it. I had to rent a Mac on the other side of the world to test it so I haven’t looked too much into it.

[Jojo-1000]

Since you started piecing together parts of updated code, there is my net5-split-cancellationtoken branch, which should have all backends async and cancellable (not fully tested). It is at least compiling and running under .NET 6.

[kenkendk]

Sounds awesome! I will keep it out of this initial .net8 port, to keep it as clean as possible.

[kenkendk]

@tsuckow

My branch has an Avalonia based cross platform tray icon implementation. There was a quirk on macos where the magic bar or whatever its called still showed the app but it may just need a setting or an update may have fixed it. I had to rent a Mac on the other side of the world to test it so I haven’t looked too much into it.

Ok, that sounds like a great way to fix it. I will see if I can get it to work.