Duplicati is a free and open source backup client that securely stores encrypted, incremental, compressed backups on cloud storage services and remote file servers. I’ve worked on the project for 15 years, along with over 100 contributors, but for various life reasons have had less time to dedicate to the project in the last few years. PRs have been taking longer to get merged, and bigger changes (like upgrading to .NET 8) were dragging.
In March, I announced the formation of Duplicati, Inc, an open core company building on top of the open source project. This development means I can dedicate more time to the project and hire developers to work on improvements and add to the project’s velocity. One of the first major improvements I wanted to make is upgrading the project to .NET 8.
Despite multiple attempts to upgrade, Duplicati was still on .NET4
Background
Duplicati uses Microsoft’s .NET framework, which as many of you will know, started out as Windows-only. With the help of Mono, we’ve been able to have versions that run on other operating systems. Microsoft later expanded .NET to a cross-platform framework, with .NET Standard and later .NET Core. NET 5 was the last Windows-only version to be supported, and Mono’s equivalent is no longer maintained either.
So, we needed to upgrade Duplicati to a current, maintained version of .NET, otherwise we risked not being able to run it on all the operating systems we want to support. Duplicati was also becoming harder to develop, because some cross-platform development tools don’t work with older versions of .NET anymore.
Efforts to upgrade started back in 2018, with a new attempt in 2023. The code has only just been merged:
Honestly I gave up on this. No maintainers seems to be interested in this. Which is perplexing to me. .net framework is supported only because Microsoft have it tied up so badly to whole OS that they can’t get rid of it. But if they do this project is dead in the water.
Working with this repository code is horrible experience on Linux - which is not the case with newer version of .net - because Azure uses it and Linux machines are way better for cloud. So moving to new version of .net seems like the only choice to keep this project forward. – npodbielski
Why was the migration not straightforward?
Despite efforts from the community and myself, why were we stuck?
As I explained in my forum post, the upgrade getting stalled is partly because of lack of action on my part, and partly because the scope is much larger than originally expected. The ripple effects of the upgrade are much more widespread than the update itself, and they needed to be addressed at the same time in order for Duplicati to continue working with .NET8:
Project files update
The first update is fairly straightforward.
The project files used to be a proprietary Microsoft format, but in .NET8 they are updated to what Microsoft calls SDK style. The format is an improvement as it just contains the things you want, without all the default elements that used to be included in the old format. The new tools only understand this new format. This was fairly pain free to update and only took about two hours, and the new file format is also backwards compatible with older tools.
Builds for multiple operating systems
A more difficult challenge:
The original approach/philosophy for .NET and (therefore Duplicati) was like Java: “Write once, run everywhere”. The .NET framework now favors building published applications that are tailored to each operating system and processor pair (similar to how Rust and Go works).
This change has implications for the build process, as the historic setup was to create one “golden zip file” for the release, and then repackage this zip file to formats suitable for each operating system. So, you downloaded the duplicati.zip file, ran the executable inside it, and it worked regardless of what operating system you’re on.
With newer .NET versions, we no longer have this framework-independent version. We had to build a separate version for each operating system that we wanted to support. This means that for the upgrade to work, a lot of things—especially around the way updates are handled and how the installers are made—had to change.
Installers
All the install packages needed to be customized to work on that particular system (like InstallShield for Windows, DMG files for Mac, etc.). Each of those needed to be separate, with the logic being dependent on the operating system. We had to do it manually for each of them and that’s a lot of work.
I took a day to build the Mac version, because every attempt took 10-15 minutes, and then I would just get some error message and have to start over … another 10-15 minutes later, it’d happen again, and so on. This was basically my experience working on all the installers. It’s really time consuming.
Making it more difficult is that all the documentation is meant for something else. Mac, for instance, has lots of documentation on how to build an installer from the Apple Xcode environment, so if you’re building a Mac application it’s straightforward. In this case, we have an existing .NET project that now just needs to be packaged, but they don’t have explicit tools or guides for this use case, so everything is a bit of trial and error.
Updater
The updater checks if there’s a new version of Duplicati to install. Previously it would just check if there was a new zip file, but now it has to check if there’s a new version of the specific version you’re using.
UI for multiple operating systems
Previously, with one binary that can run on all platforms, I did a lot of small tricks to figure out which operating system it’s on in order to show the correct version of the UI. This did mean though that when you were on Windows for instance, you also had the UI components for Linux and Mac, but you just wouldn’t use them, which is annoying.
This had to change with the upgrade to .NET8, also because many of the things that I used to show the UI were no longer being maintained, and were only working for Mono, not for the new .NET setup.
When I started as CTO of Duplicati, Inc. one of my first steps was to publish a Battle plan for migrating to .NET8. This was partly to articulate the process for the upgrade, and also to understand the scope of the project. Fortunately, in the discussion on this post, contributor tsuckow mentioned that they’d already done work on a cross-platform UI which I was able to use.
Even though the actual .NET changes weren’t very big, everything around it was, and it took a long time to fix. We could get to a certain point where people could run it, but getting it out to actual users to install on their operating systems was the challenge.
What was the impact of not upgrading?
- It’s a deterrent to new and cross-platform developers
It’s easier for new developers to join in if you are using current technology and platforms. Before the upgrade, to contribute to Duplicati, you had to have some old tools and understand how things worked a few years ago before getting down to work. It can be off-putting to a lot of people, and if you’re not on Windows, then it’s especially hard. The upgrade was vital to get cross-platform developers on board.
- Duplicati was getting harder to develop in general
Using an outdated version means the tools to develop Duplicati weren’t great, as many of them were outdated or not supported. The Mono project is not maintained.
The MonoDevelop project, now “Visual Studio for Mac” is being retired in August. This was the original cross-platform IDE, later rebranded to Xamarin Studio, and now confusingly “Visual Studio for Mac” (Linux support was dropped somewhere down the line).
The new tool, “Visual Studio Code” uses DevTools for C# support and that no longer supports non-SDK-style projects.
- We were missing out on improvements
A lot of improvements have been made since .NET4, especially in terms of speed and developer experience. These are small things, but they will definitely be noticeable for everybody.
- PRs that go stale discourage contributions
Beyond this specific upgrade effort, it’s just a poor contributor experience for developers when their contributions aren’t merged.
I merged a lot of the existing work into my PR, but some contributions were obsolete and that’s frustrating for the people who spent time on them, and a deterrent to future contributors.
Why is an upgrade like this especially hard for open source projects?
Lots of work had been done before I took up the challenge. Between the PRs from 2018, 2023, and my recent efforts, there are over 300 commits. The problem is there were so many loose ends and someone needed to own the initiative. That’s really hard to do for an open source project, because everybody is usually contributing on their own time, on a volunteer basis. They have small pockets of time here and there, but not a large, coherent slot.
As a maintainer, I tried to get this done probably two or three times! I’d dedicate a Sunday evening, start on it, fix some things … then three weeks would pass. The next time I’d get to look at it I wouldn’t be able to remember where I got to, so I’d have to start over. This happened a few times and suddenly a year had passed, and I had to admit that I’d made zero progress on it. It was discouraging, and I felt bad because I wanted to make the upgrade happen.
Now that I’m working on Duplicati full-time, I could decide to take it on as a task and get it done. An added benefit of the upgrade is that we’re now able to use better, faster tools to develop Duplicati, making it quicker to work on other improvements to the project.
The upgrade wouldn’t have been possible without the community—a lot of the foundations of what I did were laid by others. But you need a unifying effort to actually get it over the finish line, which is hard to drum up when people are contributing on top of their full-time work.
Duplicati was not alone in this challenge
Ondsel (another open core company backed by Open Core Ventures) has documented the challenges of merging in an existing fix for a notorious bug in FreeCAD. The xz Utils backdoor is an extreme example, but one of the reasons that project was vulnerable to exploitation is that the existing maintainer wasn’t able to devote enough attention to it.
Less sinister, but still not ideal: the password manager I use has gone through probably four iterations at this point—starting with Password Safe, recreated as KeePass, recreated to become KeepassX and now forked to KeePassXC. Each time it’s either been recreated or forked because the existing version ceased to be maintained.
Even with an active community, without adequate ownership, open source projects can decay. I’m excited to be able to dedicate more time and resources to developing Duplicati together with the community, now that we are building a company around it.
What’s next?
Post-merge tasks are documented in this board. Please try out the new builds, and report any issues.
Credits
Rebecca Dodd is a contributing writer on this article.