I want to hear input from the community on a new release-cycle-model. I have some ideas that I think I great, but you can probably make them better
First off I want to redefine the different release types clearly so it’s easier to talk about the actual cycle.
We currently offer 4 types: Stable (when ready), Beta, Experimental, and Canary.
Unfortunately it’s more been a row of incredibly stable Canary builds and 3 nearly “Stable quality” beta and experimental releases.
I want to change that so that our definitions are:
Canary:
Compiled master branch daily / weekly / whatever period is sensible for us
Must be released consistently even if barely any changes were made.
Should be automated so nobody has to spend time actively maintaining these releases.
No patch notes / possibly auto generated patch notes based on commit messages
Experimental:
Contains any new features that have not had issues reported after X canary releases, depending on the size of the feature.
All bug fixes are immediately released to this version for validation.
No specific feature or bug fix goals for each release, it just includes whatever meets the criteria on release day.
Compiled consistently every X canary build. A human needs to sit down and collect commits for this release, so it may not be entirely consistent.
Beta:
Contains features with no issues reported after X experimental releases
Contains any bug fixes that have been in experimental for X releases
May have defined milestones if showstopper issues have been identified
Can be released whenever showstopper issues have been addressed as this more or less serves as stable until we reach 2.0 stable.
If the showstopper exists currently in the previous beta, new betas may still be released with bug fixes and stable features as long as new showstoppers are not introduced.
Stable:
it just works
Will not be released until a stable milestone is reached
Has preferably existed as a beta but is now released as stable with little or no code changes.
There are a lot of X’es in here, because it’s hard to set intervals without knowing exactly how much it will take to put together the experimental and beta releases from the master branch. I had imagined something like:
weekly canary release sunday night (it seems a lot of things are submitted/merged over the weekend)
2-3 weeks between experimental releases, depending on human factors and the complexity of new features added.
6-9 weeks between beta releases, giving time for 2-3 experimental releases which will allow time for both minor and major feature changes to make it into each release.
Stable release end of year?
For this to work, and even be humanly possible to manage, we may need to implement some rules for tagging pull requests. I was thinking we, as best we can, try to tag every merged pull request with one of a list of options:
[bug fix]
bug fixes make it straight into canary and experimental. Anything tagged [bug fix], that fixes an existing issue in Github, which isn’t a feature and doesn’t look too complicated, will make it into those releases right away.
[cosmetic]
Changes that are basically cosmetic, whether or not they have a relevant Github issue, can also be released right away to canary and experimental.
[minor feature]
Implementations that are usually tweaks of existing code or very small amounts of code, which are easy to read and understand. These could make it into experimental releases after as little as 1 week of canary testing.
[major feature]
Implementations that create completely new classes or rewrite larger parts of the code. Should be in a couple of canary builds before making it into experimental.
I volunteered to help Kenneth collect features and bugs for new releases, but I would like to have an interactive process for the human factor. I’d like to create a new topic for each experimental, beta, or stable version in #releases that lists all the features and bug fixes I intend to include. This will then be open for people to chime in with opinion and additions to the list before it’s released. Ideally the topic is created right away after a version was released so it can be discussed for the entire duration of a release cycle and allow people to focus on the upcoming features.
This ended up being a bit longer than I expected. I will edit updates into the original description here so that it will always serve as up to date information on release cycles.
It seems you have spent a lot of time to think about Duplicati releases. I am at the moment discussing this with Kenneth, too.
Your release cycle seems easy to understand and makes a lot of sense. Actually, we had something versy similar in mind, when we created the different update channels. Nonetheless, the main problem still remains unsolved: Assume you are working on a feature A and a feature B. A is stable after 10 canary releases but B is not. How do you want to release an experimental version then? You cannot remove B from the software. Even if you can, you cannot be sure that this has no bad impact on feature A or anything else.
From my point of view, there is no technical solution to this problem. But you can increase the development and testing speed, so that you get less operlapping work: You work on A and get everything done before B is started. And even if B is started while A is in development, they both should be completed quickly, so that feature C becomes a totally independent, new topic.
I think, if we can use our community power to get the new fixes and features tested quickly, we can also speed up the release cycle and be less cautious with releases than we are today. I hope that we can use the forum and Github to get new releases supported and rolled out much faster than in the past.
As I said, we are still discussing how this could work. Then we come up with a first suggestion. This might not be a final solution but hopefully a step into the right direction. And if we get a bit faster, the second step will make it even better soon
I am naive enough to assume that features merged into master would make it from canary to experimental within 2-3 canary releases, and similarly from experimental to beta in 2-4 experimental releases
I would basically base an experimental release on a 3-4 week old canary and then manually cherry-pick any bug fix pull requests merged into master. Then add any minor/major features that have been in a couple of canary releases - more or less judgement calls.
I agree it’s impossible to manage an unending number of pull requests with different “stability”. I think the solution is to agree on a “time scope” for each level. Once a change goes beyond that scope it’s stable enough for the next release. Unless it’s explicitly noted as a showstopper I think it’s fine to let bugs slip through to beta, that’s what the stable milestones are for.
Thanks @Pectojin. This seems to be a very usable release model. A few random thoughts:
Apart from how many or few Canary/Experimental/Beta versions are released, I guess it’s useful to define what’s the main goal of those releases.
All upcoming releases should contribute to releasing a stable version as soon as possible, but not before all bugs/improvements for stable milestone are fixed and known UX issues are polished up.
There still is a decent group using v1.3.4, just because of fear of the word beta and/or experimental. If this group makes the switch to Duplicati 2, support for 1.x could be officially dropped.
Main goal for releasing Canaries is testing new functions and fixes by a decent group of people running Duplicati in different environments. For this purpose, I think Canaries should have notes about what’s new and/or changed to make effectively testing of possible. If someone using the Canary channel doesn’t know exactly what’s changed, it’s not easy to test the stability of the changes. Better fewer documented Canary versions, than daily/weekly without knowing what’s actually changed.
IIRC, apllying translations is a manual task. When and how should translations be updated? In experimental builds? Or can this also be automated somehow? I prefer translation updates as frequent as possible, they usually don’t have much impact on stability.
Definitely, but I think we should be careful not to have too specific goals for experimental/beta releases. They just end up preventing releases, which in turn puts us back to just getting canary releases.
I kind of tried to simplify this by making features and bug fixes default to being “stable” enough for next level after a set number of iterations - unless something is really wrong. Inevitably this will let bugs through, but it makes things more manageable and brings fixes and new features to beta much quicker.
Then there’s the headache of picking what to release as stable
Fair point. I think it’s often the case that people wanting to test new features already know what they’re looking for because it’s addressing something in an issue they opened.
Change logs are nice, but if they require too much effort Canary builds won’t go out like clockwork and instead of people not know what they test they don’t test it.
I had not thought about translations, but I see no problem in applying translations on any new release whenever it is built.
I’m completely in favour of pushing out a new stable as soon as possible just to get those people up to speed, but there is no way around it, we need to address the biggest issues in the stable milestone before it’s fair to declare it stable.
I mentioned this in a staff post but basically I asked are we relying on user reporting of issues or something more consistent like beefed up usage-reporting and related, umm, reporting?
Also, as can be seen with the current FormatStringSize negative number bug, sometimes pre-existing bugs get exposed by “unrelated” changes. We should maybe consider a stronger/faster “oops, we missed something” response and “unroll” known bugs over trying to fix what’s already rolled in a potentially broken state.
Uh, that would be very cool! Then we can get meaningful data out of reporting to help steer the release cycle or help reacting to bugs released by mistake.
That’s an excellent point. The model should describe how to react if showstoppers are discovered after releasing to stable/beta. Perhaps some kind of “fast track” that’s pushed up to any release where the bug exists and prompts for a quick release right away? Then just increment the version number 0.0.1 to get the updater to recognize the update and prompt to install it. Of course such a fix doesn’t have to be an actual fix, if it’s easier to roll back the code that can just be the “fix”
Could stability based on successful backups be done with the cooperation of the Duplicati Monitoring site? All those results dumping into a database right now.
I imagine that the best way to do it. I haven’t seen it’s data structure but it’s probably easy to pull that information out and display stability stats.
Yes and no. If you know elastic search (the “database”) and Kibana (current reporting tool) it’s probably pretty easy - assuming the data we want to use is already being recorded.
Since in still (slowly) learning both those tools I haven’t gotten very far.
On top of that, usage logging is currently optional so we may not be getting a full picture.
And even worse, right now there seems to be a spike in usage reporting related errors for which the workaround seems to be to disabling it.
And lastly, since I expect we’ll need to collect a bit not detail than we did now, we need to make sure we balance useful data collection with user privacy.
So if you know anybody with interest, time, and skill in these techs please have them stop by the forum.
Well, know is a strong word, but I work with elastic search through Graylog, so I’m familiar with the database… just not with how to code for it.
Of course. Anything I say should always be taken with a slight consideration that maybe I didn’t look at the data and wrongly assumed it was there
We can do no better than work with the numbers we get from people who opt-in. That’s okay. Although it plays out hilariously in my head with hundreds of users asking on github and on the forums “When can we get a new release? it’s been 3 years!” and Kenneth just replying “We don’t have enough usage statistics, turn on usage reporting if you want a new release”
I think a workaround for these issues may be to handle usage reporting errors gracefully when they happen and then log that through some super simple https request into Kibana. Then we can know how often it happens and take that into consideration. Plus if we handle it gracefully, then they won’t have to disable it and there’s a chance it just magically starts working again later. If users first have to disable it, they might not enable it again.
That’s what I’m still confused about - the usage reporter (Duplicati.Libarary.UsageReporter.ReportSetUploader) is already in a try/catch block that writes the “UsageReporter failed” error people are reporting.
So I’m not sure why a bad usage logging event should be causing this issue. The only things I have thought of so far are:
it’s the VOLUME of the errors causing the issue (after X times of the same error Duplicati says “I give up” and quits, since the error logging was the last error to happen it looks like that’s what “caused” the failure)
callers to the method are expecting a particular Tuple response and some point they’re getting back something they don’t expect and so crash
Right now I’m leaning towards it being the former - but I’m not sure where to go to confirm that. @kenkendk, is there an “abort after X errors are logged” limit somewhere in the code?
Would a create-report submission from a usage reporting related crash be useful? I think there are a few users who would be willing to submit one.
Also, I just got a log from this post that shows 53 entries, 51 of which are Error: UsageReporter failed.
Since my tests with usage-reporter.duplicati.com blocked in my hosts file don’t produce any errors I’m wondering if it could be that the usage-reporter server is overloaded? (Not accessible is different from (500) Internal Server Error.)
Yes! that is actually what happened (I did not see this post before now).
I did a configuration update, and then I accidentally made the local database readonly. The database is used for displaying the graph on the landing page.
Everything seemed to run fine, but eventually all clients queued up their usage reports, as all requests gave the 500 - Internal error, causing the load to increase and increase and the machine got slower and slower.
After I fixed the database, the requests went through and it seems to run stable now. I have since put a cap on the number and age of usage reports, such that the clients do not queue up huge sets of requests.
