Canary/Beta Release Logic?

Releases
1

I have multiple Dulpicati installations on various Canary & Beta releases. I keep all my more important systems on the Beta channel, while I have used the Canary channel for some of the “lesser” systems. However, I am continually confused (frustrated?) by the lack of coordination between the two channels.

I understand that Beta is more stable and (by definition) lagging behind in features from the Canary releases. But is there a clear feature path whereby Canary features/fixes are planned for specific beta releases? I am used to large development environments where there are clear Development->Test->Release plans. Bug fixes and new features are implemented in Development, moved to Test (Duplicati’s Canary channel) for user testing, then ultimately released (Duplicati’s Beta channel). At some point there is a cutoff of bug/feature development targeted for a specific release so the code can move cleanly through the various tiers. Is there any such code coordination between Duplicati Channels?

For example, for yesterday’s Beta release, is there a specific Canary release that can be pointed as the “base” code the beta was derived from? If so, is this planned out in advance or is a decision made at some point to cut off feature development and work toward a Beta release?

I’m not trying to criticize the hard work that has gone into developing Duplicati. I know you’re working with a volunteer army and release planning takes overhead and time away from development work. But I would suggest that better feature/fix planning between the various channels might make it easier for end users to plan and coordinate which Canary & Beta versions they want to install on their systems.

Thanks for listening.

HG

2

Did I miss a release? Releases only shows v2.0.4.36-2.0.4.36_canary_2019-12-08 released yesterday.

v2.0.4.1-2.0.4.1_experimental_2018-11-08

Changes in this version:
This is a collection release that is based on the 2.0.3.14 canary build.

is like what you ask, then you must know Experimental soon led to v2.0.4.5-2.0.4.5_beta_2018-11-28.

v2.0.3.1-2.0.3.1_experimental_2018-03-16

This experimental release is mostly the same as the canary build v2.0.2.20.

was an earlier example, but the exact words used depend on the person who writes the release notice, which has been consistent recently. There’s an opening for a release manager if anyone wants to do it.

v2.0.3.3-2.0.3.3_beta_2018-04-02 was where that ended. BTW this is also all in About → Changelog, which is quite a bit of reading, but if you’re interested in what changed when, that’s the way to find out.

In my opinion, there’s not a whole lot of planning in advance. Commonly, volunteers just show up with code. There’s been an effort lately to slow things down a bit, get stable, and do a Beta, which is first a Canary, then if that holds an Experimental (which are infrequent, so are good tests that update works), then if that holds it goes Beta. Sometimes an Experimental or a Beta takes a few tries before it’s good.

and probably lots of staff to manage, coordinate, build, test, and release it all. Or at least I’ve seen that. Typically there are plans, schedules, trackers, assignments, and pressure. I’m not sure it’s a good fit…

There have been (and still are) quite a few discussions on how to best do a project that’s bigger than a person, yet smaller than a large professional development organization. Currently there’s one master branch in GitHub that builds Canary (and might build Nightly soon), and no other permanent branches. Having more and moving code around has been discussed, but it gets complex and needs a volunteer.

This has been tough, but is a bit better recently when it’s clear that the November 2018 Beta is too old. Perhaps even more importantly than new features, Canary has a lot of bug fixes that users should get, however it sometimes has new bugs. Its purpose is to let volunteers find them. So which one is better? Likely it varies from time to time. Right now, Canary is more reliable, but any update could change that, which is why when I recommend it to people suffering from Beta bugs fixed in Canary, I suggest being careful about future Canary, maybe not installing right away, and reading Releases category for clues. Settings can also be changed from Canary to Beta (or perhaps Experimental) to avoid surprise issues.

v2.0.3.6-2.0.3.6_canary_2018-04-23

Beware that this update contains a lot of new code, and should only be used in test environments.

was one example of one where the release note specifically warned, so don’t put those into production. Eventually things get better, and the trick is to try to find a point for a Beta when things are at their best. Scheduled releases (and scheduled slowdowns) have been discussed, but the idea hasn’t caught on…

master turns into Canary which turns into Experimental which turns into Beta, so in a sense there’s no coordination from a what’s-where viewpoint because it’s continuous. The trick is to find a stable spot… Nobody’s been eager to make this complicated, but it’s not clear how long current process will hold up.

Discussion: release cycle shows the above flow, hits hard spots on specifics, and shows links to it from process discussions such as this one which you can follow back to see what the other discussions said.

Can you give a concrete example based on the current changelog which says pretty much everything? What’s missing is “but how good is it now” which isn’t an easy thing to say when it’s initially announced.
Canary = new features with unknown bugs plus known fixes to known bugs, hopefully not adding more.
Beta = known bugs that are fixed in Canary, but lots of people running it, so it’s a better-known release. There are also fewer features (if it matters), but if a Canary release note has a feature you want, try it if you’re willing to put up with whatever bugs it might have, or whatever other bugs have been introduced.

That’s my view on where things stand now. Possibly other people will add in some additional input on it.

3

https://github.com/duplicati/duplicati/branches/all shows the branching scheme. I’m not very familiar with GitHub, but have worked in environments that had release branches. You can see Duplicati’s branches, however one difference from my large-company experiences is that Duplicati tries not to handle multiple branches concurrently (which is complex even with tools to help place a given change in many at once).

The branch creation does allow additional changes to continue in master and Canary without bothering release branches, but the usual goal is for a quick ship rather than a lot of parallel branch maintenance. The release branch start could be earlier, but then its change management becomes more complicated.

https://github.com/duplicati/duplicati/commits is the flow of Duplicati commits. There are probably some techniques for reducing tracking, decisions, and movement to a tolerable level. I don’t know specifics…

If you have ideas on how to do this a bit better for a small resource-limited volunteer project, please say.

4

Thanks, @ts678. Clearly there’s more release planning going on than is evident to the casual observer.

Yes, I meant Canary, not Beta, so the argument is a bit circular but the basic point of the post still holds.

What’s involved in this role?

And that’s where it falls down. Those projects work because there is (typically) a central authority which designs and manages the whole affair. And everyone gets paid (hopefully well) to obey that authority. Open Source project managers (or teams) typically don’t have that sort of clout or it’s not their full-time gig, so there has to be compromise.

Anyway, I don’t want to belabor the point too much, it appears I’m not breaking any new ground with this thread. I appreciate your taking the time to listen to my venting & respond gently.

HG

5

I suspect there’s flexibility. The hints dropped have included picking up some of what @renestach once did, and this predates me, but the old web site has some clues in the form of the sort of feature-oriented publicity that you might have been looking for. 2nd beta version of Duplicati 2.0 now available made it to current forum home, and you can also see his public-facing work in the forum Announcements category.

Then there’s the behind-the-scenes task of deciding what goes where in terms of something like GitHub branches, and maybe deciding when it’s time to release what. There are also some hints of planning of directions based on surveys. Rene also participated in general forum discussions, and sought feedback.

Get In Touch describes some of the early Duplicati history. I think @kenkendk was mostly writing code. Contributors to duplicati/duplicati shows the history with a large set of infrequent contributors (who need some way to contribute without having to learn about a complicated process), and frequent contributors who could perhaps guide. You can see current frequent contributors on the release notices whose style recently (probably due to change of writer) is more at the code level. I think both styles have their value.

Thanks for dupReport by the way. A lot of above looks much like what you’re doing. To quote your page

Bug reports and feature requests can be made on GitHub in the Issues Section. Please do not issue pull requests before discussing any problems or suggestions as an Issue.

The discussion group for dupReport is on the Duplicati Forum in this thread.

Please follow dupReport on Twitter @dupReport

and above that, I see you talk about your code branches and what inspired the plan. Duplicati’s is more user-facing, but Channels is the 2016 article that I could have pointed you too, but it’s not totally clear if Experimental was a pre-Beta then, whereas it is now. I guess the text still fits, so look at it all if you like.

Duplicati discussions that have gotten down to the what-can-we-do-with-Git? level include this and this, however everyone has recently been trying to get a long-needed Beta out, and discussion will continue, however to pull it off takes a combination of a workable process, and some volunteer able to help drive.

Probably the best next step if you have an interest in any of this is to PM kenkendk, however with many volunteers around, there’s probably an additional level of discussion. As you said, it’s not just top-down, however the main contributors seem to want Duplicati to succeed (duh) and seem to work well together. One question is how much time you have, but Kenneth has little, so any support might be very valuable.

6

What version should I be running for production (understanding nothing beta is “production” and another backup should also be run, like with a native machine software to a local drive.

It’s great reading all about what’s going on with Duplicati. One of the Canary releases (not sure which one) said “close to beta” but I have to admit, I am so confused and apprehensive for any of it. The website home page indicates “Latest and Greatest” V2.0.4.23. So is this the answer? Or a Canary “almost ready” to migrate to beta a safe bet? Oh, yeah… what comes after Beta?

7

https://www.duplicati.com/

Clickable box says “Download Duplicati 2.0 (beta)”

https://www.duplicati.com/download

If you hover over the clickable boxes you can see what they are in their links, and they say beta.

I’ve been feeling kind of bad about “Download the latest and greatest version of Duplicati here”, however trying to get someone to reword that might have taken some work. It’s now latest beta, however because it’s basically November 2018 beta because 2.0.4.23 was just a new warning,
Canary’s probably surpassed it on “greatest” (probably – until a bad one shows up) and “latest”.

The clickable box “Other versions – older or testing” adds Canary, Experimental, and the source. Browse around there if you like. It also has all the release notices in reverse chronological order.

For my own use, there is no Beta that holds up, mainly due to “Unexpected difference in fileset” happening too often. This was fixed in v2.0.4.22-2.0.4.22_canary_2019-06-30, but later fixes are present as well. Because some people are doing fine on Beta, I’ve only been advising Canary if something that’s bothering them is fixed, however many people find current Canary working well.

See previous remarks about how to protect oneself somewhat from surprises if one goes Canary.
There is no universal answer. See previous remarks about known fixes, versus unknown bugs in features and fixes that got added. Also see remarks about how Canary reliability varies with time. Sometimes there’s a warning (see 2.0.3.6 example). Sometimes unexpected problems show up.

Canary in the coalmine is probably what “Canary” as a release name came from – its job is to do warning duty (by showing problems to volunteers who run it) early enough to save later channels.

8

Well… I think the work is doing the planning you are looking for.

A release manager would need to make the “go” decision for when we decide to branch of the canary build for more stable.

The easiest and quickest way to fill the role, is to look at open issues and forum posts, and then making a decision about what needs to be fixed before we move.

If you are comfortable with Git, you can create the release branch, and cherry pick away. If you do not like to work with Git branches, someone else can take on that part.

9

could also include asking the staff who deal with this daily. There are a ton of old issues being chipped away at, but you’ll probably develop some guidelines of your own such as “good fixes and few breaks”.

Critical issues with the current canary [2.0.3.9] was the most formal “ask the staff” I’ve seen, but it was after the huge rewrite of 2.0.3.6. Ordinarily it doesn’t take much tracking, though one attempt is kept at Upcoming beta, though it’s not clear how the milestone is maintained or how super-seriously it’s taken.

“Google Drive (full access) login” restricted in early 2020 #3875 was a recent question on whether it’s a blocker or not. My response on the milestone followed, and I gave my opinion that we should Beta with this undone (partly because the task seemed poorly defined) but might need to push another Beta later.

10

So the role might be called, “Master Beta Manager” ?

1 Like