At around 4am PST today, the last of the Firefox 3 Beta 1 release candidate builds appeared on our public FTP. This was mistakenly reported on Digg as the official release of the first Firefox 3 Beta.
— Mike Beltzner, Weââ¬â¢re happy that you Digg us, but...
Numbered beta releases have numbered release candidates now? C'mon MozCo, just stick the phrase Street Fighter in there somewhere so we all know that this is satire. It is satire, right?
… version numbers mean something?!
Re:Wait…
chromatic on 2007-11-08T05:46:37
Having release candidates for betas means you lack confidence in your ability to produce releasable software. This is a sign of quality deflation. This is not good.
Re:Wait…
Aristotle on 2007-11-08T06:26:07
Having release candidates for betas means some of any number of things.
Re:Wait…
btilly on 2007-11-08T16:52:25
Not necessarily.
Suppose that you have an official goal that betas will work on some fairly large set of platforms. Suppose that your key developers do not use all of those platforms. Suppose that the people you have who can test those platforms are not interested in testing all of the experimental versions.
In that case you should have release candidates for betas. The fact that it is a release candidate is a sign that your key testers should test it to be sure that it meets the official goal for a beta release. Once the key testers have verified that, then you make a beta release. And if they prove the opposite, then you'll be able to fairly quickly turn around a fix, and you'll avoid a thousand similar bug reports for a stupid problem.
Furthermore in that case you would legitimately be unhappy that the candidate was widely publicized. The point of having it was to avoid a thousand redundant bug reports in the event of a thinko. And publicity interferes with that.
Please note that such a procedure and policy could only make sense if you had a product with a very large and wide user base of widely varying abilities that are interested in beta testing. This obviously applies to end-user software like Firefox. It does not apply to Perl.Re:Wait…
chromatic on 2007-11-08T17:21:36
I thought the point of releasing a beta version was so that people could test it.
I thought the point of releasing a release candidate was so that people could test it.
I thought the point of releasing a nightly build was so that people could test it.
If you're going to gripe that too many people tested a version of the software you released for people to test because it wasn't ready for people to test, you have a problem completely unrelated to people outside your organization testing your software, and that's that you didn't prevent the release of a version of your software that wasn't ready for external people to test.
Feature freezes won't fix this. A series of release candidates won't fix this. If you really want to catch stupid thinkos, you have to test for them in an automated fashion instead of relying on magical volunteer effort which, in my experience, doesn't materialize until you actually release software that you want people to use.
Re:Wait…
Aristotle on 2007-11-08T22:17:56
you didn’t prevent the release of a version of your software that wasn’t ready for external people to test.
This isn’t the first time they put RCs of betas on their nightly build FTP, it’s the norm. It has always worked in the past. Now consider how much heat without light would be generated by trying to regulate access to these builds. And then consider that the source code they’re built from can be checked out of the repository via a tag, anyway, since, after all, they’re an open source project. Just how do you suppose that Mozilla should fix the problem they’re purportedly having?
And to what end – so some reading-challenged Digg user (“it’s the crowd the Slashdotters laugh at!”) doesn’t end up sending a avalanche of equally comprehension-challenged other Diggers their way? (I don’t like falling into clichés like that, but Digg really is scary.)
(Where did you “quality deflation” argument go, btw?)
Re:Wait…
chromatic on 2007-11-09T07:48:46
(Where did you “quality deflation” argument go, btw?)Thanks for reminding me--it's actually a very important point related to my whole line of thought. The best place to start reading about it is Better Testing, Worse Quality, by Elisabeth Hendrickson.
Just how do you suppose that Mozilla should fix the problem they’re purportedly having?There's no general fix for the problem where someone sees an empty directory show up on the master server for a new version and posts that link to the Scary Undernet.
There's a good (though not easy) fix for the problem where Mozilla needs to know specific information about a particular revision number before the developers feel comfortable releasing a build at that revision to the wider public. All Mozilla has to do (again, I never said this would be easy) is to figure out the specific information necessary to decide if the project is releasable and then figure out how to get that information and to keep the project at that state reliably.
I know projects that can do this. One in particular could make a new stable release more than once a day if someone wanted releases that frequently (and I don't know if it has more or fewer users than Firefox, but it has millions of users). There's none of this busy-work about "Oh, it would be nice if we could have more confidence in our beta software, so we're going to have a nice feature freeze for a few weeks and then some release candidates, and now we really promise to start fixing these bugs for sure."
I know that this has been Mozilla's policy for a long time, but I'm really starting to think that software development processes that encourage long feature freezes and release candidates are actually counterproductive in a lot of important ways.
Re:Wait…
btilly on 2007-11-09T00:28:30
Yes, I know that you've helped developed Perl, and I know how widely deployed Perl is. But there are a couple of orders of magnitude more Firefox users than there are Perl programmers. While in your experience people don't show up to test software that isn't ready, Firefox expects tens of thousands of people to download and try out a beta. Many will use it as their primary browser.
Take a look at http://wiki.mozilla.org/Firefox3/Schedule and see the difference between alpha and beta software. As you see, beta software is supposed to meet a quality standard that includes being appropriate for daily browsing use by a large number of people. If that is your standard, then you need to have it go through a certain level of validation first. And yes, having scheduled release candidates would be a reasonable way to do some of that validation.
This is in addition to, not instead of, other forms of validation. Which should include code review, unit testing, and so on. And no, I don't know whether or not Firefox is doing those things. Nor am I defending their quality. But it really does make sense to me to have release candidates for a beta release that you expect to wind up in wide distribution.
