Gabor Szabo requested comments on several new CPANTS metrics. Short comment: these are the worst metrics yet.
Medium comment: Yeah, yeah, Kwalitee isn't a measure of Quality. I get that, but it's no excuse for piling on even more lousy metrics that not only don't know Quality, but have never even bumped into Quality accidentally at the grocery store.
Longer comment:
80% of these new metrics only measure Kwalitee of a distribution to Debian users. That may be useful for Debian users, but it's highly inappropriate (and worse, irrelevant) for everyone else. (I believe Ubuntu has a bug somewhere that not all of the world uses a Debian-based distribution.)
Constructive criticism: How about a way to opt into only the useful CPANTS metrics? I'd hate for users of my code to get confused by meaningless Kwalitee Theater.
Wow. I have to confess that I can't see much point to those. I would much prefer to focus on serious issues which clearly point to Kwalitee. Not many people debate strictures, but Test::NoWarnings? I often have code to specifically disable that module because it's lovely, but problematic.
Kwalitee shouldn't be focusing on specific modules, but I can see where he's going with the Debian stuff. But yes, there should be greater control. I'd prefer "core" Kwalitee metrics with the other issues being "opt-in". Fewer complaints. If people want to play, great! If they don't want to play, that's great too.
There is no nOops
sjn on 2008-06-09T13:19:24
Well, with metrics like these, CPAN module developers can get a small link into the Debian package review process, can't they?
"Given enough eyeballs, all bugs are shallow", IIRC. Putting a module into a larger context (more users => more developers => more eyeballs => more feedback) SHOULD increase the quality of the software - at least over time.
Wether or not to make a "competition" of this metric is another issue, and possibly solved by splitting up the CPANTS games into smaller ones, where the "Software Kwalitee" index is one, and "Debian-Perl Feedback Kwalitee" or "Redhat-Perl Feedback Kwalitee" are separate.
CPAN is no longer a closed environment, and it's on high time we formalise feedback channels across to the different communities that depend on CPAN modules and their authors.
And even better, by accepting Gabor's proposals we're taking a first step towards making CPAN a one-stop shop for Kwalitee products.:)
I think it would be good to make this work.Re:There is no nOops
petdance on 2008-06-09T13:36:29
All of this kwalitee hoohah assumes that authors give a crap about their point score.
I wonder what percentage of CPAN authors indeed give a crap. I'm guessing, what, maybe 1%, max? Let's call it 5% to be generous. I'd suggest that the better time would be spent on increasing the author_gives_crap score than adding more arcane little tweaks that can only damage the current author_gives_crap rank.Re:There is no nOops
sjn on 2008-06-09T13:55:06
Remember that author_gives_crap just is a product of all the other metrics.;-D Re:There is no nOops
petdance on 2008-06-09T14:08:21
My very point. I can't imagine that does_debian_whatever is going to help the author_gives_crap rating.My concern with CPANTS all along has been that the metrics are being written for the CPANTS developers, not the CPAN developers.
Re:There is no nOops
sjn on 2008-06-09T15:22:49
Sure it helps. It helps me as a user of some module to see if it's author gives_crap about Debian. If he doesn't, then I'll at least know in advance that the module may need some Debian love, or that I may have to look for another one that can fit my needs.
Of course, this assumes that the author _wants_ feedback, but I think that's a pretty safe assumption (and those who don't want feedback, probably trickle towards the bottom end of the CPANTS ranking anyway.)
My point is that metrics like has_no_bugs_reported_in_debian can give a useful level of precision for determining if a software project is worth my attention.Re:There is no nOops
petdance on 2008-06-09T15:27:38
My point is that there are so many authors out there that are probably entirely unaware of CPANTS, and more still that know about CPANTS and think it's silly. I think it would serve CPAN better if more people were to use the good stuff in CPANTS more than tweaking what's already there.Re:There is no nOops
sjn on 2008-06-09T23:45:53
What you describe here is a different type of problem, requiring another solution - e.g. a CPANTS rating score + link to more info, located somewhere on the main package description page found on CPAN.Re:There is no nOops
sjn on 2008-06-09T23:49:22
Oops, this was meant for petdance-xdg:-P Re:There is no nOops
chromatic on 2008-06-09T17:31:05
It helps me as a user of some module to see if it's author gives_crap about Debian.That's exactly backwards.
These metrics only hint if Debian cares about the distro. That may be useful to you if you use the Debian-packaged version, but it's useless to everyone else. If the distro has a Debian package, that means that some Debian developer went through the work to package the distro for Debian. If the distro doesn't have a Debian package, that means some Debian developer didn't go through that work. That's all.
Similarly, the presence or absence of open bugs in the Debian queue is a useless indicator. Those bugs may never come to the distro author's attention. CPAN already has a request tracker. Any bugs which aren't Debian specific should go there -- and again, Debian-specific information is only useful to people using the Debian-packaged version.
If you still don't believe me, consider using "Packaged in ActiveState's PPM repository for Perl 5.8.x" as a sign of Kwalitee. A significant chunk of CPAN couldn't meet that metric, not because of any lack of quality on the part of the authors, but because ActiveState stuck with a poor technical decision which made certain versions of XS portion of Scalar::Util unavailable.
How does that possibly reflect on the quality of anything other than ActiveState's repository?
Re:There is no nOops
sjn on 2008-06-10T10:54:48
These metrics only hint if Debian cares about the distro. That may be useful to you if you use the Debian-packaged version, but it's useless to everyone else. If the distro has a Debian package, that means that some Debian developer went through the work to package the distro for Debian. If the distro doesn't have a Debian package, that means some Debian developer didn't go through that work. That's all.
And conversely, if the distro HAS a Debian package, it means that this Debian developer trusts the distro enough to put it into Debian. It's not the work that is important, it's the question of trust that is.
Similarly, the presence or absence of open bugs in the Debian queue is a useless indicator. Those bugs may never come to the distro author's attention. CPAN already has a request tracker. Any bugs which aren't Debian specific should go there -- and again, Debian-specific information is only useful to people using the Debian-packaged version.
Open bugs in Debian could mean one of several things (and if you use your imagination, I'm sure you'll find a few more):
- The Debian dev doesn't trust the distro completely, only just enough to let it into Debian with some modifications.
- The distro has platform-specific issues, perhaps including assumptions about install locations that aren't compatible with Debian
- The original distro author is unresponsive to requests about bugs, so the Debian dev is forced to make local changes in order to ship something that works.
ALL these points are useful to know about if you are considering using the distro.
If you still don't believe me, consider using "Packaged in ActiveState's PPM repository for Perl 5.8.x" as a sign of Kwalitee. A significant chunk of CPAN couldn't meet that metric, not because of any lack of quality on the part of the authors, but because ActiveState stuck with a poor technical decision which made certain versions of XS portion of Scalar::Util unavailable.
Hm... I never considerd ActiveState's product as something comparable to an OS distribution like Debian or Redhat or FreeBSD. You may have a point here, but even if so - it's useful to me to know that ActiveState trusts a CPAN distro.
How does that possibly reflect on the quality of anything other than ActiveState's repository?
It tells about trust, availability and working communication lines between CPAN disto developer and distribution platform developers. Each of these are signs that there has been done some kind of judgement about the software, and if the signs are there, it means that the software has at least a good likelyhood to improve, even if other Kwalitee metrics don't measure up.
Re:There is no nOops
Alias on 2008-06-10T01:53:11
Your response makes the common mistake of assuming that we can only do one or the other, which is of course not true.
If there's isn't something for authors to care ABOUT, there's zero point in making them aware of it.
It's quite reasonable to work on making the kwalitee metrics more accurate, more relevant, and less vulnerable to cargo cults.
That way when authors DO give a crap, at least they have good information to work from.
And the better the information, the more likely they are to agree with them, once they are aware.
Awareness is completely orthogonal from analysis, and we can easily do both.
There's no need to choose.Re:There is no nOops
petdance on 2008-06-10T05:35:44
Exactly. I look at http://cpants.perl.org/author/PETDANCE and it tells me that many of my distributions fail certain checkboxes. So what?Here it says WWW::Mechanize has no README. So what? Why do I as an author care?
It says that Mech's META.yml doesn't conform to a known spec (At least that's what I think the arcane code in the hover box tells me). So what? Why do I as an author care?
My META.yml doesn't have a license in it. So what? Why do I as an author care?
Perl::Critic::Bangs fails the test "proper_libs" which tells me " Move your *.pm files in a directory named 'lib'. The directory structure should look like 'lib/Your/Module.pm' for a module named 'Your::Module'." It should? Why do I as an author care if I don't put my
.pm file in the lib/ directory? It's only overstating slightly to say that the entire CPANTS structure seems to be built upon the premise of "These are things that should be a certain way because I say so," whoever "I" may happen to be.
That's not to say that the things checked for aren't worthwhile, but nothing says WHY they are worthwhile.
Further, and worse, it's presented as "You should just know this stuff and appreciate us for telling you."
Re:There is no nOops
sjn on 2008-06-10T11:00:31
This is actually a good point. There should be more documentation available about _why_ a Kwalitee metric is a good thing.Re:There is no nOops
petdance on 2008-06-10T14:11:09
Not just why Kwalitee is good, but why each individual data point is good. Let the author make the decision about whether it's to his advantage or not to have a META.yml that conforms to a spec, or a more likely example, whether it's worth putting out a release of a module that he wasn't going to work on for a while for the sake of putting out a META.yml that conforms to a spec.Re:There is no nOops
pudge on 2008-06-19T23:39:37
I don't use META.yml in any of my dists. I have no reason to. This apparently means I am bad not just on that score, but all the others that have to do with META.yml. ONOZ.
Re:There is no nOops
btilly on 2008-06-10T15:59:26
I just got curious and looked at my modules. They all got docked for not having a human readable LICENSE. So I looked. Under AUTHOR AND COPYRIGHT they say, This may be modified and distributed on the same terms as Perl. Somehow I think that humans would disagree with CPANTS on how human readable that is.:-)
All of my modules are docked for my deliberate policy of not introducing unnecessary dependencies. That includes warnings, Test::Pod and Test::Pod::Coverage.
Sometime I should get around to removing the accidentally included Build file and the missing author in the META.yml for Text::xSV. The rest I have trouble caring about.Re:There is no nOops
Alias on 2008-06-12T01:35:08
Most of those problem reinforce one of my main problems with CPANTS, which is that many of the metrics are very badly implemented.Re:There is no nOops
mr_bean on 2008-06-10T05:20:31
This is an important point, I think. But I'm not sure what you're talking about.can you give us an example?
This is test washback. We want a test that just doesn't tell us whether the author gives a crap, but in fact MAKES, or at least encourages, the author to give a crap. Or at least tells the author how to give a crap.
As it stands, each checkbox is just a rap over the knuckles for not having your shoelaces tied.
How do we learn to dress properly?
Perhaps we have to be working at a higher level here, rather than surface-level code specifics. How are we going to reward effort, where it may not be possible to discern effort from what we have as evidence before us.
Perhaps some randomization of results, as in gambling, or at least a LESS clear relationship between my code and my kwalitee from my viewpoint, might be motivating.
It's fun when you win. The danger is it is demotivating when you don't. So one rule to increase author_gives_crap score should be, everyone is a winner.
Irrelevant to CPANTS
chromatic on 2008-06-09T20:01:59
Measuring the quality of Debian-specific packaging makes a lot of sense for Debian developers and users, but it's irrelevant for everyone else. As you mention, pkg-perl already tracks these statistics. What possible value is there for CPANTS do to the same?
(Win32::OLE is no less useful because there's no Debian package for it and no more useful because its Debian bug queue is empty.)
Re:Irrelevant to CPANTS
Alias on 2008-06-10T02:01:18
If CPANTS can centralise tracking for all the main downstream channels, I for one would find it useful.
Also, I wouldn't write off Win32::OLE in Debian just yet.
I know of at least one person that has been installing Win32 Perl modules on Debian on top of wine.
As for Win32 modules getting a free "no bugs in debian" point, that at least won't cause any harm.
Re:Irrelevant to CPANTS
chromatic on 2008-06-10T02:10:12
As for Win32 modules getting a free "no bugs in debian" point, that at least won't cause any harm.No one has reported any bugs for my distros on ReactOS, Multics, MVS, Windows 7, VxWorks, or IOS. Some of those are in wide use. I struggle to decide why anyone who doesn't use my code on those platforms should care, and why someone should care to inform them on my behalf.
Re:Irrelevant to CPANTS
Alias on 2008-06-10T07:26:41
Because it shows that you pay attention to all of the downstream channels, no matter how weird, and make sure none of them accumulate unfixed bugs?Re:Irrelevant to CPANTS
chromatic on 2008-06-10T17:17:07
Trolling umpteen downstream bug trackers for operating systems I don't use times a few dozen CPAN distributions seems more likely to indicate that 1) I have too much spare time on my hands or 2) I should hand out my rate card more liberally.
If the BUGS section in my documentation doesn't say "RT please", it will, and then your indication of Kwalitee is that downstream packagers either can't or won't read.
Re:Irrelevant to CPANTS
TeeJay on 2008-06-10T11:57:53
If SomeThingElse can centralise tracking for all the main downstream channels, I for one would find it useful.
