My personal overall opinion is that CPAN Ratings has been a mild success. Key to its usefulness is that all these reviews, noise or signal, are easily reachable directly from the CPAN search itself. This close association makes it better equipped to be relevant than any other venue we currently have.
Of course, most reviews are just about useless and donââ¬â¢t manage to tell you more than that the author thinks ââ¬Åthis module is great!ââ¬Â This is no surprise; reviewing well requires empathy, which is a talent few people have; particularly relating to an audience. Reviews which show that the reviewer has put the module through its paces or at least taken an in-depth glance at it, however short, are relatively few and far between. Still, their absolute number is high enough for the system to be useful, in my opinion. Iââ¬â¢ve spent quite a bit of my own time trying to write reviews I hope are useful, in hopes of contributing positively to the system.
What really annoys me are the cases of which this one or that are only the latest examples: people use the rating system to write bug reports or to respond to a review. These things are not reviews! They donââ¬â¢t belong where they are posted.
With bug reports, the case is clear: they belong on RT, not on CPAN Ratings. One might argue that grave bugs might warrant a rating (and thus review) all by themselves, but then how many people are there who will actually go back once the bug has been fixed and amend or retract such a ââ¬Åbug review?ââ¬Â Better to keep these things were they belong. Letââ¬â¢s not forget that someone who checks the ratings can easily take a look at the bug tracker as well.
With commentary on reviews, the case gets murkier. CPAN Ratings does not provide a way to contact the author of a review nor does it offer a way to comment on a review. Whatââ¬â¢s worse is that the lack of a discussion platform is compounded by the abuse as a bug reporting system. I do not have a link handy, but Iââ¬â¢ve seen authors respond to bug reports on the rating system byââ¬Â¦ writing a post on the rating system. Frankly, I have no good ideas on how to solve this problem. The most practical approach (note: not solution) might be to allow posting ââ¬Åreviewsââ¬Â which carry no rating so that they can be used for responses and comments. Still, the rating system is not a discussion forum (and in my opinion, rightly so), so the format will always be ill-suited to discussion.
And of course, the question which remains is: how do we get people to direct comments (even if we knew where those should go) and bug reports to their respective, correct venues?
Maybe one should construct a commenting system for the reviews. I.e: be able to comment on the review and hold a discussion about it. Either a nested discussion, but it's possible a flat one will also do.
This will resolve most of the problems with the current system. Gabor Szabo has already implemented CPAN-Forum, so one way would be to post the review on this module forum, and place a link from the review in cpanratings to ths discussion in the forum, with a text of "comment".
How about some review guidelines just above the "Review" box? Such as:how do we get people to direct comments (even if we knew where those should go) and bug reports to their respective, correct venues?
Re:Bug report redirection
Aristotle on 2005-04-10T23:39:23
Hey, thanks for the comments. I am aware that this is not the bugreport forum for cpanratings
:) (though I’m not sure what is), but I wasn’t reporting bugs so much as hoping to kick off a bit of discussion. (Maybe I should post to Perlmonks as well.) Some explanatory text would certainly help. It won’t suffice, I think, but it sure would be an improvement over the current situation. Having ratings mailed to authors is a good suggestion, but I’m not sure how it relates to my points; was it supposed to, did I miss something?
As for the ratings: basically, to me, 1 means “I recommend against using this” and 5 means “I can recommend this module to anyone in good conscience.” The intermediate ratings obviously signify recommendations with varying degrees of caveats.
Do you have any suggestions on the general “talking back” issue? I think that is the most important to address, and it won’t easily be helped by a bit of explanatory text or such either.
Re:Bug report redirection
schwern on 2005-04-11T00:13:28
I am aware that this is not the bugreport forum for cpanratings:) (though I’m not sure what is) About says to email ask@perl.org.
(Maybe I should post to Perlmonks as well.)Speaking of places where bugs are reported which never get back to the author.
Having ratings mailed to authors is a good suggestion, but I’m not sure how it relates to my points; was it supposed to, did I miss something?I was just sort of on a roll at that point. If nothing else it means the authors might actually read the ratings and consider them. It also means that should someone report a bug via CPAN ratings the author will see it.
As for the ratings: basically, to me, 1 means...Which is the whole point. Rating numbers mean different things to different people. One needs to assign a standard meaning to each number. Or better yet, eliminate the discrete numbers entirely and just use a slider with descriptions of what each range on the slider means.
Do you have any suggestions on the general “talking back” issue? I think that is the most important to address, and it won’t easily be helped by a bit of explanatory text or such either.Email still works.
:) Just about every author has an user@cpan.org address. Its best way to guarantee the author, any author, will get your message. rt.cpan.org works for bugs as it emails the author.
The rest don't push content back to the author so they're unreliable. Maybe the author reads it, maybe they don't. This includes cpanratings and the new CPAN Forum. CPAN Forum has some support for message push but you can't say "push me content about all my modules" you have to sign up for each one individually and I'm not about to bother for 50+ modules. Its also opt-in and most module authors don't even know it exists.
So the problem common to most existing ways to bitch about modules is that the people who can do something about it, the author, likely never sees it. I don't read Perlmonks. Every once in a while I sweep it for people reporting bugs about my modules and throw up another "talk to the author, damnit!" rant but that's work. If any feedback system is going to work it has to:
- A: Push content back to the author.
- B: Be opt-out, not opt-in.
- C: Carefully manage the volume so as not to annoy the author.
rt.cpan.org and user@cpan.org succeeded in this despite many authors howling that they didn't want it imposed on them and never wanted to hear about it. An opt-out option for rt.cpan.org was proposed for those who objected the loudest but I don't think it was ever implmented as, in the end, it turned out to be a great thing for all.
Re:Bug report redirection
Aristotle on 2005-04-11T15:00:53
Speaking of places where bugs are reported which never get back to the author.
Uh, I did say that the intent was to get some discussion going to exchange ideas maybe, not to report a bug. The issues I brought up are no bugs; see also the title I chose.
Just about every author has an user@cpan.org address. Its best way to guarantee the author, any author, will get your message.
I agree, but that isn’t the direction I was talking about. I was wondering how the author (or someone else) could respond to a review(er), because that is what currently goes wrong in the system. The reviews seem to reach authors just fine in many cases: several authors have reacted to my reviews, one of them even had the sense to search long enough to find my email address. It won’t hurt to make this more convenient for authors too, of course, but that's not the pressing issue.
As for bugs, yes, I had already mentioned that they shouldn’t be reported on cpanratings in the first place.
Re:Bug report redirection
schwern on 2005-04-13T20:12:36
Speaking of places where bugs are reported which never get back to the author.
Uh, I did say that the intent was to get some discussion going to exchange ideas maybe, not to report a bug. The issues I brought up are no bugs; see also the title I chose.Just a general hate flung in the direction of Perlmonks. You just happened to be in the path.
I agree, but that isn’t the direction I was talking about. I was wondering how the author (or someone else) could respond to a review(er)Oh, that's pretty simple.
- Push the reviews to the author.
- Let the author publish his response to the review on cpanratings (by simply replying to the review email sent out by cpanratings)
- Allow the reviewer to retract his review if the author has satisfied his problem or if a later version of the module changes his view. Make these edits/retractions visible so people can see the author is responsive, a point of review itself.
Re:Bug report redirection
Aristotle on 2005-04-13T20:50:34
Ah! Of course. Noone but the author really needs to respond to a review. Good thinking. I’ll add one minor point: an author should not be able to post “self-reviews.”
I think together with the stuff I wrote in reply to Ask that makes a cohesive set of proposed changes to mitigate the problems we currently see. I’ll summarize these in an email to him and see if I can lend a hand with the implementation.
Re:A few comments
Aristotle on 2005-04-11T00:33:47
Hey, thanks for dropping by.
:) I’ll see about how to be more useful (and where I can squeeze this into my crowded headspace (finding time is not really an issue)). As I said, mostly I wanted to stir up some talk, because I can see this issues but I’m not bright enough to have solutions. I suppose “was this helpful?” is already on the todo list as “Helpfulness ratings”? That sounds like a good idea, yes. I thought of a meta/moderation system, but I’m unsure if the place gets enough traffic to sustain such. OTOH it can’t hurt in any case. I was pretty sure comments were consciously decided against, and I agree that it is the right choice in this case. I’m just wondering what could possibly be done in order to keep “discussion ratings” out of the place, because goodness knows it is ill-suited and ill-chosen for that purpose.
After some thinking, how about the following suggestions?
The “add review” link would be removed and only the distro page on search.cpan.org would have a link for writing reviews. You can still write a review from the ratings page, but it would take a roundtrip through the distro page. The intent is to associate writing a review with the distribution rather than with the ratings, and to discourage the reflex to “reply” to a rating.
There would be some kind of “respond” link which either sends mail to the rating author or leaves a private message to him somewhere in the system (the latter option requires extra infrastructure of course).
The idea is that moving these links around this way would act as “syntactic salt” for doing the wrong thing, while funnelling people towards other venues. How do these sound?
