For a talk I'm planning to present, I've been working through the complete NNTP archive of postings to the CPAN Testers mailing list. There are over 200,000 now, which is quite a phenomenal amount. However, as well as all the well defined CPAN Upload and PASS/FAIL/NA/UNKNOWN report postings, there are quite a large amount of other posts. Many are conversations following fail reports, some are questions or announcements and others are simply spam. There are a batch that are completely missing, due to a error in the PAUSE mailer some years ago, causing several thousand posts of the same mail. These are all easy to spot and ignore.
However, not so easy to ignore and causing my adapted parser from WWW::CPAN::Testers::Generator to fail, are several incorrectly posted reports. Most appear to be hand written reports, but I suspect there are a few that are from automated processes. Seeing as most are from several years ago, it's not worth following them up. But a few from this year I thought I would. Two reporters have replied, who it turned out had just created a few manual reports. Now knowing the problem, if they need to do manual reports again they know the correct format. I'll try and keep an eye out for any others.
So what is the problem? Well, as Acme's distribution trawls the NNTP postings to create the CPAN Testers website, it would be a shame for reports to get ignored. However, as some can't be parsed, they do get ignored. In virtually all instances it is simply the subject line that is incorrect. In several the 'perl -V' output is missing. They are both simple things, but quite crucial to successfully parsing the reports.
Best to give an example:
BAD: PASS XXXX::XXXX Linux-1.2.3 BAD: PASS XXXX::XXXX-0.12 Linux-1.2.3 BAD: PASS XXXX-XXXXX Linux-1.2.3 BAD: PASS XXXX-XXXX-0.12.tar.gz Linux-1.2.3
GOOD: PASS XXXX-XXXX-0.12 Linux-1.2.3
The key point is how the distribution is named. It is named as the directory that it would usually be unwrapped into, once CPAN/CPANPLUS have downloaded it. This means it only has dashes (-), no colons, between the namespaces. It also must include the version number, otherwise there is no way of knowing whether this was an old version or the current version you were testing. All extensions should also be removed. And please, please include the output of 'perl -V'.
The regular testers all use automated scripts and distributions to test and produce reports, and have it covered. It's the occasional testers or sometimes new starters that need a bit of guidance. As such, as part of the yacsmoke project, I'm planning to write a practical guide to cpan-testing, so in future I can point people to that.
Of course no-one on use.perl posts bad reports do they ;)
Re:
barbie on 2005-05-25T08:34:46
Well I have a series of scripts that are doing that for me, that I run by hand. However, it really needs a human to review the results, just in case the mail is simply part of a conversation and not a report. I'd be very reluctant to have anything automated. Plus these erroneous reports are only posted occasionally (about 20 so far this year) and usually in a batch by the same person, so a simple mail is probably enough to point them in the right direction.I don't want to discourage anyone wanting to help with cpan-testing, and canned messages might give the wrong impression.
Re:
Aristotle on 2005-05-25T09:10:58
Ah. I was attempting to address a situation different from what you describe.
