There was a dramatic change in reports last month, with a significant increase in the number of FAIL and UNKNOWN reports. This is in part to a change in the way that some of the reports were being generated. It has been discussed on the CPAN Testers Discussion mailing list and on the Perl QA mailing list, as well as between the main developers of the tools, and several have been generated due to libraries being unavailable on the testing platform and a FAIL or UNKNOWN report being generated. These reports should have been treated as a missing prerequisite and not have been generated for the tested distribution. However, as the installer tool chain treats libraries and other Perl distributions differently, this makes it a little tricky to notify the automated testing tools that the reporting should be abandoned. Thankfully Andreas and David have quickly patched and released upgraded versions of CPAN and CPAN-Reporter, so if you're using that side of the tool chain please upgrade now. CPANPLUS already has a mechanism to detect library failures, so YACSmoke users should not being the same issue, however if you do, please let me know so I can investigate further.
We broke several records for last month in the history of CPAN Testers. Aside from the highest FAIL and UNKNOWN reports being submitted, we also had the most PASS and NA reports in a single month, the most testers in a single month, the most platforms tested in a single month, the most perls tested in a single month, the highest number of reports submitted from a single tester in a single month. But then we did also have the most UPLOADS in a single month too, so it's perhaps not too surprising we've seen some high numbers during September.
This month also the first reports being submitted for 5.10, all 15 patch variants! It has made me wonder whether merging the patched versions into their main release version would be a good thing, to simplify the stats. I'll keep them as they are for now, but anyone has a strong opinion one way or the other, let me know.
Currently in the CPAN Testers Statistics database there are 556 unaccounted tester addresses, with 1318 addresses mapped to 558 real testers. Only 1 new tester spotted this month, although I've added 31 new address mappings. This is great in advance of my sixth month request for confirmation, as there are several addresses that just use 'root' or similar, and it makes it difficult to know they are owned by. So hopefully when I get the chance to send the mails out, at least some of them will reply with more details. If you get a mail, please take the time to update me.
Even though only 1 new tester was spotted, it is wonderful to see several previous testers, who have dropped off for one reason or another, submitting reports again last month. So much so that we had 120 testers submitting reports in September, the highest we've ever had and the first time we've had over 100 testers in a single month. However, it should be noted that ten only submitted 1 report ... but they all count :)
Virtually all the top testers upped their game last month, with all but David Cantrell in the top 6 beating their previous month's number of submitted reports. Chris had another phenomenal month, submitting over 25,000 reports, such that his total in the leader board is a few reports under the combined submissions of his next 4 rivals. David made the leap from 4th to 2nd, while Andreas leapt from 7th to 5th, although I expect at the current rate Andreas will be in the 3rd place by the end of the year. Gabor and Slaven have also made notable jumps further up the standings and following their increased submissions after this month, I expect we'll be seeing them both in the top 20 shortly. The biggest jump this month is from Johannes Plunien, from 33rd to 22nd, who as mentioned last month only started testing in August. It just goes to show that you don't need to be a tester for long to make a worthwhile impact in CPAN Testing.
If you want to get involved in CPAN Testing there is now a couple of established places to get help and advice. The definite resource for setting things up is the CPAN Testers Wiki, with the CPAN Testers Discussion mailing list your first point of call for advice or suggestions for improving the tools or resources. There are some changes planned for the future, so if you're willing to muck in and help out, your help would be greatly appreciated. See the discussion mailing list for more details.
I just want to clarify the changes to CPAN/CPAN::Reporter.
CPAN::Reporter does not check for a "missing library" string as CPANPLUS does, as I am not convinced that this is a robust approach. The actual ExtUtils::MakeMaker string includes the phrase "(probably harmless)" -- which implies that a harmless missing library could cause a legitimate test failure to be discarded, which we don't want.
Instead, the change to CPAN fixes a bug where CPAN continued to try to make even if a Makefile was never created (resulting in a FAIL report).
Going forward, a Makefile.PL (or Build.PL) can simply "exit 0" without generating a Makefile (or Build file) and CPAN will not continue a make attempt, nor will CPAN::Reporter issue a FAIL report. This "graceful exit" gives module authors a mechanism to check for non-perl dependencies and refuse to continue if they are not found.
To make this process and the clumsy method for signaling "OS unsupported" easier for authors, David Cantrell has volunteering to put together some modules to help do the right thing in Makefile.PL. The first of these, Devel::CheckOS, has been released already.
-- dagolden
Re:CPAN/CPAN::Reporter and library failure
barbie on 2007-10-02T17:04:11
Thanks for the clarification David, and EvilDave++ for doing all the work on Devel::CheckOS.
