For the past few months Robert and I have been hard at work on CPAN::YACSmoke. As Robert was the original instigator for the idea, it's been his baby to upload to CPAN. However, as I've increasingly expanded the functionality and fixed bugs (mostly mine ;)), the patches I've been sending have been getting bigger and bigger and much more frequent. As such we decided that we needed a code repository somewhere, where we both could check-in patches. The obvious choice was SourceForge. So as of yesterday, Robert created a brand new project....
We haven't designed a homepage or anything yet, and Robert has only just checked in all the code. We still intend for all the bugs/wishlists to be reported via rt, so if you have any suggestions please log them there.
For those wondering what CPAN::YACSmoke is, it a very resourceful front end to CPANPLUS for cpan-testing. Aside from the usual testing, that was previously handled by cpansmoke, it also allows the tester to manage what they test better. With cpansmoke you had to manually configure a list to test, unless you wrote your own smart script. With CPAN::YACSmoke the plugins allow you choose from several sources for the current list of distributions to test. As this list could contain many distributions that may not work on your OS, you can compile an excluded list, which filters out the difficult distros (such as ones requiring external libraries that aren't installed).
Each test then produces a grade. Four grades map directly to reports (NA, UNKNOWN, FAIL and PASS), but there are additional ones that don't produce reports (IGNORE, UNGRADED, ABORTED and NONE). Currently the lines are a bit blurred between UNGRADED and ABORTED in the code, but we know what they are *supposed* to represent :)
Robert has asked advice, on the perl-qa and module-authors mailing lists, regarding the NA situation. There are several distribution that are applicable to several OSs, but may have problems on some. As such, how does CPAN::YACSmoke or the CPAN author decided to report a NA report instead of a FAIL report?
There are some other thoughts on reporting we have been discussing too, but many of these require further patches to CPANPLUS, which I've been planning to do for a while now. We're also looking to expand the FAQ, so if you have any questions you think should be added let us know.
When I see a FAIL on search.cpan, I usually click on the link to see what failed. The reports returned by YACSmoke are very long, but they lack the information I want: the details of which test, in which test script, failed.
See http://www.nntp.perl.org/group/perl.cpan.testers/197254 for example.
Thanks in advance.
Re:The reports are not very useful, though
barbie on 2005-04-13T12:48:46
This is a problem with the new CPANPLUS rather than CPAN::YACSmoke. To get the test run for reports you switch verbose mode on. As such it then gives you everything! However, I'm already looking to add an option to CPAN::YACSmoke to filter the 'Extracted' messages.It's also a know problem that a distribution with pre-requisites can produce the test run for all in all reports within that test run.
Another problem, which is the problem you've unknowingly spotted, is that CPANPLUS doesn't currently catch the 'perl Makefile.PL/Build.PL' failures. However, I'm compiling the patch and tests for that at the moment.
I realise that CPAN::YACSmoke is probably going to get the most stick for test reports that aren't correct, but bear in mind there is a huge framework that actually does the work.
