One of the modules that we rely on at work is PDF::API2, and I'm deeply grateful for the fact it exists. But we're also highly sensitive to it, so if it ever goes rotten we have some hard thinking to do. But lets ignore that part for now.
If you were considering writing a PDF application, and happened to be searching around for something to use, and you stumbled across the PDF::API2 module on CPAN you would see this.
Now, looking at that page, I can see that it's failed a third of all CPAN Testers reports, and it hasn't had a release of note in a while. To me, that looks like a fairly rotten not very well maintained module that I should avoid.
So lets look at it's CPAN Testers report page.
And if you want to see what it looks like with my settings, which only shows production releases of the module that have occurred on production versions of Perl (that is, in the real world) then you see something like this.
And what you'd see, if you only care about the real world, is that in the real world PDF::API2 has no less than 100% PASS rate (ignoring the 2 UNKNOWNs).
Not a single damned FAIL.
And yet looking at the graph on the CPAN testers page, you could be forgiven for thinking that the module is decending into rot.
The real situation is that the module works perfectly, unless you try to build it one of two dozen patched versions of Perl that nobody other than the P5P team will ever see, and which are probably already fixed.
It's all well and good that we run tests of patched versions and developer versions, and that we collect results from patched versions and developer versions.
But the poison is spreading and as a result we're staining the reputation of reputable modules in public forums.
EVERY feed of information coming out of the CPAN Testers database needs to be limited to production releases of modules on production versions of Perl BY DEFAULT, unless someone has a specific need to request otherwise.
This bullshit needs to end.
I like that people test my modules on any version of Perl, but I'd like to see the Testers figure out how to have two sets of results: stable version results and everything else. The stable results are for the public. The everything else gets sent to developers and isn't in the testers database.
I'd like to see things such as CPAN Search just use the stable results. That might be as easy as slicing the data for public consumption without making any changes to the people doing the testing.
Re:Different data views please
srezic on 2008-10-07T21:44:41
That's why the CPAN Testers Matrix (e.g. the PDF::API2 results) exists. You can easily see if failures are OS or perl dependent and feel free to ignore it.Re:Different data views please
brian_d_foy on 2008-10-08T04:06:15
The CPAN Testers matrix isn't something see on CPAN Search though, which is the point. This isn't the normal techie problem of thinking that the data is there and if people would just look at it everything would be fine. People see the summary of PASS and FAIL on a module page first, and then maybe (maybe) the matrix.
I want that first impression to be better tuned to what their experience will be, which is using the module with a stable version of Perl.
Re:Different data views please
srezic on 2008-10-08T06:08:55
If you're refering to the number of PASSes and FAILs on the summary page --- these numbers are nearly meaningless, just like the cpanrating stars or the number of open RT tickets.Re:Different data views please
Alias on 2008-10-08T14:32:36
Those numbers are quite meaningful to large set of people unable to reliable judge the code themselves. They use results like 100% PASS as a guide of it they can trust the code to work in some random place.
Re:Different data views please
srezic on 2008-10-08T21:44:12
People really choose a CPAN distribution based on a single number?Re:Different data views please
chromatic on 2008-10-08T22:28:35
Pretend you're not an expert Perl contributor for ten minutes. Imagine that someone told you to find an XML module for Perl from the CPAN. You shuffle through several dozen pages of search results and read the distribution pages. Some of them have a big red "CPAN TESTERS SAYS THIS CODE GETS A FAIL!!!" notice.
How many non-experts are going to dig into matrices of platform combinations and bleadperl patchlevels to realize that, oh, you can't rearrange the struct members between a PV and a PVMG and retain source code XS compatibility, but that's okay, because someone on p5p just proposed a patch to fix things?
I don't normally guess at hypothetical situations like that, but in this case, I don't have to guess. The number is zero.
I suppose an alternate approach is to educate some 6.2 billion potential CPAN users that there's little useful meaning in the numbers posted on the CPAN distribution pages, but that seems like more work than removing meaningless data and publishing only meaningful data.
Re:Different data views please
srezic on 2008-10-09T06:35:31
For a question like "which XML module is the best" search.cpan.org is not very well suited. If I was a beginner, I would rather ask Google and hope to find some comparison or recommendation page like
http://www.perlfoundation.org/perl5/index.cgi?pbp_module_recommendation_commenta ry http://www.xml.com/pub/a/2002/08/21/perl-xml.html
or
http://perl-xml.sourceforge.net/Re:Different data views please
chromatic on 2008-10-09T17:11:41
For a question like "which XML module is the best" search.cpan.org is not very well suited.
I know that and you know that and almost everyone reading this knows that, but you had to take off your beginner hat to know that.
Re:Different data views please
chromatic on 2008-10-08T21:28:15
Why publish meaningless data?
Re:Different data views please
srezic on 2008-10-08T21:43:06
It's just a teaser. If it's 100% PASS, then I don't need to look at the results. If not, then I can look at the matrix and see if there are patterns. Or dive deeper into the results.
Yes I get it. You don't like it. You've bitched about it all over the place. I get the message. Instead trying to beat the hell out of me, perhaps you can help fix it.
Do I really deserve this public thrashing?
Stop and think for a moment. Would you be so ready to fix problems within your code when someone is persistently telling you how crap you are for not fixing their specific problem?
Re:Please STOP Bitching
Alias on 2008-10-07T22:23:56
You wouldn't believe the amount of crap I get about some of the stuff Strawberry does that people want changed.
And yes, I'm trying to fix it.
Also, I deeply grateful for the stuff you in particular have done to fix the problem on the places you control. You've probably done more to fix this problem than anyone.
Your argument may or may not have merit, but PDF::API2 is a terrible example to make your point.
Have you looked in the t/ directory of PDF::API2? Here, I'll save you the time:
use Test;
BEGIN { plan tests => 1 }
use PDF::API2; ok(1);
With no unit tests, a PASS is meaningless. Only a FAIL has any meaning at all.
That said, I think PDF::API2 is good code and Fredo has been remarkably responsive in my experience, but my own not-really-competing CAM::PDF library has unit tests with code coverage over 50% -- not great but infinitely better than PDF::API2.
Re:NO TESTS!
Alias on 2008-10-08T14:35:11
True that, but the sheer percentage of FAILs all of which are on non-prod environments provides an excellent visual example.
Re:NO TESTS!
Alias on 2008-10-08T14:41:04
Also, alas, we're doing PDF creation, and we've implemented pretty much an entire graphic design environment, with box geometry model and a ton of other crazy stuff.
PDF::API2 is the only thing we've found with enough features and control over the generation of the PDF content.
The old system this replaced used to use PDFLib, but it's not good enough any longer.
We'll probably end up contributing patches and improvements to PDF::API2, rather than switching.
I think you could spend your time better.
Re:5.10.x
Alias on 2008-10-08T14:34:27
Who's to say that it will REMAIN broken for the final prod version though...
Re:5.10.x
srezic on 2008-10-08T21:45:24
It will remain broken. Until somebody notices and fixes it.
