Last week, while at the same Test Automation Conference Acme attended, I had an inspiration. Given that Yahoo! is using my TAPx::Parser, Torsten Shoenfeld has written a GTK front end for it (I need to include his name directly in the POD), Shlomi Fish is integrating TAPx::Parser with Test::Run, people are starting to send me patches and there's a general increase in interest for it, it's time for me to get this thing out of alpha.
How better to do that than to run TAPx::Parser against all of CPAN? Since I use CPAN::Mini, I have most of the CPAN on my hard-drive. As it turns out, testing everything isn't practical, but I can test most of it. My initial plan is to discard all Apache and Win32 specific modules, anything without a Build.PL or Makefile.PL file or a t/ directory. I might be tossing out too much, but I still have thousands of packages I can test on my iBook.
I get through about 200 of them before my code can no longer fork (something about "resource not available") and I can't even run ls in a directory until I kill my code. This has turned out to be very difficult, but I've learned a lot about my code and have already found some strange edge cases it can't handle.
In other news, Acme wants to install your tests with your code so you can test that your installation works. What a brilliant idea!
Re:Testing installation?
chromatic on 2006-09-11T17:56:02
I want you to install your (
Test::Class-based) tests as normal modules so I can reuse it when I reuse your modules.Re:Testing installation?
jjore on 2006-09-11T18:12:41
That is just the mental shove I needed to see why I'd possibly ever care about Test::Class. Thanks!Re:Testing installation?
Ovid on 2006-09-11T18:37:54
There are plenty of other reasons to use Test::Class. Do you have thirty t/*.t programs? You've just loaded the perl runtime at least 30 times (probably more). You've also probably reloaded the tested modules many times and possibly duplicated a lot of test code. With Test::Class, you only load the Perl interpreter once and the associated modules once. Large test suites can run much faster and are easier to refactor.
And inheriting tests to ensure that derived classes don't break their parents makes me giggle with happiness.
I should note that I blogged about it here, with some more information.
Re:Throwing away less testable modules
Ovid on 2006-09-12T15:41:13
I had considered that, but a number of test.pl modules don't output TAP (largely because they're older code). As a result, my parser can't reliably parse their output. My goal here, in reality, is not to test all of the CPAN, but to have a massive stress test of TAPx::Parser. I still have several thousand test suites left which I can use and I've found a few bugs I need to address.
That being said, I doubt all tests in t/*.t are based on Perl's regular testing tools, but I have to start somewhere.
Re:Throwing away less testable modules
grinder on 2006-09-13T11:28:02
Yes, I had considered that you had considered that.
In which case, it would be handy to know what test.pl scripts aren't TAP-compliant. I know early versions of Compress::Bzip2 (before it was taken over by the current maintainer) featured a test script that produced nothing useful.
It would be nice to have an exhaustive list. Then we could go about fixing them, if the distributions in question are useful enough to be worth the attention.
