I've decided that tests are one of the most useful thing I've learnt about Perl in the last 12 months. In the past I've seen tests whizz past like a blur when installing anything from CPAN, and they are noticeable by their absence in PPM files, but now I grok them.
They are a great help to the module developer. I know I'm preaching to the converted (well I hope I am), but I'm constantly amazed by the defects that turn up during testing that I didn't know were there. You know the story you add a new feature, add a few tests, and then think you'd better add a new test to an old method, and then you discover a new unhandled exception, so you fix your module, and add yet more tests.
My current task has moved XML::RSS::Tools up to 94 tests. The latest tests require a live Internet connection, to test the HTTP clients. It's a bit clunky as the user has to select a reachable URI, but that allows the test to work behind firewalls, or over SSH tunnels where things are odd! I've made it so that the user can select to abort the test during the build Makefile and during the testing stage, but it's messy requiring user input....
As for the absense of tests in PPM files, I heard a rumor that ActiveState only allows modules to be turned into PPMs if they pass all tests. If that's true, that means some of the largest, most useful modules may never see the light of day because one test out of two thousand might fail and small, broken modules that only have a use_ok() test might get included. I wonder if anyone can (or will) confirm this?
The latest tests require a live Internet connection, to test the HTTP clients. It's a bit clunky as the user has to select a reachable URI [snip] I've made it so that the user can select to abort the test during the build Makefile and during the testing stage, but it's messy requiring user input....
Have you considered SKIP tests? You can have a list of URIs that can probably be reached and if you can't reach any, select a block of tests that can be skipped. Nice and automatic that way.
SKIP: {Re:Testing Good!
ajt on 2003-05-17T20:17:36
I've often wondered about ActiveState's PPM policy. Most pure Perl modules don't need PPMing, CPAN works perfectly well, and I don't think it's that hard to use on Windows. The modules that require a compiler, something Windows doesn't usually have, are more troublesome, and it's nice of them to provide the files all nice and packaged up. BUT, it's a lot of work for them to keep up with all of CPAN, and it's clear that there is a lot of popular stuff they just haven't done, as it's only available elsewhere. As to their testing policy, I can see holes in the policy you propose, so one can only hope that they are more logical.
I don't know enough about the various testing options, I think there is a gap in the market for a book- or at least a pocket guide. I've read the POD, but it feels like I'm going round in circles, I understand the usefulness of testing, but I don't quite grok the methods fully. I've found this, An Introduction to Testing, which I'll have to read again for a start.
My live test solution, currently is to offer a user selection before the Makefile is built which adds a bunch of extra tests. The when the tests are running, the user is asked to input a URI (it's nice to let the user choose at some point, in case they are cut of from the Internet but have a HTTP server within their intranet). If they don't bother to choose something, then I SKIP the test, otherwise I try to do a HTTP GET, to make sure all works. Actually adding this test brought to light a defect in one of the methods, so it's been useful even if the test gets changed before the module gets updated on CPAN.
Re:Testing Good!
jand on 2003-05-17T22:46:10
Yes, I can (and will) confirm that only CPAN packages that successfully pass their regression tests are included in the ActiveState PPM repositories.The processes that build the repository are still being tweaked, but ultimately, a module will have to build and test. Otherwise it is broken on that platform.
It is the package authors responsability to make sure that tests work in an environment that for example don't have a test database set up or where STDIN is not available to Makefile.PL.
We are still working on making the build logs available from a website so that package authors (and users) can check for themselves why their module didn't build.
Re:Testing Good!
Ovid on 2003-05-17T23:41:09
So ActiveState does realize that this means that the overall quality of modules made available will probably be lower? While I will agree that it's the author's responsibility to ensure that there modules pass all tests, it can be very difficult to test complicated modules on platforms that the authors don't have access to. For example, the page you direct us to lists the following platforms:
- windows
- linux
- solaris
- hpux-pa-risc
- hpux-ia64
- hpux-ia64-lp64
Will you create a PPM for a module for those platforms on which it passes, or is this an all or nothing deal?
I also saw that my module Acme::CodePolice passed, but that's a joke module that's useless. However, most of the Inline modules fail, but they are incredibly useful. That's disappointing.
Re:Testing Good!
jand on 2003-05-18T01:09:12
If the tests fail on a platform, it means that either the module or the tests don't work on that platform. I totally disagree that it is likely that the module will work and it is just the tests that are failing. If tests are known to fail on some platform, then they should automatically skip, and not just fail (if the failure is acceptable). Known failing tests are worse than not having tests at all, because they are bogus tests.
For the 5.6 repositories, it is unfortunately necessary that all platforms are at the same version number, because they all share the same PPD file. This has been fixed for the 5.8 repositories. Each platform will have the latest version of the module that passed all tests on that platform.
Please note that missing modules are not necessarily due to test failures. It is also possible that the modules need special (manual) build preparation (like extra libraries), or that a prerequisite module is missing. Or a non-core prerequisite is not listed in the Makefile.PL.
As I wrote before, the build processes still need some improvements. But there will always be modules that don't build automatically.
It's up to you to write probes that make sense for the group of tests you're skipping.SKIP: {
skip "skipping online tests... not connected", 17
if not_connected();
... 17 tests here ...
}
Of course I want the foo option, dammit, it's the default!
Normally, most of these modules provide some way to avoid interacting, but every single one is different! It's a most unsatisfactory state of affairs.
You might also wish to review some of acme's old journals for more details on the subject.
-Dom
Re:(interactive makefiles)--
ajt on 2003-05-18T17:03:22
I couldn't agree more that returing from lunch to discover that something stoped 5 minues after you left it is a big pain. Then again, it's also annoying when a tests fails just because you're behind a firewall. I'd love to make it work like magic, testing if it can, skipping if it can't!
I've had some good ideas suggested, even some via Perlmonks from this jorunal thread, so things are looking up.
Re:(interactive makefiles)--
jand on 2003-05-18T19:35:45
Look at Makefile.PL from Net-DNS! It shows you how to detect if a direct connection to the internet is available.It also shows how you use the prompt() function of MakeMaker. That one will automatically choose the default value if STDIN is not a tty, or if the PERL_MM_USE_DEFAULT environment variable has been set. This makes it much more likely that your module will build (and test) in an automated environment.
Oh, and a hint from looking at another PPM build failure: Make sure that you always accept the default value you give to parse(). Saying that the default is not valid in some situation and prompting again just turns this into an infinite loop when STDIN isn't available.
:) PS: Could you give me a pointer to the Perlmonks discussion?
Re:(interactive makefiles)--
ajt on 2003-05-18T20:23:56
Jand,
Actually it wasn't a discussion per se, rather Corion suggested that you would include HTTP::Demon in the test suite as WWW::Mechanise::Shell does. According to the credits in the file, Merlyn made the original suggestion.
The Perlmonks node that started this whole thing was: LWP::Parallel vs. HTTP::GHTTP vs. IO::Socket but there have been several emails, and P2P messages that are missing from my use.Perl journal.
