An Author's Guide to CPAN Testing

barbie on 2006-02-01T12:47:39

There's been a recent discussion on the perl-qa list, that has made me wonder the best way to advertise features in CPANPLUS and CPAN::YACSmoke, that CPAN authors really ought to know about. There are 2 in particular.

CPAN testing seems to be something that some authors, particularly new ones, are unaware of. CPAN testing provides the capability to test CPAN distributions on many different versions of Perl and OS platforms, and report the results. So when these authors suddenly get FAIL or UNKNOWN reports, it's often a surprise. Every now and then it starts up a discussion on how to work around issues they have, without knowing there are already features in place to help them.

Nearly all CPAN Testers run automated smoke testing environments. A large proportion use CPAN::YACsmoke and CPANPLUS. Being a significant contributor to the test reporting part of both, I've implemented/patched the two to help with some common issues that some CPAN authors have to deal with. The following are just 2 features in particular that I think should get better exposure.

Automated Testing

In some cases distributions being tested need to be interactive. They need human input because either there is no default, or the default isn't working. A example of this might be knowing where a 3rd party application/library is installed. Running under a smoke testing environment, any prompted question will result in a simulated return key press. This can then cause an endless loop, which is exceedingly annoying for the CPAN tester when they have to CTL-C the app or kill processes, which on occasion has even crashed the machine ( but then I was testing on Windows ;) ). A better way is for the author to detect that their distribution is being CPAN tested, and bail out or take some action that would allow configuration to occur after installation, so that at least some basic tests can be run.

When CPAN::YACSmoke starts up, the environment variable AUTOMATED_TESTING is set, which can then be checked either in Makefile.PL or the testing scripts as:

  if($ENV{AUTOMATED_TESTING}) {
      die "Unable to determine setup, cannot continue";
  }


or

  SKIP: {
      skip "Unable to determine setup, cannot test", $n
	  if($ENV{AUTOMATED_TESTING});

# .... }


OS Unsupported

For distributions named Win32::, Linux:: or any other OS specific name, it is not unsurprising if they fail to work on another OS. CPANPLUS already knows this and reports NA to CPAN testers. However, some modules can support a variety of OSs, or might not support a specific OS, and therefore their name may not bear any OS specific name. In this instance it is still possible for authors to check a list of known m/(un)?supported/ OSs and bail out if testing is inappropriate.

CPANPLUS now checks to see whether the strings "No support for OS" or "OS unsupported" exist in the message buffer. Again this means that the author can include something like the following (taken from File::FDpasser):

  %ostofile=( 
         linux=>'bsd44.o',    bsdos=>'bsd44.o', 
         openbsd=>'bsd44.o',  freebsd=>'bsd44.o', 
         netbsd=>'bsd44.o',   solaris=>'svr4.o', 
         dec_osf=>'bsd44.o',  irix=>'bsd43.o', 
         hpux=>'bsd43.o',     aix=>'bsd44.o', 
         darwin=>'bsd44.o'); 
 
  if (!defined($ostofile{$^O})) { 
      die "No support for os: $^O\n". 
          "Edit makefile.pl and send mail to amh\@mbl.is\n"; 
  } 

This then results in an NA report being sent to CPAN Testers.

Conclusion

I've mentioned both of these, and others, in talks I presented at YAPC::NA and YAPC::Europe last year, but obviously they still need to get aired. I finally completed the TestersGuide for CPAN::YACSmoke before Christmas, but it's not been released yet, as Rob and I are still reviewing and writing tests for the next version. Although it is available via CVS on SourceForge.

However, perhaps there also needs to be an AuthorsGuide. A guide that provides hints, tips and links for solutions to commonly experienced problems. There are several guides to creating modules, but none that I've seen that deal with the after effects. Questions generally get posted to perl-qa, module-authors and perlmonks, but perhaps a resource is needed that provides a starting point. As a first step I'll look at writing an AuthorsGuide within CPAN::YACSmoke, but I'm not sure whether this will be its final resting place, as it could do with reaching a wider audience.

If anyone has thoughts, ideas or suggestions of what to include, email me (via my CPAN address) and I'll investigate further.