All the Perl that's Practical to Extract and Report

More Custom Testing Goodies

Ovid on 2008-01-11T15:13:53

Our latest custom Test::More looks something like this:

package Our::Test::More;

use strict;
use warnings;

use Hook::LexWrap;


# XXX don't use 'base' as it can override our signal handlers
use Test::Builder::Module;
our ( @ISA, @EXPORT );

use Test::More;
use Test::Differences;
use Test::Exception;

BEGIN {
    @ISA = qw(Test::Builder::Module);
    @EXPORT = (
        @Test::More::EXPORT,
        @Test::Differences::EXPORT,
        @Test::Exception::EXPORT,
        'explain',
    );

    if ( Test::Differences->VERSION <= 0.47 ) {

        # XXX There's a bug in Test::Differences 0.47 which attempts to render
        # an AoH in a cleaner 'table' format.
        # http://rt.cpan.org/Public/Bug/Display.html?id=29732
        no warnings 'redefine';
        *Test::Differences::_isnt_HASH_of_scalars = sub {
            return 1 if ref ne "HASH";
            return scalar grep ref, values %$_;
        };
    }
}

sub import {
    for my $i (0 .. $#_) {
        if ('fail' eq $_[$i]) {
            splice @_, $i, 1;
            wrap 'Test::Builder::ok', post => sub {
                if (![ $_[0]->summary ]->[-1]) {
                    die "Test failed.  Halting";
                }
            };
            last;
        }
    }

    # 'magic' goto to avoid updating the callstack
    goto &Test::Builder::Module::import;
}

sub explain {
    return unless $ENV{TEST_VERBOSE};
    Test::More::diag(@_);
}

1;

This does several things:

  1. Imports all test functions from Test::More, Test::Differences and Test::Exception.
  2. Adds an explain() function. It's like diag(), but produces no output unless in verbose mode.
  3. Adding 'fail' to the import list will cause the current test program to die on the first failure.
  4. Fixes an annoying bug in Test::Differences.

The three test modules included are ones that we use constantly, so it makes sense to include them.

The explain() function was needed because several developers really want diag() output for times when you want to read the test narrative more clearly. However, that output is useless when running the full test suite in non-verbose mode. It requires the latest version of Test::Harness in subversion because the $ENV{TEST_VERBOSE} functionality was accidentally left out (that would be my fault, I think).

The 'fail' argument to the import list is very handy when you have several hundred tests scrolling by and you hate scrolling back up through reams of junk to find the failure. Use it like this:

use Our::Test::More 'fail', 'no_plan';

BEGIN {
    use_ok 'Some::Module', ':all' or die;
}

ok defined &foobar, 'foobar() is in the house!';
throws_ok { foobar( 5 ) }
  'My::Exception',
  "... but he don't like no arguments";
eq_or_diff foobar(), foobar(),
  "... and he repeats himself a lot"'

ok defined &barfoo, 'barfoo() has somehow shown up';
foreach my $thing (barfoo()) {
    explain("Testing $thing");
    do_some_test($thing);
}
...

As you can see, you get more testing functions, the "explain()" stuff might spew out loads of junk, but you won't see it when you run the full test suite, and it will halt as soon as the first test fails.

explain() should be in Test::More core

markjugg on 2008-01-11T17:33:45

Ovid,

Thanks for posting this. In 2005 I provided a patch for an explain() equivalent to Test::More, but it was never added:

http://rt.cpan.org/Public/Bug/Display.html?id=14764

It's such a sensible, helpful feature, I don't understand why it remains a feature which people continue to have to rediscover and patch themselves.

Re:explain() should be in Test::More core

Ovid on 2008-01-11T18:01:42

It's getting to the point where I'm thinking about just releasing a custom Test::More::Extra (or something) to handle this stuff. The problem is figuring out which test modules are the most common. Just choosing mine seems a bit cheeky.

Dude, where's my diagnostics?

schwern on 2008-01-12T01:19:02

I've tried to make this approach work. Here's the problem in a nutshell:

$ perl -wle 'use OurMore "no_plan";  is 23, 42'
not ok 1
#   Failed test at -e line 1.
#          got: '23'
#     expected: '42'
1..1
# Looks like you failed 1 test of 1.

$ perl -wle 'use OurMore "fail", "no_plan";  is 23, 42'
not ok 1
#   Failed test at /usr/local/perl/5.8.8/lib/Test/More.pm line 329.
Test failed.  Halting at OurMore.pm line 44.
1..1

Re:Dude, where's my diagnostics?

Ovid on 2008-01-12T09:35:19

Thanks. I'll look into this more closely.

Test::Differences vs Test::Deep

colink on 2008-01-12T04:31:11

I'm curious. Why do you prefer Test::Differences vs Test::Deep? Is it better diagnostics for failing tests, speed, personal preference, or something else altogether?

Re:Test::Differences vs Test::Deep

Ovid on 2008-01-12T09:52:44

I just added Test::Deep. Consider the code:

use Our::Test::More 'no_plan';
my ( $foo, $bar ) = ( [qw/1 2 3 4 5/], [qw/1 2 trois 4 cinq/] );
eq_or_diff $foo, $bar;
cmp_deeply $foo, $bar;

Consider the output:

#   Failed test at test.t line 10.
# +----+-----+----------+
# | Elt|Got  |Expected  |
# +----+-----+----------+
# |   0|1    |1         |
# |   1|2    |2         |
# *   2|3    |trois     *
# |   3|4    |4         |
# *   4|5    |cinq      *
# +----+-----+----------+
not ok 2

#   Failed test at test.t line 11.
# Compared $data->[2]
#    got : '3'
# expect : 'trois'

With Test::Differences, I can see and potentially fix all of the problems at once. I've used Test::Deep for the bag tests, but that's about it.

Also, last night I decided to figure out which Test modules were the most popular. Out of almost 300 I found used in CPAN modules, here are the top ten:

  Test::More              44473
  Test::Exception          1380
  Test::Simple              731
  Test::Base                316
  Test::Builder::Tester     193
  Test::NoWarnings          174
  Test::Differences         150
  Test::MockObject          139
  Test::Deep                127
  Test::Warn                118

Test::Deep certainly warrants more attention here.