Kill Off Test::Builder?

Ovid on 2009-11-11T11:39:24

We've been waiting for Test::Builder 2.0 for quite some time but many of us need some of the promised benefits now. I've been giving some thought to what can be done about this.

One of the things we want, for example, is structured diagnostics. Another thing we'd like is for the existing diagnostics to be tied to the test function they're associated with. Unfortunately, here's a typical test function:

sub is_valid_json ($;$) {
    my ( $input, $test_name ) = @_;
    croak "usage: is_valid_json(input,test_name)"
      unless defined $input;
    eval { $JSON->decode($input) };
    my $test = __PACKAGE__->builder;
    if ( my $error = $@ ) {
        $test->ok( 0, $test_name );
        $test->diag("Input was not valid JSON:\n\n\t$error");
        return;
    }
    else {
        $test->ok( 1, $test_name );
        return 1;
    }
}

You'll note that the diag and ok are separate functions. This hurt me with Test::Most because I wanted the "die or bail on fail" to hit as soon as there's test failure, but that won't work because diagnostics are output after the test function. Thus, we have to wait until the next test is run to die or bail. This probably doesn't affect many people, but for those it does, it's a real pain.

I'm trying to think of a way around this and my first thought is simply sidestep the entire Test::Builder problem. Phase 1 of the attack:

package Test::Runner;

our $VERSION = '0.01';

BEGIN {
    require Test::More;

    $INC{$_} = "Replaced by Test::Runner $VERSION" foreach qw(
        lib/Test/Builder.pm
        lib/Test/Builder/Module.pm
        lib/Test/More.pm
    );
}

And in another package:

package Test::Runner::Builder;

sub Test::Builder::new  { ... }
sub Test::Builder::ok   { ... }
sub Test::Builder::diag { ... }
... and so on

And in another package:

package Test::Runner::More;

use Test::Runner qw(
  ok is isn't isnt like unlike cmp_deeply and so on
);

The idea is to create a compatible interface to the old, familiar functions, but the Test::Builder functions would record their responses but not output anything until after the test function is over. Thus, anyone who wanted to have proper diagnostics from their test program would use Test::Runner instead of Test::Builder and they would have to register every test function. This would give us the needed information to control what can and cannot be used with tests.

The reason we'd have to override much of the current Test::Builder infrastructure is because older test modules aren't going to use Test::Runner. We'd have to check what was in @EXPORT (or wrap &import in some cases?) and assume that anything which gets exported into someone's namespace from a test module is a test function.

It's an ambitious plan and fraught with error, but in the absence of Test::Builder 2.0, I think it's a worthwhile experiment.