The new Test::Builder is giving me fits. It incorporates some lovely changes, but there's a problem. The old Test::Builder required some very nasty internals hacks by Test::Aggregate to work. Unfortunately, Schwern's changed the internals for the first time in years. This is his right and I shouldn't have relied on them, but there wasn't much else I could do unless I bundled my own Test::More distribution and that would have introduced an entirely different set of problems (just bundling Test::Builder would fail as an updated Test::More might try to call new Test::Builder features as it does now).
The Test::Aggregate I'm working on now faces the problem of multiple calls to done_testing(). Further, there are issues with calls to any tests in BEGIN/CHECK/INIT/END blocks. I think I'm going to go ahead and bite a huge bullet (this was marked as ALPHA code, after all). I'm going to deprecate tests in those blocks and eventually call BAILOUT on them.
The problem with code in BEGIN/CHECK/INIT/END blocks is that there are timing problems associating them and associating the right test with the right test program for diagnostics. Now, however, you can do this:
use Test::Aggregate; Test::Aggregate->new(\%options)->run; done_testing(217);
And that will quite nicely provide an ending plan check for you. However, if any of your aggregated tests provide any tests in END blocks, those are automatically run after your call to done_testing().
As further incentive to remove BEGIN/CHECK/INIT/END blocks, we're moving to nested TAP. It's critical that tests be nested correctly and there's no way to guarantee this with tests run in those blocks. As a result, I have the following really, really nasty hack in Test::Builder::ok():
{
my $ok;
BEGIN { $ok = \&Test::Builder::ok }
my %FORBIDDEN = map { $_ => 1 } qw/BEGIN CHECK INIT END/;
sub Test::Builder::ok {
__check_test_count(@_);
my $level = 1;
while (1) {
my ($caller) = ( ( ( caller($level) )[3] || '' ) =~ /::([[:word:]]+)\z/ );
last unless $caller;
if ( $FORBIDDEN{$caller} ) {
my ( $self, $test, $name ) = @_;
$test = $test ? "Yes" : "No";
my ( $filename, $line ) = (caller($level))[1,2];
$self->diag(<<" END");
Aggregated tests must not be run in BEGIN, CHECK, INIT or END blocks.
File: $filename
Line: $line
Name: $name
Pass: $test
END
$self->BAILOUT('Illegal aggregated test call');
}
$level++;
}
local $Test::Builder::Level = $Test::Builder::Level + 1;
$ok->(@_);
}
}
I know that people aren't going to be terribly thrilled with this, but it solves many, many problems.
The main issue that people are going to have is this:
BEGIN { use_ok 'My::Module' or die }
I recommend that you stop using this meme. Simply put a 'use My::Module' at the top of your code and put all of your 'use_ok' tests into one 'load.t' test which is not aggregated.
I'll try and release the next Test::Aggregate with a strong deprecation warning for tests in those blocks instead of the BAILOUT I have in my test code, but I'm sad I have to do this. Still, the long-term benefits you'll gain from nested TAP in Test::Aggregate should be worth it.
I hope.
It should be noted that at the last Perl-QA hackathon, neither Schwern nor I found a clear way of avoiding messing with Test::Builder internals. Doing this cleanly waits on Test::Builder 2.0.
Suggestions welcome :)