All the Perl that's Practical to Extract and Report

Quieter Test Output

chromatic on 2005-01-12T02:43:39

I never really understood the point of prove; I write my tests as normal Perl programs, launchable from the command line. perl t/testname.t does most of what I want to do.

With more than 24 tests in a file, though, I often promised myself to write or find someday a variant test harness that ignored successes and reported only failures or an all clear message.

I looked at the code of prove today to find where to add it, but again, it doesn't seem to do anything I really need and there was no way to add it. I moved on to Test::Harness::Straps. I don't know if anyone's ever actually used this module (and it's admittedly a bit of a mess inside, though the long-promised Test::Builder refactoring may make it easier to write), but it took 67 lines of code to do what I needed (and a little more) in a well-factored way.

I call it qtest:

#!/usr/bin/perl

use strict;
use warnings;

use Test::Harness::Straps;

my $strap = Test::Harness::Straps->new();

for my $file (@ARGV)
{
	next unless -f $file;

	my %results = $strap->analyze_file( $file );

	if ($results{passing})
	{
		report( sprintf('All (%d) tests passed in %s',  $results{seen}, $file));
	}
	elsif ($results{skip_all})
	{
		report( sprintf('All (%d) tests skipped in %s', $results{seen}, $file));
	}
	else
	{
		report( find_failures( $file, \%results ) );
	}
}

sub report
{
	my $message = shift;
	print "$message\n";
}

sub find_failures
{
	my ($file, $results) = @_;
	my $report           = create_header($file, @{$results}{qw( max seen ok )});
	my $count            = 0;

	for my $test ( @{ $results->{details} } )
	{
		$count++;
		next if $test->{ok};
		$report .= create_test_result( $count, @{ $test }{qw( name reason ) } );
	}

	return $report;
}

sub create_header
{
	my ($file, $expected, $seen, $passed) = @_;
	my $failed                            = $seen - $passed;
	return sprintf "File '%s'\nExpected %d / Seen %d / Okay %d / Failed %d\n",
		@_, $failed;
}

sub create_test_result
{
	my ($number, $name, $reason)   = @_;
	$name                          =~ s/^-\s*//;
	$reason                      ||= '';
	$reason                        = " ($reason)" if $reason;
	return sprintf "\tTest #%d: %s%s\n", $number, $name, $reason;
}

In practice, I may tweak the formatting somewhat. It might be nice to report skips and surprisingly-passing TODO tests, too. I also might want a flag to end after the first failure, to solve the problem of cascading failures rolling off of the screen. Still, this was a lot simpler than I thought and already makes my life easier.

The Point of Prove

Ovid on 2005-01-12T02:57:02

prove was based on a program that I wrote a long time ago to manage my tests better. I sent that off to Andy and it got warped into prove. Unfortunately, the one thing that I miss that I had code which would allow me to run sets of tests, based upon patterns on the command line. Thus, when working on stuff that touched "products", I might run a particular test program, or I might run all test programs with "prod" in the name. When I was done, I'd run the full test suite. It was very handy. Now I don't find it quite as useful, though I still use it.

What I think is curious, though, is that I made reference to this back in 2003 and you had a response about wanting to filter out successful tests and possibly using Test::Harness::Straps to do it ...

Re:The Point of Prove

chromatic on 2005-01-12T05:09:20

I don't understand why you need prove to run sets of tests, unless it's for the nice summary at the end, which does seem like an advantage.

(I just realized that that may be what you like as I started to type "my shell expands patterns just fine, so I don't need another program to do it for me". I think that's the conceptual part I didn't understand.)

Re:The Point of Prove

phillup on 2005-01-12T06:03:11

I'm not sure about his reasoning... but I use it a lot.

My entire test suite takes about 30 seconds (dual 2.8 GHz Xeon system)... and growing (since I only have about 10 percent coverage on three years of programming). That is long enough to get me out of the zone if I have to do it too much.

So... I run prove, the setup routines, and the particular tests I'm focusing on. The setup routines take about 5 seconds and the individual tests run in about 2 seconds. Which, is short enough not to take me off task (too often ;-)).

Re:The Point of Prove

petdance on 2005-01-26T18:09:16

Really, prove was more based on smoke that I wrote at work.

Why should you use prove? Here are some reasons!

http://petdance.com/perl/use-prove-lt.pdf

Test::Verbose

Matts on 2005-01-12T13:15:27

You really want to look at Test::Verbose - it does exactly what prove should have been (via the "tv" command line) and has -q, -qq, and -qqq options to achieve exactly what you desire here. Moreover, it logs its full output to a file so you can scan that if you want more detail of what has happened.

It's a shame that Test::Verbose is greatly overlooked. I find it absolutely indispensable.

Re:Test::Verbose

chromatic on 2005-01-12T19:08:53

You're right, that is pretty close to what I want. I've just sent Barrie a patch to add Module::Build support.

Thanks for the pointer.

I don't understand your dislike of "prove"

merlyn on 2005-01-12T13:33:06

I like to be able to run the tests for my modules without waiting to copy them into the staging area, which is pointless for a pure-Perl module. So I can simply cd into the t directory, then repeatedly say "prove -I../lib *.t", and alter that to a specific test or add -v. It's like "make test", but a dozen times more convenient.

It's also a lot easier than typing the equivalent

perl -MTest::Harness -e 'runtests(@ARGV)' *.t
which I also had used in some of my early docs on testing. Thank goodness we hide that complexity now, replacing that all with "prove *.t".

I really appreciate that "prove" came along.

Re:I don't understand your dislike of "prove"

pudge on 2005-01-19T04:00:10

Yeah, for me, it's just like using an alias for a slightly longer alternative. Aliases are good.