Over the last year, I've seen a disturbing trend on the part of some of Perl testing thought-leaders (ugh... what's a better word for this... Testerati?) to demonise the testing plan.
I thought I'd take a moment to come to the defense of a plan (and why I don't like done_testing() except in a very specific situation).
When I wrote my first CPAN module, and as a result discovered Perl testing, the thing that impressed me most of all was not the syntax, or the fact that tests were just programs (both of which I like).
It was the testing plan that I found to be a stroke of brilliance.
Even though it can be a little annoying to maintain the number (although updating this number sounds like a good feature for an editor to implement) the plan catches two major problems simultaneously.
1. It detects ending, aborting, dieing and crashing tests, even if the crash is instantaneous with no evidence of it happening.
2. It detects running too many tests, too few tests, bad skip blocks, and other soft failures, preventing the need to write tons of explicit list length tests.
It also prevents the need to say when you are done, reducing the size of your test code.
For example, look at the following two code blocks, both of which are equivalent.
This is the new no plan done_testing way.
use Test::More;
And this is the original way.
my @list = list();
is( scalar(@list), 2, 'Found 2 members' );
foreach ( @list ) {
ok( $_, 'List member ok' );
}
done_testing();
use Test::More tests => 2;
foreach ( list() ) {
ok( $_, 'List member ok' );
}
The difference is stark. In the original, I know I'm doing 2 tests so I don't need to test for the size of the list from list(). If list returns 3 things, or 1 thing, the test will fail.
There is one clear use case for done_testing (and btw, who decided that done_testing was a great name for a function, what ELSE are you going to be done with? Surely just done() is good enough) and that is when the number of elements returned by list() is unknowable.
Even in this case, I still prefer explicitly telling Test::More that I have NFI how many tests will run. That at least gives it some certainty that I have actually started testing.
The new paradigm of not using a plan is far far messier, for no obvious benefit I can see and open up new areas with bugs can creep in far too easily.
It seems to me that the example you give is ultimately a poor test. You’re relying upon an implicit feature of the testing interface to test for explicit behaviors of your code. As a person new to your project how I am supposed to know that list() is only ever supposed to return two elements? Sure the tests fail but how do I know that the tests were correct in the first place?
By making an explicit test for the explicit behavior, I can communicate to someone else that I’m expecting list() to only ever return two elements, and that doing differently is actually changing known behavior.
Say what you will bout done_testing, you may like it or you may not (personally I hate having to update my test count, but I was willing to use no_plan). Relying upon implicit behavior of your environment to provide you with explicit behavior of the code your writing is a bad practice.
Re:Perhaps you've chosen a bad example...
Alias on 2009-11-19T04:53:13
I agree with your assessment of no_plan, at least when you used it you were explicitly saying there was no plan...
Re:Perhaps you've chosen a bad example...
frew on 2009-11-19T06:09:09
Agreed. There are times when you need a plan for sure (my current project for example, where I test for lots of different types of exceptions) but in general I find it thought cruft.
Also note that no_plan doesn't (I think) detect die'ing halfway through.Re:Perhaps you've chosen a bad example...
perigrin on 2009-11-19T07:17:13
Which is why I've happily moved todone_testingfor future code.Re:Perhaps you've chosen a bad example...
Ovid on 2009-11-19T08:08:29
no_plan won't detect early termination, but Test::Harness will note the non-zero exit status on a die.
Re:Perhaps you've chosen a bad example...
btilly on 2009-11-20T00:36:15
It is more common than I'd like to see END and DESTROY blocks do something that clears the exit status of a Perl program on its way out the door.
In fact it is common enough that I'd prefer not to rely on it to detect abnormal ending of a test suite. Particularly since many test suites like to do cleanup when they are done.
The Aegis software management tool has been around for a long time, but I strongly suspect that part of the reason it never became very popular is because it enforced a particular belief on users. For example, your code is always in one of multiple stages and for the "being developed" state, you cannot check in unless you have new tests and your code must pass those tests. There are numerous other states and numerous other preconditions which developers much satisfy before they can move the code to the next state and integrate it.
Or I can check out my code, change it, check it in.
Ultimately, we all know which we'd rather develop on, even though many of us might find it tempting for force developers to have tests.
By insisting upon a numeric plan:
Me? I'm laid back about this issue now. I know most developers aren't going to sweat things too much and many detest and mistrust evangelism. That's OK. Find other ways to bring them into the fold.
(And for all of my arguing about an iteration count being off and that's why I need a plan, I've discovered that this happens so seldom to me that I was arguing a use case I rarely had. If later it turns out to be a bug, I'll add a test for it. Your mileage may vary.)
Re:Gotta have a plan ...
Aristotle on 2009-11-19T09:46:30
Are we talking about organised religion or programming? I find the language in your last couple of comments… creepy.
Re:Gotta have a plan ...
Ovid on 2009-11-19T10:09:28
The religious allusions were deliberate. The dogmatism of those who insist that we "must have a plan" put people off in the same way that dogmatists for anything will put off certain parts of the population. I don't care for dogmatism because it's often a fancy word for "we think we understand this, so we don't need critical thought". Since dogmatism is often accompanied by a near religious fervor, I thought it was OK to continue with the metaphor.
Sorry if I was a little too zealous in that. Obviously it didn't come across as intended
:) Re:Gotta have a plan …
Aristotle on 2009-11-19T10:29:39
You lost me when you started talking about bringing people into the fold and such. I want to teach trade-offs and how to weigh them, not convert people to a doctrine – any doctrine.
Re:Gotta have a plan …
Ovid on 2009-11-19T10:44:48
My apologies. I meant that strictly as tongue-in-cheek. I should hope that that those who know me would appreciate the irony of me using religious allusions. I'm not serious, dude
:) Re:Gotta have a plan …
Aristotle on 2009-11-19T11:10:12
I know, which was part of why I was a little taken aback. All good then.
:-) Re:Gotta have a plan ...
Alias on 2009-11-19T14:21:21
We've never insisted on a plan, you just had to explicitly say that you didn't have one.
What I'm seeing now is the opposite, you (and others) seem to be actively encouraging people to NOT use plans. Or at least, that is the impression I get.
People have never HAD to use DBI placeholders either, but the default documentation comprehensively refers to it.
Re:Gotta have a plan ...
Ovid on 2009-11-19T14:43:57
Not using DBI placeholders is far more serious than not using a plan. It's can leave you wide open to serious security holes. Lack of a plan, however, while risky, is far less risky.
When I'm moving, my friends and I are carrying stuff into my house and I leave the door unlocked. I lock the door when I'm done. Similarly, when I write tests, I set them as no_plan and add the plan when I'm done. What I want to do is minimize the accounting when writing tests and when the developer is done, they lock the door, er, add the plan.
Encouraging developers to write tests is what's important here. As they progress in testing, they'll find out what works for them and what doesn't. If they find something cumbersome (as has always been the major complaint of plans), either they'll ignore it (common), or find a different way of accomplishing it (done_testing()). I don't want my personal dogma to discourage developers from progressing.
Again, I note that it's easy when writing jUnit tests to have similar conceptual errors, but jUnit clearly scratches an itch for Java developers and is better than no tests at all.
I like to use the argument done_testing accepts that lets you say how many tests you expected to have run at that stage.
Its like specifying a plan, except you can do so pro grammatically after the fact, which eliminates the need for tedious manual counting.
All you need to know is how many tests occur within a given block of code and increment your counter respectively.
use Test::More;
my $t;
$t+=2;
foreach ( list() ) {
ok( $_, 'List member ok' );
}
$t+=3; # list called the second time emits more items.
foreach ( list() ) {
ok( $_, 'List member ok' );
}
done_testing( $t ); # $t is 5
There's surely a better way to get the same effect.
Re:done_testing( $arg )?
kentnl on 2009-11-19T09:43:34
/me-- # tiny bloody text box ==> me splits a word causing bad grammars.Re:done_testing( $arg )?
Ovid on 2009-11-19T13:40:21
There's surely a better way to get the same effect.
Switch to Test::Class. I've an extensive article on its use and best practices. When nested TAP is finally stable, Adrian Howard has a new version available which utilizes it. It's much cleaner.
Re:done_testing( $arg )?
Alias on 2009-11-22T17:58:06
I've found that that Test::Class testing code is significantly harder to maintain that the "bunch of Perl scripts" of the regular way.
That's just me though...
Re:done_testing( $arg )?
mauzo on 2009-11-19T14:07:12
Yes, this seems the way to go to me, especially since I have been writing my tests like
use Test::More;
my $tests;
BEGIN { $tests += 2 }
for (list()) {
ok $_, "List member ok";
}
BEGIN { plan tests => $tests }for quite a while now. AFAIC
done_testing($tests)simply removes the need for the uglyBEGINblocks.The other thing I quite like about
done_testingis that is removes the need forSKIP:blocks. Rewriting this (which is generally how I would write a complicated skip block)SKIP: {
my $skip;
$reason and skip "reason", $skip;
BEGIN { $skip += 3 }
ok...;
ok...;
ok...;
BEGIN { $tests += $skip }
}as simply
{
$reason and last;
$tests += 3;
ok...;
ok...;
ok...;
}seems like an improvement to me. If I think the reason is particularly important I can put in a
skip "reason", 1just to get it recorded in the TAP somewhere.
Re:New fancy stuff breaking my CPAN :)
Alias on 2009-11-19T23:26:45
That is just the regular kind of stupid, forgetting that because you are using new syntax, you need new versions in your deps.
Re:New fancy stuff breaking my CPAN :)
Alias on 2009-11-19T23:27:39
It should be fairly easy to write a detector for too, just look for done_testing in modules and compare it to their META.yml.
Re:New fancy stuff breaking my CPAN :)
Aristotle on 2009-11-21T03:47:09
This is where it comes in extra handy that
done_testinghas a stupidly (or so it would seem) long name.:-)
Even though it can be a little annoying to maintain the number (although updating this number sounds like a good feature for an editor to implement)
Funny, I just added a feature to do that for emacs, it's in the latest release of my perlnow.el.
If you run a *.t file from inside of emacs using perlnow.el, you can then do a "perlnow-revise-test-plan" which changes the plan to match the number of tests you actually ran.
