For the second time I've recieved a patch for a module of mine changing:
use Test::More qw(no_plan);
To:
use Test::More tests => 218;
From my perspective this just makes adding tests harder. Every time I add a test to the bottom of the file I have to remember to scroll all the way t o the top and update the magic number.
What am I missing?
-sam
If you have loops, skips, or possible early exits, it's helpful to know how many tests you expect to run, in case you run more or fewer.
It's also possible to avoid loops and skips as well as most early exits, so if you put a dependency on Test::Harness 2.x or greater, you're probably okay with no_plan.
Re:Planning X Tests
samtregar on 2003-11-13T19:45:47
I can't remeber the last time I put a call toexit()in a module or a test... What's the problem with loops and skips? I've used those with no_plan and not had trouble.-sam
Re:Planning X Tests
Ovid on 2003-11-13T20:41:01
If you're testing a module that has an unexpected exit() in it, you'll be grateful when your test count shows that you didn't run all of the tests.
As for loops, if you have tests in loops, if the loop runs more or fewer times, the test count will catch that and again you get a failure report. When this happens, I usually find it's a bug in the test, but it's nice to know.
As for skip blocks, those can be tricky. Consider this:
SKIP: {
skip "No internet connection", 23, unless $net_connection;
# some tests
}Now what happens if you've been running that repeatedly without a net connection but only have 21 tests that you've skipped? The first time you run this with a net connection, even if all tests pass, you still have the test program failing due to a poor test count. Again, this is usually due to a bad test count, but if it's not, you'll be grateful.
Re:Planning X Tests
samtregar on 2003-11-13T21:17:06
Unconvinced. It's easy to add a test for how many times a loop ran (i.e.is($loop_count, 10);). As for SKIP, I usually just pass skip an arbitrary value since I don't care how many tests skipped, just that some did.How many real bugs has using a static plan caught for you? How many false alarms?
-sam
Re:Planning X Tests
schwern on 2003-11-14T14:30:13
If you're testing a module that has an unexpected exit() in it, you'll be grateful when your test count shows that you didn't run all of the tests.People often bring this up. I always point out the simple solution: override CORE::exit in Test::More to trap exit() calls. Oddly enough nobody's sent in a patch for that yet. I've gotten lots of schemes to change the way plans work, though.
:( Loops, as pointed out, are often based on lists who's length are not fixed. Even if they are of a fixed width its often irritating to try and do the math to figure out how many tests are going to be run. And you have to do all that math up front!
Counts on skip blocks are inconsequential with no_plan. Since there's no fixed number of tests there's no need to get the number of tests skipped right! Its actually *easier* to use skips with no_plan. You can just put in 1. I intended skip() to allow you to leave off the number of tests skipped if no_plan is in effect, but that appears to be busted.
About the only real drawback of using no_plan is you need a relatively recent Test::Harness (circa 5.8.0).
Re:Planning X Tests
chromatic on 2003-11-14T00:07:46
The point is not to test only things you expect to fail. If you're looping over something not under control of the test, say, the number of files reported found by File::Find, the number of elements exported by a module, it's really handy to know if you have more or less than you expect.
There are often other ways to test this, but most of them involve keeping some sort of magic number in your test anyway.
You may never be bitten by this. I've ran into trouble and think it's worth my time to use a test plan. That's fine; it's your code and your call.
Re:Planning X Tests
samtregar on 2003-11-14T00:14:01
Some magic numbers are more magic than others. I don't mind updating this line when I add a new method:is($method_count, 10);But hiding that test up in the plan statement, along with every other counted loop... That seems like deep magic to me. And what a pain to debug when it fails too! How do you know which part of your test file ran too long or too short?
-sam
On my todo list for Test::Harness I have the ability to optionally require a plan. Right now, I don't have a way to enforce that every test in my project has a proper plan.
Sure, it means you have to manually change the test count, but it also guarantees that you won't inadvertantly change the number of tests without knowing it.
Re:It's a test
samtregar on 2003-11-13T19:53:27
There must be a better way to make sure the number of tests remains constant as long as the test file's mtime doesn't change... The only hard part is finding a place to store the data between runs. Hmmm...-sam
Re:It's a test
chromatic on 2003-11-14T00:01:47
Ugh,
mtimeas a change notification. That'smake's second worst legacy. (The first, obviously, is syntactically significant whitespace.:) Re:It's a test
samtregar on 2003-11-14T00:06:27
Find, MD5 then. Whatever. Just don't make me update a magic number everytime I add some tests!-sam
Re:It's a test
schwern on 2003-11-14T14:38:49
As I say in one of my talks, having the correct number of tests is itself a test.As I say, you're a nutter plan nazi.
;P For those who don't know. I think the plan is an obsolesent idea that should fade away. Andy thinks plans are really important should be made even more sophisticated. We each control one half of the Perl testing equation (Test::Builder/More on one side, Test::Harness on the other) which is a Healthy Thing from a checks and balances PoV.
But you're still nuts.
:) Re:It's a test
petdance on 2003-11-14T15:03:24
But you're still nuts.:) Yeah, well, your mama writes COBOL!
:-) My big concern about not using plans is that if something doesn't happen right and the wrong number of tests is run, it's just ignored. chromatic's mention of loops is a good example, and exiting early is another.
But you know all this already. This is just for the benefit of those watching at this point.
:-)
