A while ago, there was a debate on Perl-QA as to whether or not one must have test programs run in separate processes. What a few people ignored is:
There are pros and cons to each approach.
Run all tests in separate processes and you can know that your tests are focusing on exactly what you want to test. Run them in the same process and you can catch strange state issues. Sometimes those are in the tests, but sometimes they are in the code. Both approaches offer benefits and drawbacks.
Today's navel-gazing exercise involves being able to have a test stop when a failure occurs or optionally bailing out of the test suite when a failure occurs. For example, right now I have one test program whose 4,039 tests take over six minutes to run (yeah, it sucks, but hey, it happens). Since I'm using Test::Most, I just add a 'die' to the import list and the test program halts at the first failure. This failure happens about 38 seconds into the run and then I can see what causes the failure and rerun it with a subset of its data and recreate the error faster. This makes debugging much easier than being hunched over the keyboard waiting to hit ctrl-C if I see a failure. Or maybe I should just pipe everything through less and manually page through those results, eh? (Because this is programming and we want to increase manual involvement?)
Or I could just tell the damned tests to stop when a failure is encountered. Simple, eh?
The debate, however, has ungracefully broken down into all sorts of positions, the two most prominent which seem to be:
You know, while I advocated the second position, I know that each approach has strengths or weaknesses. Unfortunately, somehow the "harness should stop the tests" camp has become entrenched in their position and are arguing strongly for it, even going so far as to hint that their's is the "one true way." While I think their position has a lot of merit, I'm distressed at the dogmatism surrounding the topic.
Meanwhile, Test::Most does what it says on the tin.
Last I checked, Test::Builder was pretty much at the center of testing, and as far as I'm concerned, pretty much all functionality possible should be there.
Just to get really wild, I don't know a whole lot about how all the various parts of testing interact, so I'm looking at it with a clean (and probably ignorant slate), and I wonder if there should even be a harness.....
Re:Test::Builder
Aristotle on 2008-01-15T18:02:19
Well, the problem with that is that TAP is the Test Anything Protocol, which is to say you don’t know if the tests on the other end of the harness use Test::Builder, or if they are written in Perl in the first place, or even if they run on something that has a microprocessor at all – for all you know, you’re parsing the self-test output of an embedded device received over a serial port. So the “harness should be responsible” camp has a point.
On the other hand (which is the argument that Ovid made on perl-qa and which I would have made myself if I could bring myself to care)… well, who cares? Just because not everything uses Test::Builder doesn’t mean we can’t put a feature in Test::Builder that other TAP emitters may not have, nor does it mean that we can’t put supplementary features in the harness.
It's about separation of concerns. Test::Builder produces TAP. TAP::Parser parses TAP. The harness is the mediator between the two.
