All the Perl that's Practical to Extract and Report

Massive TAPx::Parser regression tests

Ovid on 2006-09-14T20:42:24

After Schwern pointed me to the sample tests for Test::Harness, I've started adding them to TAPx::Parser. I got to 613 tests without a single bug.

Then I added a regression tests for programs with no test output. Just that simple requirement added 22 tests. That's when I finally hit my first bug. My parser died a horrible death. That bug is now fixed, but that uncovered another bug, also fixed. I don't know how soon I can upload, though, as I still have 21 more sample test programs to add. I think that's the point where I can bump the parser up to 0.30.

This pushes me up to 1096 tests altogether.

Question: does anyone ever use the exit status of a test program? I haven't provided this, but it strikes me that some folks might use it.

Test::Harness

Theory on 2006-09-14T21:24:00

Test::Harness uses the exit status of a test script.

—Theory

Re:Test::Harness

Ovid on 2006-09-14T21:48:54

Yes, but it's apparently to represent the number of failed tests and that caps out at 255, so you can't even report the number of failed tests correctly unless it's less than that magic number. VMS doesn't even get that right, so the exit status doesn't seem terribly useful. On the Perl-QA list, chromatic suggested that I just ignore it. However, if someone is actually using it, I'll need to know how and why.

Plus, this is part of a project to replace Test::Harness and its limitations. This appears to be another one, but I'm not sure.

Oh, and I'm up to 717 regression tests now, bringing me to 1177 total :)

Re:Test::Harness

jjore on 2006-09-14T21:52:26

I don't see why you don't validate that the exit status is zero. Your programs never segfault or die without ever printing useful information?

Or maybe the thing runs all the way through and segfaults in an END block. Maybe the official part of the test is already over but you can notice that the test still died a bad death.

Re:Test::Harness

chromatic on 2006-09-14T22:43:38

Yes, that's what I meant, though that's certainly not what I wrote to perl-qa. An exit status of zero or not-zero is useful information. Anything else seems somewhat useless.

Re:Test::Harness

Dom2 on 2006-09-15T05:21:17

Absolutely. It's not like we're dealing with sendmail, where it does need to deal with the different varieties of code in sysexits.h.

-Dom

CPAN/CPANPLUS

kane on 2006-09-15T09:08:54

The exit status is crucial information for automated installers, to know whether the test (program) succeeded or failed.

See this bug in Test::Harness for what can go wrong.

The precise exit number is not quite relevant, but zero vs non-zero is.

Re:CPAN/CPANPLUS

Ovid on 2006-09-15T09:39:09

Yeah, I've been convinced. I'll be adding support for this.