One of the things which we faced at the Oslo Perl QA Hackathon is that there are precious few tests for TAP itself. TAP consumers will have a hard time verifying that they are doing the right thing if we don't make it easy to test. As a result, along with the spec, YAML descriptive tests have been started. Two gentleman (Oystein and Andreas? Last names? Spelling?) started on the draft TAP specification and Adrian Howard was writing the Perl code which tested their YAML. A sample test looks like this:
---
name: parse error, no plan
TAP: |
TAP version 13
TAP done
Result:
Error:
Type: Parser
Desc: No plan found before TAP done.
...
With this, anyone writing a TAP consumer in any language can just write code to parse the YAML and verify that their consumer is correct.
However, there's another problem we stumbled on: nested TAP. We'd like something like this:
TAP version 13
1..5
ok 1 some name
begin 2 - fribble
TAP version 13
1..3
ok 1
ok 2
not ok 3
TAP done
not ok 2 - sucess or failure of above block
ok 3 - a separate test
... more tests
That turns out to be completely backwards compatible since lines with leading whitespace are ignored. However, that was shot down because this means that a TAP producer also needs a TAP consumer to parse the nested TAP to determine if it succeeded or not. Producing TAP is easy. Parsing it is much harder than people would suspect and embedding a TAP parser into a TAP consumer raises the bar to what we felt was an unacceptable level. Thus, this non-backwards-compatible suggestion was adopted:
TAP version 13
1..5
ok 1 some name
begin 2 - fribble
TAP version 13
1..3
ok 1
ok 2
not ok 3
TAP done
ok 3 - a separate test
... more tests
Note that we switch from 'ok 1' to 'ok 3'. 'not ok 2' is implied by the failure of the nested TAP. This is not backwards compatible, but it was felt that this was OK because the consumer would specifically have to ask for nested TAP.
That sucks. It sucks hard. Just because the consumer didn't ask for nested TAP doesn't mean that the producer won't produce it. Producing TAP that is not backwards compatible is very disturbing to me if we can figure out a way around it. However, since it was agreed that interpreting the nested TAP required a parser, we would have to leave off the 'not ok 2' line.
Last night I figured a way out. Interpreting the nested TAP does not require a parser, it only requires that we check three things:
That's all we need. Things like 'tests out of sequence' are parse failures, not test failures. and the parser is required to catch that. Thus, I think we can get backwards-compatible nested TAP. To check this, I am now writing this:
#!/usr/bin/env perl
use strict;
use warnings;
use Test::Most 'no_plan';
use Test::TAP;
is_failing_tap 'ok 1', 'Invalid tap if we do not have a plan';
is_passing_tap <<'END', '... but it should be valid if we do have a plan';
1..1
ok 1 - some message
END
is_passing_tap <<'END', '... even if it is a trailing plan';
ok 1 - some message
1..1
END
is_failing_tap <<'END', '... but not if it is an embedded plan';
ok 1 - some message
1..2
ok 2
END
is_passing_tap <<'END', '... but it is ok if the plan is in nested TAP';
ok 1 - some message
begin 2 - fribble with nests
TAP version 16
1..2
ok 1
ok 2
TAP done
ok 2 - refers to preceding block
1..2
END
Backwards compatibility matters.