Oslo Perl QA Hackathon, Day 2

Ovid on 2008-04-06T22:33:31

The groups of developers here broke into different groups to focus on their areas of interest. Naturally, I'm in the TAP group and we're working on the next version specification. The short result: standards are hard. Very hard. Extremely hard. I have a certain sympathy for standards committees that I previously lacked.

Two of use were tasked with taking our notes and turning them into a combined standards document and test suite. The test suite is a YAML specification mapping inputs to expected results. Thus, anyone who wants to write a TAP producer can use this test suite to determine how compliant they are.

We've broken things down into what you normally see, 'TAP Core' (just the plan and ok/not ok lines), and the 'TAP envelope' (stuff which wraps the TAP core) and 'TAP Complete' ('TAP Core' and 'TAP envelope' plus YAML diagnostics). Amongst other things, you'll be able to add tags to tests, have properly structured diagnostic output, nested TAP and possibly understand the 'context' under which a test is run. The latter allows an author to say 'in "install" context, POD test failures won't halt installation'.

For the most part, few people will see any changes. If there are test failures, they'll see structured diagnostics. The other "behind the scenes" stuff is needed to support many testing needs that we have, but which we've never been able to properly support before. We're trying to add most of this to both Test::Builder and Test::Harness at the same time, but it will only be out in developer releases.

Two of us who have been tasked with writing the specs and tests have done an excellent job of showing edge cases in our decisions. It's frustrating to stop all of our programming to work out the difference between "file" and "url" keys in TAP diagnostics, but these are important issues. We will have the potential to hurt a lot of people if we get this wrong, but fortunately we've had many months, if not years, to debate the needs on the Perl-QA list.