Tomorrow I'll be arriving in Oslo as part of the Oslo QA Hackathon. I'll be presenting my "Turbo Charged Test Suites" talk and generally looking forward to meeting people heavily involved in Perl QA. I'm particularly interested in the Test::Builder 2.0 discussion and TAP improvements.
I want the new Test::Builder because I keep hitting the limitations of the current one. Its plan handling is awful, it's difficult to alter the output and diagnostics are a mess. Few people hit these problems, but when they do, they're hard to overcome.
The TAP improvements are possibly more important. People are constantly talking about how to generate test results as XML. When I was at a Google Test Automation conference, one topic which came up over and over again is how they needed a common XML test result serialization format. Somehow, the assumption that it must be XML was made over and over again. TAP, however, isn't an answer because we simply don't capture the needed information. It is, however, stable and proven.
If we feel TAP is a good tool for others, we need to make sure that it supports how they do things. For example, here's a jUnit test:
public void testEquals() {
Money m12CHF= new Money(12, "CHF");
Money m14CHF= new Money(14, "CHF");
Assert.assertTrue(!m12CHF.equals(null));
Assert.assertEquals(m12CHF, m12CHF);
Assert.assertEquals(m12CHF, new Money(12, "CHF")); // (1)
Assert.assertTrue(!m12CHF.equals(m14CHF));
}
That's one test, not four. One suggestion for handling that in TAP:
ok 1 - testEquals
ok 1 - True(!m12CHF.equals(null))
ok 2 - Equals(m12CHF, m12CHF)
ok 3 - Equals(m12CHF, new Money(12, "CHF"))
ok 4 - True(!m12CHF.equals(m14CHF)
With nested TAP, the top level test would fail if any "subtests" fail. Also, it's completely backwards compatible with current TAP because the regex for a TAP line is basically /\A(?:not )?ok/, thus ensuring that indented TAP doesn't get recognized.
Nested TAP has its own problems, but Schwern's been here for a couple of days and we've discussed some of the issues involved. I really hope we get some serious traction on this.
Re:regarding JUnit XML
jmason on 2008-04-07T14:37:17
damn, guess who forgot to preview.;)
here's the missing text:
=head1 SYNOPSIS
tap-to-junit-xml "test suite name" [ outputprefix ] < tap_output.log
=head1 DESCRIPTION
Parse test suite output in TAP (Test Anything Protocol, http://testanything.org/ ) format, and produce XML output in a similar format to that produced by the junit ant task. This is useful for consumption by continuous-integration systems like Hudson ( https://hudson.dev.java.net/ ).
"When I was at a Google Test Automation conference, one topic which came up over and over again is how they needed a common XML test result serialization format".
Do you remember any conclusions from that discussion, i.e., whether any XML test result formats were proposed or demonstrated?
John.
--