With the increasing usage of the xt directory, and with a more robust understanding of the appropriate mechanics behind release tests, I feel the time has come to build support for xt into Module::Install by default.
But before I do so, I'd like to lay out the rules I'll be applying to the implementation for public review and commentary.
-------------------------------------------------------------------
1. The xt directory, if it exists, will automatically be added to the list of tests to be executed if the Makefile.PL is running with either..
1.1. Author mode enabled
1.2. With RELEASE_TESTING true.
1.3. With AUTOMATED_TESTING true.
2. Recursion on the xt directory will be triggered with the same tests_recursive command that triggers recursion on the t directory.
3. The behaviour of the individual test scripts in xt is assumed to be...
3.1. Always run when RELEASE_TESTING is enabled.
3.2. Run during AUTOMATED_TESTING only if the support modules are installed and at a new enough version.
3.3. Run during author mode only if the support modules are installed and at a new enough version. If the test script is unable to M:I's author mode status, the test should be skipped to prevent accidental executing during end user install.
I believe this combination of rules will give us the following scenarios.
Module Authors
Having checked out a module the xt tests will be visible during make test and will run if they can, but if not they will skip without error while telling the author what extra dependencies they need to install.
When building a release tarball, the xt tests will forcefully always run and fail if they are missing dependencies. This will ensure releases are always subject to full Quality Assurance.
CPAN Testers
The xt tests will be included in the list of tests to run, but will only be run when the system already has the appropriate dependencies in place to prevent excessive dependency recursion (which might fail and prevent any testing of the package at all).
End User
During installation, the end user will never see the xt tests displayed on their screen, even if they would otherwise always skip.
-------------------------------------------------------------------
If you have any comments, questions, or suggestions, please reply now.
Assuming no obvious issues arise, my plan is to implement this in the next M:I release some time in the next month.
I generally think of CPAN Testers as trying to reflect the "end user experience" -- and some people just happen to report in bulk using automated smokers. I would rather just have xt tests skipped if AUTOMATED_TESTING is true, since an xt FAIL is meaningless to an end user.
-- David
Re:Just skip if AUTOMATED_TESTING?
Aristotle on 2010-01-15T12:21:57
Agree.
Is there a reason for assuming all xt tests are "release tests"?
I'm currently looking at using "xt" for all "extra tests", with "xt/release" for release tests, and "xt/slow" for slow tests.
I'd want to run the release tests for both automated and release testing (as you've described), but I'd only ever want to run the slow tests if I'd explicitly asked for them.
Now I could put them in yet another top-level tests directory, but that's getting messy, I'd rather keep all the "extra tests" in the "extra tests" directory.
I could also add checks for AUTOMATED_TESTING and RELEASE_TESTING and so on to each test in the xt/slow directory, but that's adding still more boilerplate to each test file, with all the problems that cut-n-paste boilerplate code has.
Of course my current distros all use Module::Build so this doesn't directly effect me, but if a convention is established, it will effect me eventually. (And getting release-tests working "right" in xt under Module::Build is causing me enough grief that I'd like to switch if something offered me a better alternative.)
While it might not be the default behaviour, it would be good if there was the option to say "only these dirs of xt are release-tests, not the lot".
