I've got a somewhat automated regression suite generator for a webapp I'm working on. The process starts out with a script that crawls a portion of a site, and produces a list of URLs to examine. That list of URLs is passed to a script that fetches each URL in turn, munges it, and builds a test script based on that munged data. At that point, I have a bunch of test scripts that prove whether or not the (munged) HTML varies in any request.
(The process is slightly more complex in reality; there's some Makefile trickery involved as well.)
The whole process is parameterized by location and cookie file. That is, a baseline server runs on one webserver, while the system-under-test runs side-by-side on another. Once the baseline/sandbox specific prefixes are factored out, the URLs should match across servers, simplifying testing. By varying the cookie file, I can mimic multiple users with a minimum of fuss.
Here is some of the Makefile trickery I mentioned before. At some point, it all boils down to two targets, make check and make test:
SANDBOX=http://localhost:8001
BASELINE=http://localhost:8002
PROVE=$(HOME)/bin/prove
test: $(TESTS)
-HOST=$(SANDBOX) $(PROVE) $(TESTS)
check: $(TESTS)
-HOST=$(BASELINE) $(PROVE) $(TESTS)
The make check target exists to run the tests against the baseline server to see if any test failures are because of a bug in the new code, or a bug in the test. This is important, because any page that adds today's date will cause a failure on a ~daily basis, and you need to isolate those problems from the real ones.
As you can see, this works great, so long as you're willing to let prove work its way through hundreds of tests. If you want to test in parallel, you're out of luck.
Until you start hacking up your Makefile, that is. ;-)
Here are some GNU Make recipes to convert those serialized tests into parallel ones:
.PHONY: *.t-parallel-check
.PHONY: *.t-parallel-test
%.t-parallel-check: %.t
-@HOST=$(BASELINE) $(PROVE) $< 2>&1 | perl -e 'print <>' ; echo
%.t-parallel-test: %.t
-@HOST=$(SANDBOX) $(PROVE) $< | perl -e 'print <>'; echo
parallel-check: $(subst .t,.t-parallel-check,$(TESTS))
parallel-test: $(subst .t,.t-parallel-test,$(TESTS))
That works reasonably well, except all of the outputs get intermingled and unintelligible. Until they are piped through this oneliner:
perl -e 'print <>'Isn't it wonderful when less than few characters of Perl save the day? ;-)