Cheese speleology

masak on 2009-09-05T14:14:21

Here's an idea I had the other day.

We're in the process of upgrading how proto installs Perl 6 projects. mberends++ has taken the initiative by creating a branch and a TODO. Today, the same mberends++ also landed a commit in Rakudo such that all our PERL6LIB worries just vanish. It's looking good.

One thing I would like to fall out from this refactor is a way to run all the tests in all Perl 6 projects. Think of it as a kind of community-level introspection — proto's Ecosystem.pm was my early attempt at this. And we can easily set up cron jobs and HTML serializers for a nightly run of these things on feather. This would have two benefits, I think:

• We get an ecosystem equivalent of the spectest suite, in the sense of "these things should all work, and if it doesn't, a human should check it out". Such testing helped us before — apps helping us find holes in the Rakudo cheese — but by not running all the tests regularly, we're doing ourselves a disservice, because it might be days between the introduction and discovery of a Rakudobug. The sooner these things are flagged down, the better for everyone.
• We could make the community more positively aware of project tests, and make the community celebrate them. Which Perl 6 project out there has the most tests right now? I don't know. I have a guess, but I would like to know for sure. Which project passes the highest amount of tests? Which projects pass all of their tests (barring :todo and :skip tests)? Which project has had all its tests passing for the longest time? Which project has had all its tests passing for the longest time while also adding more tests lately? Which project has the best test coverage? And so on. It's not a contest, but if some authors choose to think of it that way, what we'll end up with is better-tested code.

Testing all of the Perl 6 ecosystem sure is simpler than testing all of CPAN. But if it can help improve Rakudo and the Perl 6 community, I think it can be very worthwhile.