All the Perl that's Practical to Extract and Report

Testing Installed Modules: Opening a Can of Worms

DAxelrod on 2006-09-16T18:53:26

Ovid recently asked what should be a simple question: How should we install tests along with modules?

Of course, this is not a simple question at all and breaks down into several distinct problems. The first is actually "How do we test an installed module?"

kwilliams responded with the brilliant idea of a Module::Build retest action, that would look for modules in @INC instead of blib.

Unfortunately, this doesn't completely solve our first problem.

Why? Because way too many tests are written with the assumption that they will be running from within an unpacked module distribution. Which means they require context outside of the installed .pm files.

Off the top of my head, the kinds of tests that have this problem include

  • Tests that pull in specific files from the distribution
    • Tests that pull in test modules not in blib
    • Tests for scripts installed with the module
    • Tests that pull in data files (which may end up being installed)
  • Tests that depend on the whole distro structure
    • Test::Signature
    • Test::Distribution
    • Test::Pod
    • Test::Pod::Coverage
    • tests that rely on the MANIFEST
    • tests that scan files for spelling errors, minimum versions, or prereqs

You could argue that any of these that are abstracted into Test::Whatever modules have enough of a layer of indirection to account for however you test installed modules. But I think that misses the point. A list that large is not evidence of special cases. It's evidence that it's important for tests to have more than the .pm alone.

One solution that sucks could be a way of marking which tests assume they're being run in a distribution and which ones don't care. That would be horribly tedious.

Maybe this is evidence that we need a metadata database to go with our installed modules. We already have a PACKLIST that sorta kinda serves as an installed module database. The problem is, now we need an API for tests to query that database.

Maybe we're going about this wrong. Tests need a distribution. Modules work in a distribution and out of a distribution. It's common to develop modules inside a distribution.

Why not install an entire module distribution?

Have a lib-like directory which is searched for entire distributions rather than just .pm files. When you use a module, it will actually be pulled from its distribution.

Running modules from within distributions? Doesn't that sound like PAR?

Works for me

Aristotle on 2006-09-16T19:45:29

Because way too many tests are written with the assumption that they will be running from within an unpacked module distribution.

Which is exactly the situation that the retest feature in Module::Build assumes, so I don’t quite see why this is a problem.

Re:Works for me

DAxelrod on 2006-09-16T20:11:05

The problem is that installed modules have more to them than the .pm files, and those are basically the only parts of your installation you're going to be testing when you use retest.

Tests for scripts will be tests on the scripts in the distribution, not the ones installed. Likewise for all the other things that a distro can have.

Simple

acme on 2006-09-18T12:42:05

It was my original request. I want to make sure that things still work even though I've upgraded a module low down in the dependency tree. I don't want to solve this in every case. I don't want the moon on a stick. I just want something simple, like copying t/* somewhere during install. I was afraid of posting about it because I knew people would go overboard and get philosophical. And look what happened when Ovid mentioned it...

Re:Simple

DAxelrod on 2006-09-18T13:15:22

I'm interested in the future of the module ecosystem.

This was an opportunity for me to get philisophical, but it wasn't your fault. :)