Do you need to upgrade your Module::Install?

Alias on 2006-06-09T13:15:52

In the great Module::Install vs Module::Build "paradigm war", one of the more notable scenarios is when a major problem is discovered.

In the Module::Install model, all authors need to do incremental releases of the modules affected by the problem, but users need to do nothing.

In the Module::Build model, all users affected by the problem need to upgrade their version of Module::Build, but authors need to do nothing.

At present, both of these don't solve this problem correctly.

Module::Install doesn't have a method for ensuring authors upgrade, and Module::Build doesn't have a method for ensuring users upgrade.

To start fixing this for Module::Install, I've been announcing Module::Install updates in this journal, with notes on if and when authors need to do upgrades or incremental releases.

With Module::Install going through a moderately sane and non-flakey period at the moment Steffen Mueller has taken the next step, and has written a script to scan all of CPAN and work out which distributions have old/bad versions of Module::Install, and need to be upgraded.

From Steffen:

There are three common toolkits for installing CPAN distributions on a computer. The venerable ExtUtils::MakeMaker, the newer Module::Build and Module::Install. Module::Install is different from the aforementioned alternatives in that it is included in the distribution and hence does not require an installation on the client machine.

This has the benefit of usually being simpler to set up than a distribution using Module::Build (in case Module::Build has not been installed yet). If the distributions using Module::Install do not use a reasonably current version of Module::Install, however, they might be hard or impossible to install on systems that are not supported by the old included version of Module::Install.

An incompatibility between ActivePerl 5.8.8 build 817 for Win32 and older versions of Module::Install has caused me some headache recently: Distributions using old versions would not install at all and running the included "Makefile.PL" would result in a cryptic error message.

Therefore, it is important to keep the included versions of Module::Install reasonably up-to-date. I have generated a list of CPAN distributions that include Module::Install and listed them with the corresponding versions. If you are a CPAN author and have any such distributions, please consider upgrading. Thank you.

You can find the list at

[UPDATE] 0.61 now considered "Okay"...

tsee on 2006-06-09T13:57:18

In Your Build.PL

chromatic on 2006-06-09T19:02:01

... Module::Build doesn't have a method for ensuring users upgrade.

use Module::Build '0.xx';

Alternately, stick the dependency in build_requires.

Re:In Your Build.PL

Alias on 2006-06-10T07:24:21

use Module::Build '0.xx';

As far as I'm aware, that doesn't ensure that Module::Build is upgraded, it just crashes the installer with an obscure error message that will be buried somewhere in the middle of 3000 lines of CPAN.pm output.

If you are doing a one-off install and running Build.PL directly, and you understand error messages, then maybe in that small case it's good enough. but of course almost nobody does that.

As for the build_requires dependency, the Build.PL needs to be run in order to confirm that, and if you have a dependency on a new version of Mobule::Build to run it, then you fail the circular dependency.

Of course, I'd be happy to be proven wrong (with full test script).

Re:In Your Build.PL

chromatic on 2006-06-10T15:53:30

I don't have a distribution with a custom Module::Build subclass where the build_requires trick won't work, but if the use line doesn't, how about printing the same error message? M::B and CPAN.pm communicate somehow, so I see no reason why following that protocol won't make this scheme work.

Re:In Your Build.PL

Alias on 2006-06-10T18:17:02

The continuing problem is that error messages are of no real use.

There's a fairly good chance that the user will never see it, being buried deep in a larger installation.

Any solution which involves wetware (the user having to think) is no solution at all.

"M::B and CPAN.pm communicate somehow, so I see no reason why following that protocol won't make this scheme work."

As I said, Module::Build hasn't solved this problem yet.

The solution, as I've mentioned before, requires CPAN clients to have the capability to force-upgrade certain toolchain modules to the most current version when they detect a change.

Otherwise we are back to painful reality of circular dependencies. If Module::Build has a major problem, and someone tries to install a module that uses it, then installation fails. Period.

That they have to try and work out why it failed from the huge installer output, then work out what the hell this "Module::Build" thing is, then upgrade it seperately, then rerun the installer again, is a case of the user working around a broken installation concept.

When Module::Build fixes this by implementing a back-compatibility strategy that doesn't involve the user being smart, this problem goes away.

Re:In Your Build.PL

chromatic on 2006-06-10T20:25:15

You are not paying attention. Let me explain very slowly.

CPAN.pm and CPANPLUS both somehow detect when a distribution has unfulfilled dependencies. Both offer to install those dependencies.

Both MakeMaker and Module::Build somehow indicate unfulfilled dependencies to the installer. I don't know if it's screenscraping or an API or whatever. I skimmed CPAN.pm this morning, but couldn't find it in two minutes.

Now if there's an API or if there's a particular error message being scraped and detected and if both MM and M::B use that communication mechanism (and there's no short-circuiting logic in this conditional, so evaluate this next clause), so can any other code.

If necessary, that code can even run before attempting to load M::B or MM or whatever.

What's the problem?

Re:In Your Build.PL

Alias on 2006-06-10T21:24:04

You are not paying attention. Let me explain very slowly.

There are three different conceptual code elements involved in a distribution.

The Installer.

The Build Process.

The Installed Module.

CPAN.pm detects the Installed Module has unfulfilled dependencies, and is informed of them by executing the installer.

This can theoretically be shortcut for static dependencies in a setset of cases, but in practice nobody ever sets that flag and so the installer MUST be run.

CPAN.pm detects the Build Process has unfulfilled dependencies, and is informed of them by executing the installer.

This can theoretically be shortcut for static build dependencies in a setset of cases, but in practice nobody ever sets that and so the installer MUST be run.

CPAN.pm does NOT detect the dependencies of the installer.

The idea that you can execute a program in order to find dependencies needed in order to execute the program is logically impossible. It is a circular dependency.

The installer MUST be able to run correctly and without crashing.

Once run, it reports to CPAN.pm (by modifying META.yaml to contain data tailored for the local host, or via Makefile parsing) the dependencies required for the build phase, and for the distribution at run-time.

In a vanishingly small number of cases, the META.yaml file has a flag enabled which says that the contents are the same on all platforms, and that the installer does not have to be run to determine dependencies.

And those cases are irrelevant for the purposes of designing Module::Build's back-compatibility feature.

There are only three solutions to this problem.

1. You bundle the installer, and never change the API (MakeMaker)

2. You bundle enough functionality to work regardless of the environment (Module::Install and Windows installers)

3. You make the installer FORCE upgrades to itself. (Windows Update, Steam)

This is fairly straight forward stuff.

Ken understands this quite well, and Module::Build may well be stealing some tricks from Module::Install down the track.

I suggest you read more into how CPAN.pm and the installers work together, and it should become quite clear why the current situation isn't enough.

Re:In Your Build.PL

Alias on 2006-06-10T21:25:15

That should read...

"This can theoretically shortcut for static dependencies in a subset of cases"

Re:In Your Build.PL

chromatic on 2006-06-10T22:43:25

...but in practice nobody ever sets that flag...

I am suggesting that, in those cases where the installation program depends on a specific minimum version of the installer module (and in several years of maintaining a couple of dozen of publicly available modules, I can think of one case where this was necessary), the author should do precisely that.

There already exists a perfectly good mechanism to mark and install dependencies through both of the installer shells. Why complicate the process?

I can see one potential drawback, and that is if, after resolving the dependencies, the shells do not re-execute the installer.

However, it seems to me that fixing the existing dependency resolution system in only two places is much, much less work than reimplementing a dependency resolution system in every module.

Re:In Your Build.PL

Alias on 2006-06-11T10:46:03

I am suggesting that, in those cases where the installation program depends on a specific minimum version of the installer module (and in several years of maintaining a couple of dozen of publicly available modules, I can think of one case where this was necessary), the author should do precisely that.

That is certainly an acceptable partial solution for the cases where dependencies are static, and one that individual authors can use today.

However, unfortunately, not all dependencies are static, and there is a strong requirement (look at something like File::HomeDir) that the installer be able to handle full programatic dependencies if it has to.

That need to do programatic host-specific dependency resolution is what complicates the process, and what is the cause of the M:B circular dependency problem.

So the static dependencies flag does not provide us with a complete solution.

I do though think it is quite reasonable for CPAN.pm to rerun the installer (or in the static case run it for the first time) after resolving the dependencies. If the installer introduced any additional dependencies, then CPAN.pm should bail out with an installer failure error.

However, it seems to me that fixing the existing dependency resolution system in only two places is much, much less work than reimplementing a dependency resolution system in every module.

I completely agree with you on this.

BUT currently this problem hasn't been fixed, and so long as Module::Build continues to have this problem, the Module::Install approach is valid.

This is why Windows installers are done as they are. Bundling the logic is a solution to the problem of a diverse or non-complete installation environment.

If Module::Build were to truly fix the dependency problem properly (and more likely it would be something that CPAN.pm, CPANPLUS, and M:B would need to collaborate on) then Module::Install would no longer be necesary, and we could either end-of-life it, or alter it into something smaller that integrated more cleanly with M:B.

But as long as the choice is a broken-but-idealistic Module::Build or an working-but-inelegant Module::Install, the imperative in installation-systems should always be to go with something that is working NOW.

Re:In Your Build.PL

chromatic on 2006-06-11T17:01:41

I'm still not sure I've explained myself well.

I mean, "Authors of distributions that rely on a specific version of the bundling module should be able to mark that dependency in such a way that the installer can detect that dependency immediately." Whether that means adding a new entry in META.yml or running a little bit of code at the start of Build.PL or Makefile.PL, I don't particularly care.

The latter seems easiest.

The point is, I believe you can run just enough of the bundling program to flag an unmet dependency on the bundling module that the installer shell should go off to resolve that dependency immediately. You don't have to continue running the program at all at that point.

I don't think getting this done means that Module::Install has to go away; I've always seen its biggest value for distributing full applications with their dependencies to users who can't, won't, or shouldn't install it themselves from the CPAN.

Re:In Your Build.PL

Alias on 2006-06-11T19:26:37

Again, I do agree with you.

There SHOULD be a way of doing it :)

I can see a number of ways in which it could be done, a few of which you have described.

It's just that the situation remains that despite what should be possible, it still hasn't been written, and what I try to keep pointing out is that there needs to be, and that whatever the solution is, it needs to be completed and working before we can consider M:B to be stable and suitable for the core.

The fact that Module::Build is in the core is what I would consider premature. Certainly whatever the solution is, there needs to be a clear peer-reviewed (in the sense of me, Andreas, Audrey and the other people writing installer code) specification for doing it, and it needs to be implemented before the release of 5.8.9.

Oh, and apology accepted :)

Re:In Your Build.PL

chromatic on 2006-06-11T20:42:30

That is all reasonable; I quite agree.

Is the best place to discuss this on p5p or at a BOF somewhere?

Re:In Your Build.PL

tsee on 2006-06-12T07:28:49

I would think this could get lost amidst p5p traffic. How about the perl.qa mailing list? Various related issues have been discussed there in the recent past.

By the way, I'm glad the two of you agree now. I was really waiting for you (two) to realize you were not even contradicting each other. :)

Steffen

Additional correction:

Aristotle on 2006-06-10T08:09:31

Even users only need to upgrade their M::B when they’re about to install something. And CPAN.pm could check if there is an upgrade to M::B and suggest doing it just like what it already does for updates to itself.

Seems much more tenable than the M::I model to me.

I mean, look at that list – good luck getting all 83 authors to push out 469 new releases total, just to fix their installers. And that’s now that M::I is relatively new – what will the situation look like when the CPAN is full of stable or even forgotten modules?

Re:Additional correction:

Alias on 2006-06-10T18:25:04

How are the user's supposed to know to upgrade Module::Build.

Installers should always assume the user is completely clueless, or doesn't even exist because they've been replaced by a small shell script.

Yes, CPAN.pm could check and force-upgrade. I've had this conversation with Ken, and there's an RT ticket in the CPAN.pm queue that says "auto-upgrade a specific set of named modules". But it DOESN'T.

And no amount of handwaving about how other people (users or Andreas) could fix Module::Build's problem doesn't change the fact that right not it does have that problem.

Module::Install has suffered from the same problem as Module::Build in regards to having to incrementally release all those modules. It was hyped too heavily, too early and too often. Both M:I and M:B are incomplete installers with significant problems to be resolved.

As for getting modules upgraded, the same rules we have always had apply. When someone needs it, they ask the author, or they take co-maint and do a release themselves.

The number of modules existing is indeed too high.

But I don't recommend people use EITHER M:I or M:B until they reach version 1.00 and become more stable. So I continue to only recommend M:I only in the specific case the author will be doing regular releases.

Fortunately, M:I is trying to avoid hyping any more until the major problems have been removed and it has been tested properly.

Re:Additional correction:

chromatic on 2006-06-10T20:27:45

How are the user's supposed to know to upgrade Module::Build.

If only there were some way CPAN or CPANPLUS could know about dependencies during an installation... yeah, that'd be SWEET.

Re:Additional correction:

Alias on 2006-06-10T21:33:29

I find that not only a cheap shot, but a rediculous statement coming from somebody that doesn't actually know how the mechanism by which CPAN learns about dependencies during an installation. Both of them.

Until you learn how the Perl installation process works, it's pointless to continue this too and fro.

Re:Additional correction:

chromatic on 2006-06-11T02:05:17

You're right, it was a rude and unhelpful statement. Please accept my apology.