To: perl5-porters@perl.org
From: adamk@cpan.org
X-Mood: Extremely annoyed and frustrated that nntp.perl.org appears read-only now and that nothing I do and nothing I try can make any messages get to any of the Perl lists any more, or even generate bounces.
Subject: The need for a YAML parser in the core (or dual lived)
I've written this mail three or four times now, in various variations. I've given up trying to send it to the mailing lists (the brick wall is hurting my head), so this is my only way to post to P5P.
--------------------
By necessity, the CPAN uses a turing-complete process to determine dependencies.
This allows us to provide configuration that adapts to different platforms and situations, for example in the trivial case platform-specific dependencies.
Currently we also provide the META.yml metadata file with most distributions. This provides meta-data in static form generated form the configuration script, but is considered to be advisory only, and not canonical in any way, as it reflects on the resolved dependencies for the author of the module.
However, we are left with the problem of what to do in the cases where the configuration phase itself has dependencies, as there is an obvious circular dependency.
The code needed to determine the dependencies cannot be executed due to missing (unknown arbitrary) dependencies.
There are two ways to workaround this problem that have been used historically.
The first is to have APIs and codebases simply calcify, maintaining compatibility and applying bug-fixes but not introducing any non-trivial additional features, as is the case with ExtUtils::MakeMaker.
This method also assumes that the end-user will remember to upgrade their toolchain regularly, and that the delta of cases in which this is not done and bugs occur is sufficiently small that the maintainers do not become aware of it.
This is not a viable long-term solution, but has provided the best workaround to date, with the least damage.
The second method is to bundle the installation infrastructure with the distributions themself. This is the approach taken by Module::Install, and has been relatively successful.
However, the Module::Install bundling approach assumes that all CPAN authors are diligent and will upgrade the packages and do incremental releases relatively responsively.
Having seen Module::Install used for some time now, it is becoming obvious that this is not sustainable.
The third method (currently used by Module::Build) is to assume universal end-user education about the build system.
That is, Module::Build does not provide a dependency for itself. Instead, end-users are expected to install or upgrade Module::Build before doing an installation.
In fact, for the generic case, every single end-user is required to install or upgrade Bundle::CPAN every single time they run the CPAN client, in case Module::Build, a plugin, or something else needed by the configure script.
This, obviously, is not practical or sustainable either.
We've dodged a lot of the problem for a long time, but the Module::Build situation has brought this to the point it MUST be resolved. And there is no way yet determined by anyone that allows us to solve this fully back-compatibly.
The solution being implemented is the configure_requires: META.yml key.
This is a static list of modules and versions that IS considered to be authoritative, and declares the dependencies required in order to run the Configure script.
If the configure script itself has any dependencies, these need to be moved post-recursion, so for example, configure_requires: depends statically on Foo, which has a complete configuration depending on Bar or Baz in various situations.
Adding this fixes a whole range of problems we are stuck with at the moment. Installation and/or upgrading Module::Build is just the highest profile of these.
It also would mean we make Module::Install less "bundly" and we can force upgrades of ExtUtils::MakeMaker to the newest versions, or implement plugins for Module::Build, and a range of other things.
In short, it finally gives configuration scripts the opportunity to reach back in time and drag old out-of-date installations forwards.
For cases up to and including the current version of Perl, once configure_requires: support is added to the clients, it will be necesary to upgrade Bundle::CPAN manually once.
After that, the parts needed by the configuration scripts should auto-upgrade correctly.
Which leaves us with the problem of the default installation.
Currently, 10-15% of CPAN is using Module::Build. So if not for the workaround and compatibility code layered on, that's 10-15% of CPAN that crashes or fails to install with the default build.
Default installations are the most important case for what we need to be dragging up to the future.
As such, it is now essential that SOME form of META.yml parsing support be included in the Perl code so that the default installation of Perl is able to support configure_requires: and thus to support configuration scripts properly.
How that is done, I honestly do not care.
YAML::Tiny dual-lifed into the core is one option that will work just fine.
Embedding it into the CPAN and CPANPLUS modules is another one, which kane dislikes because it means maintaining TWO seperate versions of it.
Other options involve merging someone else's C library into the core (libyaml or libsyck solutions). I'm not a core dev so I don't know if that's a good or bad idea.
And finally, would be to fork YAML::Tiny or similar as Parse::METAyml or CPAN::ParseMETAyml or something which doesn't advertise as YAML but can parse everything needed from META.yml.
I'll note that META.yml does differ slightly from YAML. For example, Unicode is a bad idea in META.yml, as some versions of Perl don't handle unicode so well still.
In any case, ultimately we need support for META.yml in the core. And I would encourage in the strongest possible terms that this be in by 5.10.0, before we have to suffer an entire additional generation of RedHat et al having default installs that are now considered not to work with CPAN by default.
Module::Build does not provide a dependency for itself.
What does either CPAN installer do when it runs Build.PL and a use Module::Build x.xx line fails? If the answer is "not the right thing", then why not?
Re:No M::B Dependency?
Alias on 2007-06-26T05:40:59
A dying Build.PL is already a signal for something or another... N/A perhaps?
As for "not the right thing", that would require something to actually document what the "right" thing might be, and specify that as part of some notional interface between CPAN clients and the Build.PL calling interface.
I'm not sure CPANPLUS even calls Build.PL as a script, it calls modules directly or something? (could easily be wrong).
I also find the idea of 1. Parsing die messages 2. Dependency specification via just calling Build.PL over and over and seeing what crashes each time (don't assume there's only going to be one configure-time dependency) scary and fragile.
