Several binary packaging systems (notably RedHat Linux, Debian GNU/Linux and ActiveState ActivePerl) have a recurring problem with "dual-life" modules. These are modules that come with the Perl core, but are also distributed via CPAN.
The primary problem seems to be that binary packaging systems consider files to be inviolate. They freak out and can't handle the idea that there might be two packages which contain the same file, where the file from one of the packages can be legitimately installed over the top of the other.
It seems to be that we might be framing this in entirely the wrong terms. These dual-life modules aren't so much two separate packages, but represent "updates to the Perl standard library". Can we provide some assistance to the various Perl packagers in this regard? Should upgrades to, say, Test::Builder on CPAN mean that they should upgrade their core Perl distribution to a new revision?
I've already had a debian packager tell me he couldn't upgrade the debian PPI pacakge (and thus upgrade perlcritic) because I updated my Scalar::Util dependency to a version newer than provided by the core.
If we the Perl community (in the loose sense of the word) provide an official recommendation that packagers should incorporate upgrades to dual-life modules into their core Perl package, this might provide some assistance to corporate requests to people like RedHat to upgrade their Perl packages, because the official guidelines say to...
I think (part of) the solution is something that came up in the hallways of YAPC::EU a couple of times: de-core most of the modules and dual-life everything that can feasibly be.
Now it just needs to happen…
Re:Antiquated concepts
izut on 2007-09-26T07:35:01
I think another solution would be having the perl modules installed using packaging tools on a different path. Like, adding another directory to Perl's @INC and let that be the place where dual-life modules live.Re:Antiquated concepts
Alias on 2007-09-26T08:04:50
The current split is not necessarily a bad one.
What I just don't understand is why we have dual-life modules install to CORE rather than site, if the module is being upgraded beyond the "core" version that comes with Perl.Re:Antiquated concepts
rafael on 2007-09-26T08:17:49
Some distributions already split out some core modules into different packages; like, for example, CGI.pm. There's only one source package, but binary packages can be upgraded independently.
I intend to write a README for packagers for perl 5.10.0 in which I'll put that kind of advice. Of course, patches are welcome.Re:Antiquated concepts
Alias on 2007-09-26T09:42:26
Any patches from me would probably just be along the lines of changing "recommended" to "expected" or "required for TPF certification":)
Of course, then I'd have to create some form of certification and that would mean joining the TPF and that's a whole other ball game.
Perhaps "compliant"?
That way I can file a bug via RedHat Enterprise Support saying "The Perl packaging for blah is non-compliant":)
Just make sure the docs have some for of "No Really, Do It This Way" to them:) Re:Antiquated concepts
rafael on 2007-09-26T09:53:06
Ah, compliance. That means also a test suite that can be run against an isntalled perl, and that's something that's standing in the TODO list for some time too.Re:Antiquated concepts
Alias on 2007-09-26T22:33:29
No it doesn't.
I'm talking about a compliant setup for the binary packages, not for the Perl distribution itself.Re: Antiquated concepts
Maddingue on 2007-09-26T16:42:06
We must install dual-life modules in CORE in order for them to be loaded. Otherwise, if we install these modules in SITE, the old version in CORE will shadow the new version. That's what theUNINSTExtUtils::MakeMakerparameter is for (--uninstallforModule::Build). See for example CPAN-RT#22130.Re: Antiquated concepts
Alias on 2007-09-27T03:49:11
But if SITE is before CORE in @INC, won't they be loaded first?Re: Antiquated concepts
Ron Savage on 2007-09-27T05:00:28
But site is/not/ always before core. And that's a Perl problem big time. Re: Antiquated concepts
Alias on 2007-09-27T06:29:31
The I guess that's part of the "recommended/certified" setup we need to be testing for:) Re:Antiquated concepts
Smylers on 2007-09-28T12:24:02
What I just don't understand is why we have dual-life modules install to CORE rather than site, if the module is being upgraded beyond the "core" version that comes with Perl.
A distribution-supplied update to a core module should go into ‘vendor’. That leaves ‘site’ (probably under
/usr/local/) free for the local sys-admin, who may wish to manually install newer versions of modules than those supplied in the vendor's packages. Re:Antiquated concepts
iburrell on 2007-09-26T16:42:11
This is done on the recent Fedora packages. The @INC goes/usr/lib/perl5/site_perl, /usr/lib/perl5/vendor_perl, /usr/lib/perl5/5.8.8. The vendor_perl is there for the RPM packaged modules to keep them separate from locally installed ones. And it is deliberately before the core one so it is possible to install updated versions of core modules. Perhaps they should use an "Override" directory
doughera on 2007-09-27T12:53:39
This issue is in some ways broader than just whether or not a module has a dual-life. For example, any module for which a critical upgrade is needed can present the same problem. For some reason or another, you may find yourself wanting to replace one module without replacing the entire package.One obvious thing to consider is to have an "override" directory added to @INC. For further discussion of that option, please see the section:
=head2 Why isn't there a directory to override Perl's library?in Porting/pumpkin.pod.I'll just note that I wrote that entry about 11 years ago, and it's still the case that nobody's gotten around to making it. That section also includes an existing workaround (which I haven't tested in years, but certainly hasn't been deliberately broken).
For the general issue of @INC ordering, I refer the interested reader to the following 4-year-old proposal (which is actually just a rehash of an even older proposal). I think this proposal would have been a good way forward for 5.10.0, but nobody (myself included) actually did anything about it. http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2003-07/msg01247.html (The whole thread is worthwhile reading.)
de-core most of the modules?
bart on 2007-09-26T11:50:25
A net result would be that even more people would be less willing to use some modules. All too often, we encounter the restriction that a solution may not contain any non-core modules...Re:de-core most of the modules?
Aristotle on 2007-09-26T13:52:26
There are already many vendors who slice and dice the Perl package. There’s no real guarantee that a module will be there even if it’s “in core” anyway. We would just be blessing the de-facto situation. Hopefully an insistence on code modules only would then become untenable for enough people that maybe someone would finally find motivation for a good module bundling solution.
As for the places which would still insist that only core modules be used, well, I don’t think we can protect stupid people from themselves, nor should we let consideration for them block progress that would help on multiple other fronts.
Re:de-core most of the modules?
drhyde on 2007-10-04T09:00:39
Awesome! I'm stupid for trying to make life easy for the rsnapshot users!Re:de-core most of the modules?
Aristotle on 2007-10-04T10:28:31
You are making no sense whatsoever.
Re:de-core most of the modules?
drhyde on 2007-10-04T17:08:49
You said:
As for the places which would still insist that only core modules be used, well, I don’t think we can protect stupid people from themselvesRsnapshot doesn't require any non-core modules, and so has had to reinvent various wheels. Which, I guess, makes me, the current maintainer, stupid.
What doesn't make sense is removing stuff from the perl 5 core distribution. Although I live in hope that perl 6 will be slim and sylph-like, with only those modules included which are required to easily install other modules (so in perl 5 terms, CPAN.pm, LWP, Archive::Tar and their friends and relations).
Re:de-core most of the modules?
Aristotle on 2007-10-04T22:20:48
When I say “stupid”, I mean workplaces that could easily install modules but refuse. Writing what’s effectively shrink-wrap software in Perl (ie. rsnapshot has to run on any number of machines, none of which you have any control over) is a very different situation.
However, even in your case, I don’t know that trimming down the core would be a problem. Are you better served working against a) a large but insufficient core, reinventing various wheels as needed, or b) a minimal core and a good tool to bundle modules into your app? (It might be that we already have such a tool – PAR. But I haven’t used it, and I don’t often hear enthusiastic praises for it, so I don’t know whether it’s good enough to consider the problem solved.)
Of course, module bundling doesn’t require a minimal core, so my question might seem like a false dilemma. Point is, though: given good module bundling, is the size of the core any concern anyway? Also, don’t forget that vendors/distributors already slice and dice their Perl packages, so you can’t always even on the core. Given that, wouldn’t a good module bundling tool in fact be a better approach than relying on the core?
Debian GNU/Linux
... ha[s] a recurring problem with "dual-life" modules. ... The primary problem seems to be that binary packaging systems consider files to be inviolate.
In Debian's case it's (largely) true that one file path belongs to one package, but that isn't actually a problem for dual-life Perl modules. Core modules (such as CGI, which come in the perl-base and perl-modules packages) and Cpan modules (such as YAML, which each come in their own packages, such as libyaml-perl) are installed to different directories:
ubuntu6.06# perldoc -l CGI YAML
/usr/share/perl/5.8/CGI.pm
/usr/share/perl5/YAML.pod
And doing perl -V shows that @INC is set appropriately so that the Cpan module directory is searched first.
I've already had a debian packager tell me he couldn't upgrade the debian PPI package (and thus upgrade perlcritic) because I updated my Scalar::Util dependency to a version newer than provided by the core.
Fixing that wouldn't require any infrastructure changes to the system; if somebody created libscalar-util-perl package, that could install the latest version of Scalar::Util toPPI would find it but the original file is intact.
Though since you ask:
They freak out and can't handle the idea that there might be two packages which contain the same file, where the file from one of the packages can be legitimately installed over the top of the other.
Actually even that would be possible: see dpkg-divert(8) for how a package can provide a replacement file for one already installed a different package.
Re:Debian
jeremiah on 2007-09-27T10:12:59
I am going to have to try to use dpkg-divert since I am having this particular problem with cpanplus and debian. Bug report here: http://rt.cpan.org/Ticket/History.html?id=29259
Re:split them out altogether?
Alias on 2007-09-26T22:34:04
People that install from source aren't the problem here.Re:split them out altogether?
mcumming on 2007-09-29T22:07:06
Beard. When I was the Gentoo perl guy, we had PDEPEND's added precisely for this scenario - so that we could build newer versions of core modules immediately following the build/install of perl itself.
