All the Perl that's Practical to Extract and Report

Jifty/Catalyst are too dang big

jjore on 2007-04-10T08:40:18

I found a "nifty" bug in CPAN.pm today. If you try to install something gi-normous like Catalyst or Jifty, the CPAN.pm shell craps out halfway through because it deleted somepart of the stuff to be installed during the build for the umpteen billion dependencies.

I didn't track how many modules Catalyst adds to a clean 5.8.8 but Jifty added 174 distributions and eventually failed because of something to do with XML::Parser, WWW::Mechanize, and perhaps some simple thinkos.

Blarg. That's a whole heap of module.

Perl needs better packaging

SuperCruncher on 2007-04-10T11:05:47

This is a perfect example of why Perl needs better packaging. Something like PAR seems to be a good idea, but people don't seem to use it. Odd given that many people are having problems with installing dependency-heavy Perl modules.

The value of CPAN is greatly diminished if we don't have an easy packaging mechanism. Many PHP apps are much easier to install than equivalent Perl ones, solely due to dependencies.

Re:Perl needs better packaging

Alias on 2007-04-10T15:06:59

If I wanted to release giant spaghetti-code apps that required a large standard library and reimplemented a bunch of functionality over and over again, and had no unit tests and no code reuse between applications at all, I imagine we could do installers as simple as PHP apps too.

CPAN is complicated for a reason.

The answer is not to make packages stupider, it's to make the CPAN tools smarter.

Re:Perl needs better packaging

SuperCruncher on 2007-04-10T19:53:36

I don't like PHP and I'm not at all advocating PHP-style code and coding practices (do they have any, apart from 'hack'?). I'm just pointing out that many PHP apps, however dirty, are much easier to install.

I don't see how bundling prerequisite modules with an application (e.g. Plagger) or large module does any harm. More disk space will be used, but in most cases that won't be a problem. Some may prefer to install modules separately, and I'm not seeking to stop that - I just want an easier way of distributing Perl apps.

Making CPAN tools smarter isn't really the solution IMHO. Bundling is faster (than downloading n packages off CPAN), and good for packaging "traditional", desktop Perl apps (yes, there are several). The idea of downloading off CPAN, at least in the current form where the latest version of the module is always retrieved, isn't very good from a configuration management and QA perspective.

I don't see how I was even remotely proposing to "make packages stupider". I'm talking about bundling modules together in such a way that they can be easily installed.

Size does matter

sigzero on 2007-04-10T11:56:50

I wonder if all those dependencies are really needed. I was listening to a talk by brian d foy and he made a good point. Perl modules tend to be "all inclusive" of the spec and not "just what you need". That tends to make them rather largish. Now if you are doing something specific that probably works but once you step into framework land, I am not sure if that works as well.

Module::Tiny

Alias on 2007-04-10T13:59:48

Situations in which only a limited subset of the most basic functionality are required is why the ::Tiny series of modules was created.

Re:Module::Tiny

sigzero on 2007-04-10T14:30:39

Yup, that is what bdf said as well.

Not really a bug

Limbic Region on 2007-04-10T12:26:38

Well, I guess it is a matter of perspective since it is doing exactly what it says it does.

"Currently the cache manager only keeps track of the build directory ($CPAN::Config->{build_dir}). It is a simple FIFO mechanism that deletes complete directories below build_dir as soon as the size of all directories there gets bigger than $CPAN::Config->{build_cache} (in MB)."

Perhaps a better way of handling would be for it to recognize that it is removing directories that it needs for the current installation, pause and ask if you want to increase the build_cache parameter.

Presumably you accepted the defaults when initializing CPAN.pm which leaves a lot to be desired.

o conf init

Cheers,
L~R

Re:Not really a bug

Alias on 2007-04-10T14:01:14

Even if it does what it says it does, if what is says it does is broken, it's still wrong.

Re:Not really a bug

dagolden on 2007-04-10T15:23:21

I think that CPAN.pm should only check the cache size after it finishes running the queue, not in the middle. CPAN doesn't guarantee anything about how far over the cache limit the build cache can go and is pretty vague about how often the cache is checked. So I think this is a bug that can be fixed pretty easily.

It reminds me of the Uncertainty Principle in a way -- you get to exceed the cache limit until you actually go and measure how much cache you're using. As long as that time window is set appropriately, all is well.

Pretty vague...

LaPerla on 2007-04-10T22:06:04

There is the config variable scan_cache that determines how often the cache is teared down to its limit. You can set this variable only to two values: "atstart" or "never". This means that (modulo bugs, of course) it is impossible that the cache manager is run in the middle of a session. The usual bug that's happening instead is that somebody (usually Module::Install) calls CPAN.pm recursively. Then "atstart" can be at any point in time.

I hope that I soon get evidence about what really is going on on Joshua's installation.

Re:Pretty vague...

jjore on 2007-04-11T01:04:24

Jifty::DBI uses Module::Install which is apparently what kills the Jifty install. It turns out that upgrading CPAN fixes the problem. I suppose that modules with large dependency trees ought to have the non-errant versions of the installer in their pre-req lists.

Perhaps that should be dynamic - if CPAN is asking Jifty about installing it, Jifty can tell CPAN that CPAN needs to be at least 1.90 to be able to successfully install it.

Re:Pretty vague...

dagolden on 2007-04-11T01:39:01

Or maybe Module::Install should just die.

Re:Pretty vague...

Aristotle on 2007-04-11T06:30:45

That’s what I’d say. I understand why people consider it necessary, but I think the effort would have been better invested in fixing Module::Build’s bootstrap problem. (Last I heard, we’re getting there, though, so hopefully Module::Install can indeed just die, soon.)

Re:Pretty vague...

drhyde on 2007-04-22T01:37:36

That still leaves us with the problem of all those people who have bundled Module::Install with their own modules. Perhaps the solution is for CPAN.pm to include its own Module::Install which does nothing but die, and set PERL5LIB to make sure that its Module::Install killer always loads first.

Bwahahahaha

Re:Not really a bug

jjore on 2007-04-10T17:57:15

Andreas hinted the new CPAN.pm 1.90 might not have this problem. I'm testing it now.