Packaging complexity (huge!)

excalibor on 2006-08-14T08:46:50

The other day, the fine folks at Hiveminder announced his new site, made with Perl, using Jifty ().

I thought, let's have a look, and went to go for Jifty. It looks really cool! Maybe even as cool as Catalyst looks, yeah.

Holy cow! After three days fighting with CPAN modules, dependencies, failed tests, etc, I have given up.

It probably has to do with the packaging system of my GNU/Linux box and CPAN, which don't go along very well (I mean, there's a difference between doing aptitude install libtest-pod-perl and perl -MCPAN -e 'install Test::Pod', for example). The difference is that in the first case, apt is dealing with dependencies, paths, etc; in the former, it's CPAN, and they don't think the same way.

This is, of course, very frustrating. I had the same experience with Catalyst, and I also gave up (that's why I said that Jifty looks as cool as Catalyst looks, because I haven't been able to try none of them!

My rant is that these framworks are cool, and use a host of standard modules (in the CPAN sense) but they should be available 'batteries included'. I mean: you should not be so fragile as to be useless without all CPAN at hand.

I work behind a firewall, and CPAN access is, let's say, unconvenient. This may be the best of both worlds way of doing this: you offer the option of downloading a tarball with everything that's needed to get the framework going on, and then Makefile.PL (or Makefile, or another program) checks which of the CPAN packages it has downloaded within the tarball it really needs to install on the system.

This is the best of both worlds: I can make a Debian package from the tarball (carefully defining dependencies and so) and use apt to install the packages I am missing (so my system has track of paths, versions, and dependencies) or use CPAN from the local repository.

But I was not offer such a thing. Either it's available on debian packages or in CPAN tarballs, but I have to go fetching dependencies, and some fail (because I have something the way that package doesn't like) and I'm screwed.

So, what possibilities are there to simplify all these complexity so huge frameworks don't need a fresh system (maybe even compiled from the sources) to live happily in production boxes?

Will CPANPLUS solve this? Because if Perl 6 won't be able to benefit from CPAN easily (and I cannot, always, benefit from it by using Perl 5!) then neither Perl 6 nor these frameworks will have an easy path to success in the real world. Specially the web frameworks, with Ruby on Rails, Django, webpy, and all the Java frameworks out there...

(enough ranting, back to work)


Dependencies & Packaging

kane on 2006-08-14T09:49:42

Packaging is still a hot item in perl5 land. A few projects have adopted the 'boxed' idea, where they ship a version of their project, along with all the depenendencies needed to bootstrap it and start working.

This is of course not the type of thing you want to install as your production platform, but gets you testing/developing rather fast.

Two such projects are Catalyst with their Cat-in-a-Box and CPANPLUS with the cpanp-boxed tool that is shipped for bootstrapping purposes

As far as packaging is concerned, CPANPLUS is actually able to help now. For example, for debian, there is now the option to build debian packages automatically while you install them, or use a custom apt repository that's updated straight from CPAN. Take a look at http://debian.pkgs.cpan.org/ for details on how to use it.

Re:Dependencies & Packaging

excalibor on 2006-08-15T19:13:19

Thanks for the link to Debian CPAN packaging! I guess it will work on Ubuntu as well...

I'll give it a shot A.S.A.P.!

Re:Dependencies & Packaging

Aristotle on 2006-08-21T01:25:19

Another worthwhile (and production-worthy) effort is the Shadowcat Catalyst installer, a script that sets up some magic and then drives CPAN.pm (and ppm, if necessary) programmatically so that you end up with a working Catalyst without any questions asked. Just fire and forget.

I used to moan, bitch and gripe about Catalyst installation – but no more, ever since Matt published that script.

Aside from that: debian.pkgs.cpan.org is very cool.

Parallel install

jjore on 2006-08-15T05:23:21

So ignore your /usr/bin/perl and let Debian maintain it for you. Anything you do with CPAN is just going to muck it up anyway so you might as well just bite the bullet and install your own /usr/local/bin/perl where you *can* install from CPAN.

Re:Parallel install

excalibor on 2006-08-15T19:43:01

The parallel installation is a good idea, and it's the one I usually maintain in production (we have started using dh-make-perl and apt-get, and so far we are managing to keep everything in control, but our programs don't need more than 20 or 30 CPAN packages; Catalyst of Jifty use many, many more!)

However this would only patch the problem, not solve it...

I am thinking here (wild mindstorming with self in here) in a solution of managing complexity through layering: there's the CORE groups of packages (let's call it layer 0), then we classify layer 1 as those inmediately useful, but not so much to be on the CORE (to keep it tidy and softy), then layer 2, up to layer, um, say, layer 6 (which would be Acme:: :) This would need a careful watch of usefulness of modules (by statistics and votes, maybe?) but the same process that decides which goes to CORE and which not could be used here... Also, authors could propose the layer it would go, but this is wild thinking right now...

All packages in layer N would only have dependencies with other layer N packages, and on lower layers (N - 1, ..., 0). That would allow parallel installs, and even circular dependencies (because all those would only depend on the others and lower layers; the graph can be nightmarish, but it should be solvable: if necessary, there should be some safety modules that would provide some dependencies solved in order to break a vicious circle).

That way we could talk about a CORE (or layer 0) Perl install, a layer 1 Perl install, etc... Of course, you can have a layer 0 + some layer 1 packages, and even, if you are willing, to do all the CPAN work as it's done right now, but this would allow us to write:

cpan> show layer
Perl 5.8.9 built 2006-08-15 layer 0 (+ 9 layer 1 packages + 1 layer 2 package + 4 layer Unknown packages)
cpan> upgrade to layer 1
Installing all layer 1 packages from CPAN...
...
Done.
cpan> show layer
Perl 5.8.9 built 200-6-08-15 layer 1 (+ 1 layer 2 package + 4 layer Unknown packages)
cpan> quit

More or less... Is this hard? Sure it is! Is it useful? Of course! This would mean a "layer Max) install would intall All Official CPAN packages (all those not in layer Unknown), which may be overkill... Well, hard drives are cheap nowadays... :)

And it may be too late for Perl 5, but not for Perl 6 and CPAN6? let's see if this is doable...

Thanks!

Re:Parallel install

jjore on 2006-08-15T19:51:42

Er, wow. That's oodles more complex than I suggested. I suggested you sidestep the problem by having a completely separate perl install on your box. Instead of somehow simultaneously solving the packaging problem across CPAN, apt, and all other package systems (assuming you intend your solution to work on other OS distributions), you do an eensy bit of judo and go about your merry way without ever encountering a problem.

Well.. you get to keep the minor problem that you can never say /usr/bin/perl to get to the "real" perl. That's nothing that setting your shebang or PATH environment variables won't fix.

Re:Parallel install

excalibor on 2006-08-15T21:34:56

Well, of course you are right...

Starting all programs with a she-bang like "#! PATH_TO_PERL" and a good ole sed script on "make" or "make install" will solve the parallel installation `little problem', indeed!

I just used the sudden inspiration of all these replies to let some brainstorming with myself... your post gave me the best anchor to actually rant it... :-)

thanks and laters!

Try Catalyst again

Alias on 2006-08-15T06:07:11

I should note that recently Catalyst has been putting a lot of work into this problem, and has reduced the quantity and complexity of the dependencies markedly.

I'd have another shot at it again, if you haven't recently.

Adam K

Re:Try Catalyst again

excalibor on 2006-08-15T20:08:10

Ah, OK, I'll give it a try again, thanks!