Dave Rolsky wrote recently an interesting blog entry called The Real Problem With Dependencies. I liked the comparison of CPAN shell and Debian packages: The difference is that debian packages always install cleanly. I also never heard of debtree tool before. Dave used it to plot gimp dependencies as an example. This blog entry will be about my Perl-Debian packaging marathon.
Look at this console-shot:
$ sudo apt-get install corp-libcatalyst-modules-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
corp-libalgorithm-c3-perl corp-libany-moose-perl corp-libappconfig-perl corp-libauthen-simple-perl corp-libb-hooks-endofscope-perl
corp-libbit-vector-perl corp-libcache-cache-perl corp-libcache-fastmmap-perl corp-libcarp-assert-more-perl corp-libcarp-assert-perl
corp-libcarp-clan-perl corp-libcatalyst-perl corp-libcatalyst-view-tt-perl corp-libcgi-formbuilder-perl corp-libcgi-simple-perl
corp-libclass-accessor-chained-perl corp-libclass-accessor-grouped-perl corp-libclass-accessor-perl corp-libclass-c3-adopt-next-perl
corp-libclass-c3-componentised-perl corp-libclass-c3-perl corp-libclass-data-accessor-perl corp-libclass-data-inheritable-perl
corp-libclass-inspector-perl corp-libclass-mop-perl corp-libclass-throwable-perl corp-libconfig-any-perl corp-libcrypt-passwdmd5-perl
corp-libdata-alias-perl corp-libdata-dump-perl corp-libdata-optlist-perl corp-libdata-page-perl corp-libdata-visitor-perl
corp-libdate-calc-perl corp-libdbi-perl corp-libdbix-class-perl corp-libdbix-class-schema-loader-perl
corp-libdevel-globaldestruction-perl corp-libdevel-stacktrace-perl corp-libdigest-hmac-perl corp-libdigest-sha1-perl
corp-libemail-abstract-perl corp-libemail-address-perl corp-libemail-date-format-perl corp-libemail-messageid-perl
corp-libemail-mime-contenttype-perl corp-libemail-mime-creator-perl corp-libemail-mime-encodings-perl
corp-libemail-mime-modifier-perl corp-libemail-mime-perl corp-libemail-send-perl corp-libemail-simple-creator-perl
corp-libemail-simple-perl corp-libemail-valid-perl corp-liberror-perl corp-libexporter-lite-perl corp-libextutils-autoinstall-perl
corp-libextutils-install-perl corp-libextutils-parsexs-perl corp-libfile-copy-recursive-perl corp-libfile-modified-perl
corp-libfile-remove-perl corp-libfile-slurp-perl corp-libhtml-parser-perl corp-libhtml-prototype-perl corp-libhtml-scrubber-perl
corp-libhtml-tagset-perl corp-libhtml-tree-perl corp-libhtml-widget-perl corp-libhttp-body-perl corp-libhttp-request-ascgi-perl
corp-libhttp-response-encoding-perl corp-libhttp-server-simple-perl corp-libipc-sharelite-perl corp-libjson-any-perl
corp-libjson-perl corp-liblingua-en-inflect-number-perl corp-liblingua-en-inflect-perl corp-liblist-moreutils-perl
corp-liblocale-maketext-lexicon-perl corp-liblog-log4perl-perl corp-libmailtools-perl corp-libmime-types-perl
corp-libmodule-corelist-perl corp-libmodule-find-perl corp-libmodule-install-perl corp-libmodule-pluggable-fast-perl
corp-libmodule-scandeps-perl corp-libmoose-perl corp-libmoosex-attributehelpers-perl corp-libmoosex-emulate-class-accessor-fast-perl
corp-libmoosex-methodattributes-perl corp-libmoosex-types-perl corp-libmouse-perl corp-libmro-compat-perl
corp-libnamespace-clean-perl corp-libnet-daemon-perl corp-libnet-dns-perl corp-libnet-domain-tld-perl corp-libnet-ip-perl
corp-libobject-signature-perl corp-libpar-dist-perl corp-libparams-util-perl corp-libparams-validate-perl corp-libparent-perl
corp-libpath-class-perl corp-libplrpc-perl corp-libreturn-value-perl corp-librpc-xml-perl corp-libscope-guard-perl
corp-libscope-upper-perl corp-libset-object-perl corp-libsql-abstract-limit-perl corp-libsql-abstract-perl corp-libsub-exporter-perl
corp-libsub-install-perl corp-libsub-name-perl corp-libsub-uplevel-perl corp-libtemplate-perl corp-libtemplate-plugin-class-perl
corp-libtemplate-provider-encoding-perl corp-libtemplate-timer-perl corp-libtest-exception-perl corp-libtest-longstring-perl
corp-libtest-mockobject-perl corp-libtest-use-ok-perl corp-libtest-www-mechanize-perl corp-libtext-simpletable-perl
corp-libtie-toobject-perl corp-libtimedate-perl corp-libtree-simple-perl corp-libtree-simple-visitorfactory-perl
corp-libuniversal-can-perl corp-libuniversal-exports-perl corp-libuniversal-isa-perl corp-libuniversal-require-perl corp-liburi-perl
corp-libvariable-magic-perl corp-libwww-mechanize-perl corp-libwww-perl corp-libxml-parser-perl corp-libyaml-perl
corp-libyaml-tiny-perl corp-perl corp-perl-base corp-perl-modules libexpat1
Suggested packages:
corp-libauthen-simple-cdbi-perl corp-libauthen-simple-dbi-perl corp-libauthen-simple-dbm-perl corp-libauthen-simple-net-perl
corp-libauthen-simple-radius-perl corp-libconfig-tiny-perl corp-libxml-simple-perl corp-libconfig-general-perl
corp-libhtml-mason-perl corp-libtest-pod-perl corp-libtest-pod-coverage-perl corp-libfcgi-procmanager-perl
corp-libcatalyst-engine-apache-perl corp-libhtml-template-perl corp-libtext-template-perl corp-libcgi-session-perl dbishell
corp-libdbd-sqlite3-perl corp-libdbix-contextualfetch-perl corp-libclass-trigger-perl corp-libsort-versions-perl corp-libppi-perl
corp-libdbd-csv-perl corp-libxml-dom-perl corp-libcompress-zlib-perl libapache2-mod-perl2 libtemplate-perl-doc corp-perl-doc
corp-libterm-readline-gnu-perl corp-libterm-readline-perl-perl
Recommended packages:
corp-libauthen-simple-pam-perl corp-libauthen-simple-passwd-perl corp-libauthen-simple-http-perl corp-libauthen-simple-ldap-perl
corp-libauthen-simple-smb-perl corp-libauthen-simple-kerberos-perl corp-libfcgi-perl corp-libclass-c3-xs-perl
corp-libdatetime-format-mysql-perl corp-libdatetime-format-db2-perl corp-libdatetime-format-pg-perl corp-libfile-spec-perl
corp-libsql-translator-perl corp-libemail-send-io-perl corp-libjson-xs-perl corp-liblog-dispatch-perl corp-libipc-shareable-perl
corp-libclass-method-modifiers-perl corp-libarchive-zip-perl corp-libdbd-anydata-perl corp-libclass-dbi-perl
corp-libtemplate-plugin-gd-perl corp-libtemplate-plugin-xml-perl corp-libtemplate-plugin-dbi-perl corp-libio-socket-ssl-perl
corp-libhtml-format-perl
The following NEW packages will be installed:
corp-libalgorithm-c3-perl corp-libany-moose-perl corp-libappconfig-perl corp-libauthen-simple-perl corp-libb-hooks-endofscope-perl
corp-libbit-vector-perl corp-libcache-cache-perl corp-libcache-fastmmap-perl corp-libcarp-assert-more-perl corp-libcarp-assert-perl
corp-libcarp-clan-perl corp-libcatalyst-modules-perl corp-libcatalyst-perl corp-libcatalyst-view-tt-perl corp-libcgi-formbuilder-perl
corp-libcgi-simple-perl corp-libclass-accessor-chained-perl corp-libclass-accessor-grouped-perl corp-libclass-accessor-perl
corp-libclass-c3-adopt-next-perl corp-libclass-c3-componentised-perl corp-libclass-c3-perl corp-libclass-data-accessor-perl
corp-libclass-data-inheritable-perl corp-libclass-inspector-perl corp-libclass-mop-perl corp-libclass-throwable-perl
corp-libconfig-any-perl corp-libcrypt-passwdmd5-perl corp-libdata-alias-perl corp-libdata-dump-perl corp-libdata-optlist-perl
corp-libdata-page-perl corp-libdata-visitor-perl corp-libdate-calc-perl corp-libdbi-perl corp-libdbix-class-perl
corp-libdbix-class-schema-loader-perl corp-libdevel-globaldestruction-perl corp-libdevel-stacktrace-perl corp-libdigest-hmac-perl
corp-libdigest-sha1-perl corp-libemail-abstract-perl corp-libemail-address-perl corp-libemail-date-format-perl
corp-libemail-messageid-perl corp-libemail-mime-contenttype-perl corp-libemail-mime-creator-perl corp-libemail-mime-encodings-perl
corp-libemail-mime-modifier-perl corp-libemail-mime-perl corp-libemail-send-perl corp-libemail-simple-creator-perl
corp-libemail-simple-perl corp-libemail-valid-perl corp-liberror-perl corp-libexporter-lite-perl corp-libextutils-autoinstall-perl
corp-libextutils-install-perl corp-libextutils-parsexs-perl corp-libfile-copy-recursive-perl corp-libfile-modified-perl
corp-libfile-remove-perl corp-libfile-slurp-perl corp-libhtml-parser-perl corp-libhtml-prototype-perl corp-libhtml-scrubber-perl
corp-libhtml-tagset-perl corp-libhtml-tree-perl corp-libhtml-widget-perl corp-libhttp-body-perl corp-libhttp-request-ascgi-perl
corp-libhttp-response-encoding-perl corp-libhttp-server-simple-perl corp-libipc-sharelite-perl corp-libjson-any-perl
corp-libjson-perl corp-liblingua-en-inflect-number-perl corp-liblingua-en-inflect-perl corp-liblist-moreutils-perl
corp-liblocale-maketext-lexicon-perl corp-liblog-log4perl-perl corp-libmailtools-perl corp-libmime-types-perl
corp-libmodule-corelist-perl corp-libmodule-find-perl corp-libmodule-install-perl corp-libmodule-pluggable-fast-perl
corp-libmodule-scandeps-perl corp-libmoose-perl corp-libmoosex-attributehelpers-perl corp-libmoosex-emulate-class-accessor-fast-perl
corp-libmoosex-methodattributes-perl corp-libmoosex-types-perl corp-libmouse-perl corp-libmro-compat-perl
corp-libnamespace-clean-perl corp-libnet-daemon-perl corp-libnet-dns-perl corp-libnet-domain-tld-perl corp-libnet-ip-perl
corp-libobject-signature-perl corp-libpar-dist-perl corp-libparams-util-perl corp-libparams-validate-perl corp-libparent-perl
corp-libpath-class-perl corp-libplrpc-perl corp-libreturn-value-perl corp-librpc-xml-perl corp-libscope-guard-perl
corp-libscope-upper-perl corp-libset-object-perl corp-libsql-abstract-limit-perl corp-libsql-abstract-perl corp-libsub-exporter-perl
corp-libsub-install-perl corp-libsub-name-perl corp-libsub-uplevel-perl corp-libtemplate-perl corp-libtemplate-plugin-class-perl
corp-libtemplate-provider-encoding-perl corp-libtemplate-timer-perl corp-libtest-exception-perl corp-libtest-longstring-perl
corp-libtest-mockobject-perl corp-libtest-use-ok-perl corp-libtest-www-mechanize-perl corp-libtext-simpletable-perl
corp-libtie-toobject-perl corp-libtimedate-perl corp-libtree-simple-perl corp-libtree-simple-visitorfactory-perl
corp-libuniversal-can-perl corp-libuniversal-exports-perl corp-libuniversal-isa-perl corp-libuniversal-require-perl corp-liburi-perl
corp-libvariable-magic-perl corp-libwww-mechanize-perl corp-libwww-perl corp-libxml-parser-perl corp-libyaml-perl
corp-libyaml-tiny-perl corp-perl corp-perl-base corp-perl-modules libexpat1
0 upgraded, 148 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/18.9MB of archives.
After this operation, 67.2MB of additional disk space will be used.
Do you want to continue [Y/n]?
$ sudo apt-get install corp-libconfig-general-perl
$ catalyst.pl myapp
$ myapp/script/myapp_server.pl
$ w3m -dump http://localhost:3000 | head -1
myapp on Catalyst 5.80003
and corresponding debtree image - quite huge...
What's this??? It all started with our customer requirement:
Provide change management and quality assurance procedures to support a professional system maintenance with high quality and provide tools to facilitate promotion and rollback of software releases and configuration changes while maintaining operational availability.
The question was how? No way with CPAN shell. What about using packaging? Our operations saied "yes we are fine with packaging our own stuff but, ...":
This requirements meant that all Perl libraries + Perl binary it self, mod_perl and finally decision was made for Apache too, had to be repackaged.
Packaging all fron scratch will be not a good idea as there is really a lot of work done by Debian maintainers. At first I've tried to use packages from next stable (but not LTS) Ubuntu release - Intrepid, I've changed that all dependencies were prefixed with "corp-" + make them install in /corp but then I gave up. Stable sounds nice but we will sooner or later need the recent versions from CPAN. I've realized that Debian testing - Sid is exacly what we need. Sid has quite recent versions and package maintainers keeps on upgrading. For someone a distribution with "testing" in a name can be a bit "unprofesional" but each Perl module has an extensive test suit that has to pass before package is created. Also at least one person is using that version - the maintainer, and it's very likely that it was used by several people in Debian unstable before it entered testing. Debian package-flow allows to enter packages from unstable to testing only when there is no release-critical bug on package it self and all it's dependecies can go to testing as well. So the Perl modules in Sid are well tested befor entering testing :-) and the result is that Sid should have the latest CPAN snapshot of modules that works together fine. My result is that we now have Ubuntu LTS with Perl from Debian Sid. If it will be good or not only time will tell...
More about this idea + reasoning + how I did it + experience I gained with this project and how I would like to use it for open source world will be in my next blog entry, this one is already toooo long :-)
Re:CPAN is mostly great for developers
Aristotle on 2009-05-25T11:01:38
Tests are certainly necessary to run on every system for which the configuration is not a priori known.
OS vendor package repositories circumvent this by saying “any package in this set of packages is known with a certain degree of confidence to play well well with any other package within the same set”. This is what release engineering is about: working out which set of packages works together stably.
The CPAN has no provisions for any stable set of particular distributions. Instead, anything that gets uploaded is immediately “released”, so the state of how well everything plays together is constantly in flux. You cannot sanely do without running the test suite under such circumstances.
The real answer is David Golden’s.
Re:CPAN is mostly great for developers
chromatic on 2009-05-25T20:39:26
The CP5.6.2AN is a brilliant idea that could provide a lot of feedback to CPAN uploaders as well as CPAN users. I could imagine a useful subscription service that bundled together dependencies which tested successfully on a given platform.
Re:CPAN is mostly great for developers
jozef on 2009-05-26T08:24:46
What I really like about Debian testing is that it's already here.:-) It has plenty of maintainers that are already watching over about the modules. There is already bug-tracing tool for the packaged versions. Also another nice thing about packaging is that it is easy to include patches to the original source. For example Perl 5.10.0 it self has 22 packaging releases and includes 58 fixes patches (you can see which http://patch-tracking.debian.net/package/perl/5.10.0-22 search for "fixes/"). For a private repository everyone is fee to add patches to the Perl modules he needs when he needs and not wait for module author to accept it + release it. Re:CPAN is mostly great for developers
Aristotle on 2009-05-26T11:20:58
Thing is, with good-quality test suites (which Debian mostly lacks), you can automate much of the release engineering process. That’s part of a possible extrapolation from David’s post: a system whereby a new release automatically becomes part of the
testingpackage set if it passes not only its own test suite against when running against the rest oftesting, but with it installed, all of its (immediate and indirect) dependents withintestingalso pass theirs. That way you could get a “rolling”testingrepository that tracks the bleeding edge relatively closely yet stays fairly stable – as long as the test suites are sufficiently good. (For modules with many reverse dependencies, they tend to be.)Re:CPAN is mostly great for developers
jozef on 2009-05-26T08:51:05
The tests are important, everyone knows. One think that I want to try is to package Perl modules and keep the tests. Let's say in "/usr/share/t/package-name". Then it will be possible to run tests again when ever there will be a need. And this "need" should come up every time there is a new versions of a module as every other module that depends on the new one should be retested. At the end of the chain it includes our own "product" tests. This can not be done with CPAN shell, can it? Imagine that you start to have a problem with some production server, it will be possible to go to/usr/share/t/ run `prove -r .` and every piece of sw on that machine will be retested. That could cast some light.
Re:"Perl without threads" ?
jozef on 2009-06-02T08:06:06
Actually it was a "company decision". The Perl without threads is used here for a long time. The main reason is, that in some cases Perl without threads is 10-30% faster and as we never use threads here, it's fine to disable them. Actually I would like to know who and how uses Perl threads as one can never be sure if some CPAN code is (or stays) thread safe. And probably using POE, AnyEvent or other event loop system is much better than threads.Re:"Perl without threads" ?
Aristotle on 2009-06-02T08:52:13
Perl threads aren’t really threads, they’re a hobbled emulation of
fork()that still doesn’t work right in many cases (taint mode, anyone?). There are only two reasons to use Perl threads: 1) Windows 2) mod_perl. Both are bad ideas as far as I’m concerned. I prefer to compile Perl without threads and usefork()when I needfork().
