It's funny sometimes how easy it is to get distracted by things you need to make your programming life comfortable, before you can effectively move up to the next layer.
I for one spent a hell of a lot of time writing class-level utility distributions, from my first module Class::Autouse (run-time class loading) to a wide range of other modules in Class:: and a dozen other places.
I find these days I feel like I have pretty much have everything I need in order to write large volumes of clean, correct, fast (well fastish, but you can't please the hardcore optimizers who don't even like using hashref objects) distributions.
This has helped enormously in my commercial life, because it means I can churn out work and be sure that it will run on every platform, be unlikely to suffer from edge-case bugs, and generally just do what I want.
But this has led me to my next big problem.
As attractive as CPAN-like distributions are for building up large amounts of functionality in a controlled manner, they aren't very easy to work with once you have them packaged.
And I don't mean 1 at a time. One CPAN dist is easy to handle.
I mean 10 or 20 at a time, with all the usual CPAN dependencies and inter-dist dependencies. CPAN is very much an isolated system, and it can be hard to get similar benefits for your own stacks of distributions outside of CPAN.
That's not to say there aren't solutions. CPAN::Mini::Inject lets you tack some distributions onto a minicpan mirror, and it's a good approach. But it's very simplistic and written sysadmin-style, by which I mean it does one specific task from beginning to end, and doesn't break up the problem in a generalised manner, so that the individual solutions to the various parts of the overall problem can be reused later.
This is a similar situation to that which existed before I created Class::Inspector. It's not that you CAN'T do some of these things, but it isn't all prettified and been nicely encapsulated behind a simple interface.
So I plan to do something about that! (i.e. Time for another dist-writing spurt) :)
To kick things off, I've uploaded a very simple initial release of Module::Inspector. I plan this to be to distributions, what Class::Inspector is to classes.
That is, it's a module you can point at a tarball file, or an unrolled dist dir, or even something in a version control checkout (SVN and CVS are currently detected), and it will do whatever steps are necesary behind the scenes to make it easy to analyze complete modules.
Internally, I'm writing it in two layers.
The first layer is a document layer. This will look through the distribution and tag any files it can with a document class. In the first upload, this only covers "Perl files" (tagged as PPI::Document::File objects) and META.yaml (tagged as a YAML::Tiny object).
So to find the name and version from the META.yml document for a tarball, you would just do...
my $module = Module::Inspector->new(
dist_file => 'Config-Tiny-2.00.tar.gz',
);
my $meta_yml = $module->document('META.yml');
print "This is $meta_yml->{name} version $meta_yml->{version}\n";
Granted, this doesn't give you a whole lot, but it automates away the tedium of loading files and unzipping tarballs, and detecting (and compensating for) version control, and so on.
I plan to gradually expand the number of types of files supported, initially based on the sorts of tasks I need to do.
Some of these will be fairly trivial. While there is currently an imperative library for working with MANIFEST files, there's currently no OO/document module.
So for the next 0.02 release of Module::Inspector I've also written a simple Module::Manifest distribution (to be uploaded seperately) which can parse the Perl-specific MANIFEST format.
And as I need support for more files, I'll be writing any dists needed for this, or reusing existing ones as appropriate.
The second layer I'm planning to add to Module::Inspector is a more DWIM layer.
This will allow better ease of use for simple cases, with the methods loading documents and doing whatever behind the scenes.
So things like this should just work.
Module::Inspector->new( 'sometarball' )->dist_name;
Again, that seems fairly simplistic, but means that we have a good object model for distributions that allows the NEXT layer to be built relatively easily.
As for what this next layer will do, I have started on a Module::Collection (linked for future readers) implementation.
This will let me deal with a whole group of distributions at once.
If any of you have seen the way I structure my repositories, you'll have noticed I generally have a "/releases" directory which contains all the tarballs.
What I'd like is to be able to work with all of these at once.
So for a typical project, I'd like to (for example) take the entire releases directory, identify all the distinct distribution names in it, select the highest version of each, map their dependencies from their META.yml data, then inject those dists into the L/LO/LOCAL path of my CPAN installer, fire up CPAN.pm, and install the entire project in one hit.
This gives me a sort of "Go install that directory" functionality that then makes it extremely simple and straight forward to build commercial projects as groups of CPAN dists, and then install the the project, along with CPAN dependencies, anywhere I need to.
And of course, after THAT, I plan to start looking at CPAN::Index a bit more, so that I can take the same Module::Collection, and do an index merge into a minicpan checkout, and so on. And some of the trickier PITA problems are also removed in the process.
So... I'm I missing anything here? Thoughts? Questions?
Re:Collection
Alias on 2006-09-25T00:35:14
Can't guess off the top of my head, other than maybe CPAN::Mini::Inject...
As for where I've been, although I've moved into my new apartment, I can't get ADSL or wireless internet (or at least, not that doesn't suck hard) so I'm getting cable internet, which means a 1 week wait for the technician to turn up and connect it.
Which should happen on Wednesday I believe.
So for the last week I've been working in offline mode (although a client of mine was nice enough to donate a desk at their office that I'll be using for the next few days) so I've really only been online during business hours, and in reduced capacity (little email, mostly no IRC, etc).
Re:Collection
tsee on 2006-09-25T07:12:40
PAR::Repository does some of what you describe. The goals are different, but it also deals with scanning distributions, installing bunches, etc.
Fortunately, PAR distro as simpler than CPAN distros:) Re:Collection
Alias on 2006-09-25T10:32:47
Hmm... if so, perhaps we should look at recycling/sharing some of the code between the two?
Re:Collection
tsee on 2006-09-26T07:43:09
Hi,
thought about this again. I might have been a little too quick in my reply. Well, there *are* common ideas in between the two projects. One of the things that triggered this was what you said about selecting specifically the newest versions of a module, etc. That's something PAR::Repository::Client does by default. The code is attached, but you'll realize that it's only as much code because it also deals with arch-type and perl-version which you shouldn't need to care about. PAR::Repository::* also takes a very ad-hoc route when it comes to dealing with multiple distributions at once. If any code/module you use require()s another package, it fetches that and applies the "newest version" thing again. By the way, that reminds me the I really should think about what happens with "use Foo 1.14;".
So PAR::Repository could use a Module::Collection for some things, like insertion into a repository, but not the other way around. Right now, it just globs for distribution files unless you specify one.
Hope this makes my thoughts clearer;)
I'm looking forward to see Module::Collection. Drop me a note if I can assist.
Steffen
sub prefered_distribution {
my $self = shift;
$self->{error} = undef;
my $ns = shift;
my $dists = shift;
return() if not keys %$dists;
my $this_pver = $Config::Config{version};
my $this_arch = $Config::Config{archname};
my @sorted;
foreach my $dist (keys %$dists) {
# distfile, version, distname, distver, arch, pver
my $ver = version->new($dists->{$dist}||0);
my ($n, $v, $a, $p) = PAR::Dist::parse_dist_name($dist);
next if not defined $a or not defined $p;
# skip the ones for other archs
next if $a ne $this_arch and $a ne 'any_arch';
next if $p ne $this_pver and $a ne 'any_version';
# as a fallback while sorting, prefer arch and pver
# specific dists to fallbacks
my $order_num =
($a eq 'any_arch' ? 2 : 0)
+ ($p eq 'any_version' ? 1 : 0);
push @sorted, [$dist, $ver, $order_num];
}
return() if not @sorted;
# sort by version, highest first.
@sorted =
sort {
# sort version
$b->[1] $a->[1]
or
# specific before any_version before any_arch before any_*
$a->[2] $b->[2]
}
@sorted;
my $dist = shift @sorted;
return $dist->[0];
}Re:Collection
Alias on 2006-09-25T03:33:01
I also just noticed Module::Dependency, which is somewhat similar as well, although again, it only aims to do a specific task.
