Running the CPAN shell on ActivePerl 5.8.x on Windows XP, I can see in Process Explorer that the bare CPAN shell uses a whopping 195MB of RAM (size of working set: 120MB). I mean... wauw!!
That does not even include the programs, including perl, that are invoked from make, that actually test or install stuff. No, just the bare shell.
WTF is going on? Does the CPAN shell perhaps try to keep the whole module database in RAM? And if so: why? It's not as if looking up a distro by module name is so system critical, that it requires a sub-millisecond response time. So a slightly slower system, that greps through the data on file, would work just as well.
Fact is: memory consumption is 3 times lower before the metadata is first loaded. And it doesn't ever go down again... (As if it could.)
CPAN is now using several times more RAM than the average computer had, when CPAN.pm first came out. I doubt it used that much RAM, in those days.
p.s. For some bizarre reason, Strawberry Perl 5.10.x uses "only" 120MB. I have no idea why.
Does the CPAN shell perhaps try to keep the whole module database in RAM?
Yes, that's how it does some of the query commands it implements. It has been so for a long time now. And yes, under-powered computers may have problems with that. (But it can be worse - CPANPLUS is much heavier or at least was, a long time ago when I tried it.)
CPAN.pm has an experimental support for the module database in a SQLite db, as implemented by CPAN::SQLite. It decreases the memory consumption. You can try that.
Re:Because it is
Yanick on 2009-02-12T15:34:45
Re: CPANPLUS
It's been also quite a long time since I've looked at how much RAM CPANPLUS takes, but the last time I did, it was raking an impressive 500M. Which, of course, makes it almost impossible to use interactively on the 512M of RAM of my poor machine.
Re:Because it is
kane on 2009-02-13T11:09:32
CPANPLUS uses far less on my machine, but YMMV; for those strapped for ram, there's the option to use an SQLite backend.
To enable it, simply follow the following steps from the default shell:
### set SQLite as the backend
CPAN Terminal> s conf source_engine CPANPLUS::Internals::Source::SQLite### update any dependencies (will install sqlite drivers if you need them)
CPAN Terminal> s selfupdate dependencies### save configuration
CPAN Terminal> s save
Strawberry comes bundled with CPAN::SQLite on by default, which is why it uses less memory.
This still has problems though, because the process that fills up the SQLite database loads the entire database into memory ANYWAY, instead of processing streamwise.
Re:What Strawberry does
bart on 2009-02-13T22:37:10
Ah, I already thought I saw a different behaviour pattern in memory usage for Strawberry Perl: the bulk of memory is not used by the CPAN shell script, but by a script launched from it, for loading the database. The latter script uses up to 120-150MB, which is already better, but yet better still because the script exits after half a minute or so. The CPAN shell itself uses less: 60MB virtual (35MB working set). It still is a lot, but at least, it's already 3 times less than the default in ActivePerl.
I ran into this as a problem when using CPAN on my openmoko (Debian), which has only 128mb of ram. It only seemed to be a problem the first time I ran cpan, after that it seems better.
But for that first time I had to shut down everything (X-Windows, phone services, etc) to get it working.
