CPANdeps efficiency improvements

drhyde on 2008-10-25T12:43:42

CPANdeps has been getting more heavily used recently, and all of the users hitting my shonky code have been occasionally DOSsing the server. That's because the old code would eat up to 134MB of memory to generate a set of results. And that's because I was using Parse::CPAN::Packages to convert from module names to package names. That reads the entire 02packages file into memory and converts it into a very large data structure.

The new version of the code stores that in the database instead, so instead of using 134MB I now only use 25MB per request. This means that I'm hitting the database harder, but even so, things should still run a lot better because even serving 25 requests in parallel (the maximum it's configured to do) it will only need 625MB, so you won't all be fighting over swap space.

I would like to thank Jonathan Rockway once again for writing the excellent CPANdeps test suite which lets me be confident that the site still works just as it did before. His test suite is available on the CPAN as Angerwhale :-)