All the Perl that's Practical to Extract and Report

Survival tip #39

jplindstrom on 2002-10-16T23:36:15

I emphatically did not know this.

Jan Dubois says: "If you ever use a 10MB string in a scalar variable, the hidden string buffer for this ariable will stay allocated, even if you just assign a new short string to it. This is even true for lexical variables after their scope has closed"

http://aspn.activestate.com/ASPN/Mail/Message/895049

This affects at least two programs that I know of off the top of my head.

Check for this (and other things) with Devel:Size

Elian on 2002-10-17T02:29:12

You can see this in action by looking at the size of the scalar with the Devel::Size module. You can see similar things with arrays when you shrink them down.

Reallocation

petdance on 2002-10-17T04:48:14

And, of course, even if all memory for that string is deallocated, Perl hangs on to it and never gives it back to the OS. This is a common question on mod_perl lists: "My Apache children just keep on eating memory!"

Re:Reallocation

jplindstrom on 2002-10-17T09:01:28

Yeah, but that's something else, that deallocated memory doesn't go back to the OS.

In this case it doesn't even go back to Perl, and that's a little worrisome if you occasionally get a peak of huge indata.

To me that doesn't matter _that_ much since I tend to avoid long running processes, restarting the program every once in awhile. Even if Perl behaves and you code your stuff the right way, there is always some module that will mess up and not break a circular reference, or some external lib will forget to release some resource.

Re:Reallocation

jhi on 2002-10-17T15:36:11

What is this rampant fallacy of applications giving memory back to the OS? That is completely dependent on the implementation of the malloc, and in many systems the (virtual) memory use never goes down, only up. I think glibc malloc does memory allocation using mmap(), and the munmap() by free() does make the process virtual memory use go down. But nevertheless, expecting free() (or at Perl level, undef()) to make one's memory use (as shown by, say, ps), is a false assumption. Completely explaining it requires explaining virtual memory and some inner workings of malloc, but off-hand a simpler cure is to tell people to look at the resident set size (aka working set) size, instead of the virtual set size.