jdavidb writes "I've come to a situation where I have a very inefficient search algorithm taking six minutes to accomplish what should be a simple task. I've brought the minds of many coworkers to bear on it, and we've collectively realized that it's difficult to figure out exactly where the performance bottleneck is. I need to optimize something, but what?
Obviously, the solution is to profile the code and find out where the real bottleneck is. But I've never done that before, not even in C. I've heard rumors of DProf and stuff like that, but I don't know if that's the module I should use, or even what my choices are. What options do I have for code profiling, what tutorials, introductions, or other articles exist out there, and what common novice goofs am I about to spend today repeating?"
When you have a script that is composed of many subroutines, DProf can tell you how much time each subroutine is taking. Programming Perl, 3rd Edition has a chapter by Gnat Torkington on this very subject. Before I read that small section, I was a lost about profiling as anyone could be. There's a bit more to it than just using DProf, because that module writes it's findings in a rather cryptic format. Instead, there's a front end, dprofpp, that will help.
dprofpp -p scriptname
should do it.
Re:DProf and GraphViz?
2shortplanks on 2001-10-25T16:36:33
I remember acme talking about graphing using DProf, creating easy to understand diagrams at YAPC::Europe 2000.Tap tap tap
Ah, here are his slides: http://www.astray.com/graphing_perl/. Maybe you could convince him to post some source code somewhere....after all we do know how he loves to go on about graphing.
For larger bits of code, Devel::DProf is very good. The documentation walks you through how to use the module simply.
Oh, and Devel::Cover is starting to do profiling too...
Thanks for the help, folks. My 6 minute 40 second worst (?) case search is now virtually instantaneous. Actual problems turned out to be in reading from the disk. File::Tail::read to be specific. Not a problem with the module, just my abuse of it.
For the record, it doesn't help you to read a whole file in and then try to perform an efficient search. You absolutely have to, have to, have to read only parts of the file with seek/tell/sysread type functions. There's no getting around it! But, you'll be glad you did.
Yes, I know I should have seen that earlier.
Fortunately, as I explained to a curious coworker, it also doesn't help to rewrite the same thing in C. An inefficient algorithm in Perl will also be bad in C, only it will take you two months to write it instead of two days.
Re:Thanks!
hardcode on 2001-10-31T13:58:53
I've had a similar experience here, I need to pass thru a 200000 line text file 3000+ times.
I was converting a date+time in ISO format to epoch seconds every time using Time::Local, now I do it up front (an obvious optimisation, didn't even need a profiler) and that *did* improve the speed dramatically, but it was still slow. The solution? Pre-build an index into the file to avoid having to go thru each line (even tho I was calling next if the line wasn't in the timeframe I was looking for) and lo and behold, down from 4 passes thru the file a minute to 4 per second!!
What I'm rambling on about here is that most of the time you don't need a profiler, just some sense and a little ingenuity. I *was* however suprised at how slow Time::Local seems to be (tho to be fair I *was* working it's little arse off)
hc
