Instrumenting your code

jdavidb on 2006-04-19T17:43:50

How do you instrument your code? Almost all of my code runs automatically as scheduled jobs, and I code with the principle that I don't want to see any output at all unless an error or unexpected condition has occurred. If I need to dig into the inner workings of a program, I add debugging statements and then receive the result in my email (from a cron job) or to the terminal (when I run it from the command line).

Meanwhile, the other folks throw all kinds of junk into a log file in /tmp. Very, very hard to wade through.

Cary Millsap's Optimizing Oracle Performance remarks that through the years Oracle has provided more and more instrumentation to allow you to figure out what is going on inside. They do a lot of great performance work by parsing Oracle log files and sticking the results into database tables for manipulation.

Meanwhile, from the same folks, I noticed this seminar with an Oracle expert has a one hour session entitled "Instrumentation 101." Sounds like something I'd love, but my interest for some of the rest of the seminar contents is low at this time.

But I really would like to get into instrumenting my programs in a better way. I don't want to spew junk to a log file, and I want to know what happened prior to an error when I receive one in email. It probably all needs to be in the database somewhere. We're already seeing some mandates to start collecting some basic statistics. What do you do?

---

More about instrumentation from Tom Kyte.

rblackwe on 2006-04-19T18:13:27

More about instrumentation from Tom Kyte. http://tkyte.blogspot.com/2005/06/instrumentation.html

Just a Guess...

Dom2 on 2006-04-19T18:37:10

I would hazard a guess that you could do something like this log4perl. You'd need to configure some appender to record the last (say) 100 log lines at debug, in memory, in a circular buffer. When the crash hits, mail them off.

-Dom

Another instrumentation class

pemungkah on 2006-04-19T21:17:53

NASA finally open-sourced an older version of the CGI stuff I worked on for them. I seem to remember a Perlier logging class in there somewhere that I should clean up and re-release.

Log everything, iterate your configs

bluto on 2006-04-20T15:36:02

I think the main problem with logging is that initially you don't really know what information will be useful so it's best to log much more than you think you will need. For me this also applies to the format of the logs -- I just don't know what will be the best format to delog when a problem does occur. For some apps it seems that certain types of logs (e.g. text) are just hard sift through, when for others those same types work great.

If you use something like log4perl or Log::Dispatch, as suggested earlier, you can set up multiple levels of logs that you can adjust without altering code, and then just spew as desired. For example: initially log details to a database and a series of space-managed (e.g. rotated) log files until you decide which way is really the easiest for you to delog; log a subset of messages to another file to give you a higher level view of what is going at a glance and perhaps mail the tail of this file if an important event occurs. Over time you can adjust how these are configured, but at least you don't have to fiddle with code as much.