One of my contracts involves writing Perl scripts that read logs files and try to find out intersting data in them (that's original, isn't it?), put some of the data in RRD while also outputing some top 10 reports, etc.
After writing three times the same 20 lines script (only changing the fields names) as a proof that we could get some interesting information that way, I decided that it would more useful to write one longer script that would produce the kind of report I wanted (along with fancy filters to ignore useless information) from a configuration file. I would produce the same ouput (and give way to produce other interesting reports I hadn't thought of yet).
Then I thought that the way to go was to build a tree of nodes (input nodes, filter nodes, output nodes) that would pass data to one another tp modify it, filter it and produce the reports. Each node would be given a code reference as its "action" (the parameter being the data).
I got the client to accept a silly name for the project (DAta Munging Network - Damn) and started coding it. After having produced Damn, Damn::Node, Damn::Source and Damn::Filter (achieving near 100% overall code coverage with my test suite, some of the tests code being actually handled by nodes doing is_deeply() in their action code), I began to test it to read through the log files, filter some lines out, aggregate some data and update a RRD.
The 6 nodes network was just damn slow. I reduced it to a 3 node network that was only a little less slow. The conclusion is that it is totally unusable as it is.
Well, the good news is that it only took one and half day to fly from big idea to bit bucket. :-)
My next attempt to do something generic will probably involve code templates and a big nice eval $code as the last statement.
