Something you might find useful...

Matts on 2004-03-18T22:43:31

I have a problem: very fast moving logs that I want to monitor in realtime for various "tokens". I asked on #perl if anyone knew of a standard tool (like `watch` or `tail`) that could do this but nobody did, so I wrote this:

#!/usr/bin/perl -w

use strict;
use Getopt::Long;
use Time::HiRes qw(time);

my @Watches;
my $interval = 1;
GetOptions("watch=s" => \@Watches, "interval=i" => \$interval);

print "Watching logs every $interval seconds\n";
$|++;
my $secs = 0;
my $total = 0;
my @counts = (0 x @Watches);
my $tzero = time;
my $t0 = time;
while (<>) {
    $total++;
    for my $i (0 .. $#Watches) {
        if (index($_, $Watches[$i]) >= 0) {
            $counts[$i]++;
        }
    }
    my $curtime = time;
    my $diff = $curtime - $t0;
    if ($diff >= $interval) {
        $t0 = $curtime;
        printf "\rLines/s: %0.2f", ($total / $diff);
        $total = 0;
        if (@Watches) {
            for my $i (0 .. $#Watches) {
                printf ", $Watches[$i]/s: %0.2f", ($counts[$i] / $diff);
                $counts[$i] = 0;
            }
        }
        # fixme - don't use fixed num of spaces here
        print "                                  ";
    }
}

Run it without any options to get the number of log lines per second (pipe the logs in with tail -f) or pass in --watch FOO to count the lines containing the string "FOO".

---

Have you seen sec?

runrig on 2004-03-18T23:53:41

Maybe overkill for your purposes, but sec looks interesting. I haven't used it, but heard about it at a PM meeting.

curious

jhi on 2004-03-19T06:44:38

The tail -f from the PPT or the File::Tail (some of the very first hits for "tail" from search.cpan.org) don't do what you want?

Re:curious

Matts on 2004-03-19T21:04:00

They are totally different. I already have a tail command. If I run the above script with --watch MAIL it tells me how many lines matching /MAIL/ it sees per second.

sounds like snort

mock on 2004-03-20T00:07:46

Sounds a lot like what snort does. Perhaps it would be worth sending your logs past some sort of network interface so that snort can see them.