Found on InfoSec Writers
One of the biggest problems in computer security is how to deal with logs. Firewall logs are especially difficult because they often run into the tens to hundreds of gigabytes, and searching them requires expensive, proprietary packages. The purpose of this paper is to help instruct the reader in how to use Postgres and Perl to have a queryable database using checkpoint log files. I do not cover other firewalls but the ideas herein could be applied to almost any firewall supporting output to text files. This paper will give the security practitioner to perform data mining on otherwise useless logfiles. Using the methods described in this document enabled the author to query for almost anything imaginable from a total of 450,000,000 records in under ten minutes. The cost: The cost of the hardware.
