Heh. I'm so...grrr.
We have a linux firewall (iptables). We generate a lot of logfile data, naturally, so I roll them nightly and store them, in case I ever decided to look at them. Which I probably never will.
Recently built a new firewall system to replace the old one. Rack mounted, dual processor, RAID, etc. Decided to keep the same hostname and configuration as the old system, so I wouldn't have to reconfigure all the client machines to point to a new box. Copied all the necessary scripts, config files, etc, to a CD. Installed on new system. Flipped. All was well, until...
I rolled into the server room yesterday and found all the log output from iptables going to the console. WTF! I ran the syslogd script to restart syslogd and klogd, and things quieted down. Now to hunt for the problem.
The syslogd.conf file was fine. Checked the logs and noticed that the last entry to the logs was made at midnight last night/this AM. Found that the logging stopped and klogd never properly restarted.
Looked at my logroll perl script. The culprit turned out to be me and my lack of error checking. My logroll script created a file with the current date, gzipped it, then was supposed to store it in a subdirectory named after the month of the file (nov, dec, etc). Part of the routine was to roll through a hash to find the right month name and match the directory with the filename.
The problem occurred when the script looked for the "nov" subdirectory and it wasn't in the right place. When I wrote the original script, I had already manually created the directories ahead of time, so the script didn't have to check for their existence.
When I moved everything to the new firewall system, I neglected to create to directories, and the script didn't error-check for their existence and create them. Bad boy. The script stopped (or paused, actually) klogd and syslogd, and never restarted them because of the file error. And the iptables process kept on logging, only it all went to the console.
Amazing how something so fundamental could slip by me. You learn each day.
