For the last couple of weeks I've been stuck in ip_conntrack hell. What is this mystical thing you may ask?
ip_conntrack is the module the Linux netfilter system uses to track connection states, and thus provide a stateful firewall (which everywhere you read will tell you is more secure than a stateless firewall, but won't justify that statement).
Now for some reason the kernel we're using on our spamtrap has started having a growing ip_conntrack table. This happened when we switched over to qpsmtpd for our spamtrap, but I have a funny feeling it may not be related to that, but possibly related to a change in spamware.
What's happening is that although connections are closing just fine, the conntrack table isn't seeing the close, and leaves the connection in ESTABLISHED mode. So qpsmtpd doesn't have a connection open, and neither does a netstat see a connection, but we have a conntrack table quickly creeping up to 65k entries, at which point the kernel starts dropping packets.
So far the only two fixes I've found for this are:
- Up the conntrack_max limit and hash table size (currently at 4 million)
- Decrease the conntrack ESTABLISHED timeout from 5 days (yes, 5 DAYS!!!) to 5 hours.
I'm running with it like this for a week to see how it goes.
