One of my more recent booboos involved an perl script hooked to an SNMP OID, the Net-SNMP server and an unnamed SNMP Get client.
It's one of those "this only happens once every so often" booboos. It appeared as the SNMP client was generating double as much events as there were data collections on that specific OID.
For a minute I thought it was caused by the SNMP client. After lots of swearing ([1] because I couldn't find the reason why this was happening ; [2] because I occasionally forgot to clear the generated events for a few days and I spent lots of time clearing them), I figured it out.
It turned out the timeout value is usually set to 0,7 seconds. The script itself sometimes took more than 0,7 seconds to generate the required result and retry count was set to 2. By the time the second GET request was sent, the first response was returned and the second response was dropped automagically.
It took some sniffing and pondering before I found this out.. /me happy!
