SNMP Trap processing stopped

[tkoeferli]

Hello,
we have a customer with many SNMPTRAPS (about 2 per sec).
Processing stuck every week. That means snmptt was blocked and /var/spool/snmp/ filled up.
I found that snmptraphandling.py never returned and thus blocked snmptt.
In snmptraphandling.py
Opening the FIFO with
output = open('/usr/local/nagios/var/rw/nagios.cmd', 'w')
COULD BLOCK!

I think this happens when Nagioxi does it's weekly stuff, and doesn'nt read the FIFO.
I also could reproduce if I did ApplyConfiguration via GUI.

My solution:
if os.path.exists('/usr/local/nagios/var/rw/nagios.cmd') and stat.S_ISFIFO(os.stat('/usr/local/nagios/var/rw/nagios.cmd').st_mode):
output = os.open('/usr/local/nagios/var/rw/nagios.cmd', os.O_WRONLY | os.O_NONBLOCK)
if service == 'PROCESS_HOST_CHECK_RESULT':
results = "[" + mytime + "] " + "PROCESS_HOST_CHECK_RESULT;" \
+ host + ";" + return_code + ";" + mondata_res + "\n"

else:
results = "[" + mytime + "] " + "PROCESS_SERVICE_CHECK_RESULT;" \
+ host + ";" + service + ";" \
+ return_code + ";" + mondata_res + "\n"
os.write(output,results)

Could someone confirm? And check my solution?

Regards,
Thomas

[tgriep]

The snmptraphandling.py script was updated a while back to address the issue you are having.
I will attach it here in case you want to view it but the changes you did to the script match the attached script so it should work for you and not lock the snmptt daemon.

[tkoeferli]

Thank you.

We allready had this new script.
But between this lines
if os.path.exists('/usr/local/nagios/var/rw/nagios.cmd') and stat.S_ISFIFO(os.stat('/usr/local/nagios/var/rw/nagios.cmd').st_mode):
output = open('/usr/local/nagios/var/rw/nagios.cmd', 'w')

If you have many traps, it could happen that the FIFO is closed/has no reader, and open('/usr/local/nagios/var/rw/nagios.cmd', 'w') will block.

All this according Murphys laws!

Regards,
Thomas

[tgriep]

OK, I see the change. Need to get my eye's checked. Thanks for the info on the changes to the script.

[tkoeferli]

I've seen that the error was caused during the weekly scheduled backup at Sunday 00:59.

With my modification it's working now.

Thank you!

[scottwilkerson]

I've seen that the error was caused during the weekly scheduled backup at Sunday 00:59.

With my modification it's working now.

Thank you!

Glad to hear it is working!

Locking thread