Page 1 of 1

SNMP Trap processing stopped

Posted: Tue Nov 27, 2018 6:41 am
by tkoeferli
Hello,
we have a customer with many SNMPTRAPS (about 2 per sec).
Processing stuck every week. That means snmptt was blocked and /var/spool/snmp/ filled up.
I found that snmptraphandling.py never returned and thus blocked snmptt.
In snmptraphandling.py
Opening the FIFO with
output = open('/usr/local/nagios/var/rw/nagios.cmd', 'w')
COULD BLOCK!

I think this happens when Nagioxi does it's weekly stuff, and doesn'nt read the FIFO.
I also could reproduce if I did ApplyConfiguration via GUI.

My solution:
if os.path.exists('/usr/local/nagios/var/rw/nagios.cmd') and stat.S_ISFIFO(os.stat('/usr/local/nagios/var/rw/nagios.cmd').st_mode):
output = os.open('/usr/local/nagios/var/rw/nagios.cmd', os.O_WRONLY | os.O_NONBLOCK)
if service == 'PROCESS_HOST_CHECK_RESULT':
results = "[" + mytime + "] " + "PROCESS_HOST_CHECK_RESULT;" \
+ host + ";" + return_code + ";" + mondata_res + "\n"

else:
results = "[" + mytime + "] " + "PROCESS_SERVICE_CHECK_RESULT;" \
+ host + ";" + service + ";" \
+ return_code + ";" + mondata_res + "\n"
os.write(output,results)

Could someone confirm? And check my solution? :roll:

Regards,
Thomas

Re: SNMP Trap processing stopped

Posted: Tue Nov 27, 2018 4:16 pm
by tgriep
The snmptraphandling.py script was updated a while back to address the issue you are having.
I will attach it here in case you want to view it but the changes you did to the script match the attached script so it should work for you and not lock the snmptt daemon.

Re: SNMP Trap processing stopped

Posted: Wed Nov 28, 2018 1:48 am
by tkoeferli
Thank you.

We allready had this new script.
But between this lines
if os.path.exists('/usr/local/nagios/var/rw/nagios.cmd') and stat.S_ISFIFO(os.stat('/usr/local/nagios/var/rw/nagios.cmd').st_mode):
output = open('/usr/local/nagios/var/rw/nagios.cmd', 'w')

If you have many traps, it could happen that the FIFO is closed/has no reader, and open('/usr/local/nagios/var/rw/nagios.cmd', 'w') will block.

All this according Murphys laws!

Regards,
Thomas

Re: SNMP Trap processing stopped

Posted: Wed Nov 28, 2018 9:42 am
by tgriep
OK, I see the change. Need to get my eye's checked. Thanks for the info on the changes to the script.

Re: SNMP Trap processing stopped

Posted: Wed Nov 28, 2018 9:50 am
by tkoeferli
I've seen that the error was caused during the weekly scheduled backup at Sunday 00:59.

With my modification it's working now.

Thank you!

Re: SNMP Trap processing stopped

Posted: Thu Nov 29, 2018 9:23 am
by scottwilkerson
tkoeferli wrote:I've seen that the error was caused during the weekly scheduled backup at Sunday 00:59.

With my modification it's working now.

Thank you!
Glad to hear it is working!

Locking thread