SNMPTT not deleting traps, mysql errors in log

[globalive.nagios]

Quick description of the service:
- Remote host runs rsyslog with the omsnmp module
- omsnmp sends specific rsyslog messages to the Nagios host
- Nagios runs snmptrapd to catch the traps, deals with them using snmptthandler
- Next is snmptt, which decodes the trap and submits to Nagios
- Nagios is running 'dummy' services under which the alerts are sent out

So far, everything appears functional except that the traps are not being deleted from /var/spool/snmptt.

Any ideas? I figure there is something bigger going on because of the below errors. I have tried rebooting the Nagios host just to be sure.

The error in /var/log/messages follows:

Code: Select all

Jan 13 14:53:35 tor-nagios-02 nagios: SERVICE ALERT: astappsrv6;rsyslog_message_WARNING;WARNING;HARD;1;<180>Jan 13 14:52:05 astappsrv6 asterisk[27048]: WARNING[27067]: chan_sip.c:12021 in handle_response_invite: Received response: "Forbidden" from '"1777" <sip:[email protected]>:tag=as0d9fa270'
Jan 13 14:53:35 tor-nagios-02 nagios: SERVICE NOTIFICATION: trotter;astappsrv6;rsyslog_message_WARNING;WARNING;xi_service_notification_handler;180Jan 13 14:52:05 astappsrv6 asterisk[27048]: WARNING[27067]: chan_sip.c:12021 in handle_response_invite: Received response: Forbidden from 1777 sip:[email protected]:tag=as0d9fa270
Jan 13 14:53:35 tor-nagios-02 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='502', status_update_time=FROM_UNIXTIME(1326484415), output='<180>Jan 13 14:52:05 astappsrv6 asterisk\[27048\]: WARNING\[27067\]: chan_sip\.c:12021 in handle_response_invite: Received response: \"Forbidden\" from \'\"1777\" <sip:1777@204\.11\.120\.83>:tag=as0d9fa270\'', long_output='', perfdata='', current_state='1', has_been_checked='1', should_be_scheduled='0', current_check_attempt='1', max_check_attempts='1', last_check=FROM_UNIXTIME(1326484407), next_check=FROM_UNIXTIME(1326484386), check_type='1', last_state_change=FROM_UNIXTIME(1326484326), last_hard_state_change=FROM_UNIXTIME(1326484326), last_hard_state='1', last_time_ok=FROM_UNIXTIME(0), last_time_warning=FROM_UNIXTIME(1326484407), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(1326484415), next_notification=FROM_UNIXTIME(3218644415), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='17', passive_checks_enabled='1', active_checks_enabled='0', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='4.144740', latency='0.736080', execution_time='0.000000', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check-host-alive!!!!!!!!', normal_check_interval='1.000000', retry_check_interval='1.000000', check_timeperiod_object_id='67' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='502', status_update_time=FROM_UNIXTIME(1326484415), output='<180>Jan 13 14:52:05 astappsrv6 asterisk\[27048\]: WARNING\[27067\]: chan_sip\.c:12021 in handle_response_invite: Received response: \"Forbidden\" from \'\"1777\" <sip:1777@204\.11\.120\.83>:tag=as0d9fa270\'', long_output='', perfdata='', current_sta
Jan 13 14:53:35 tor-nagios-02 ndo2db: mysql_error: 'Column 'next_notification' cannot be null'
Jan 13 14:53:35 tor-nagios-02 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='502', status_update_time=FROM_UNIXTIME(1326484415), output='<180>Jan 13 14:52:05 astappsrv6 asterisk\[27048\]: WARNING\[27067\]: chan_sip\.c:12021 in handle_response_invite: Received response: \"Forbidden\" from \'\"1777\" <sip:1777@204\.11\.120\.83>:tag=as0d9fa270\'', long_output='', perfdata='', current_state='1', has_been_checked='1', should_be_scheduled='0', current_check_attempt='1', max_check_attempts='1', last_check=FROM_UNIXTIME(1326484407), next_check=FROM_UNIXTIME(1326484386), check_type='1', last_state_change=FROM_UNIXTIME(1326484326), last_hard_state_change=FROM_UNIXTIME(1326484326), last_hard_state='1', last_time_ok=FROM_UNIXTIME(0), last_time_warning=FROM_UNIXTIME(1326484407), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(1326484415), next_notification=FROM_UNIXTIME(3218644415), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='17', passive_checks_enabled='1', active_checks_enabled='0', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='4.144740', latency='0.736080', execution_time='0.000000', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check-host-alive!!!!!!!!', normal_check_interval='1.000000', retry_check_interval='1.000000', check_timeperiod_object_id='67' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='502', status_update_time=FROM_UNIXTIME(1326484415), output='<180>Jan 13 14:52:05 astappsrv6 asterisk\[27048\]: WARNING\[27067\]: chan_sip\.c:12021 in handle_response_invite: Received response: \"Forbidden\" from \'\"1777\" <sip:1777@204\.11\.120\.83>:tag=as0d9fa270\'', long_output='', perfdata='', current_sta
Jan 13 14:53:35 tor-nagios-02 ndo2db: mysql_error: 'Column 'next_notification' cannot be null'

[scottwilkerson]

the next_notification is being set to a time too fare into the future. 3218644415 = Tue, 29 Dec 2071 19:53:35 GMT and the mysql function FROM_UNIXTIME won't accept it and returns NULL

[globalive.nagios]

Ah, that's interesting. For some reason we had the alert setting interval set at a really high number, so maybe that was changing the SNMP trap date...

I've set it back to default and pointed the servers at the new Nagios host, so I will report back. Thanks.

[globalive.nagios]

Ok, figured it out! (the above wasn't it, btw)

Here's what we're getting in /var/log/messages

Code: Select all

Jan 16 09:07:51 tor-nagios-02 snmptt-sys[504]: Can not open log file /var/log/snmptt.log
Jan 16 09:07:51 tor-nagios-02 snmptt-sys[504]: Unable to delete trap file #snmptt-trap-1326722864817018 from spool dir
Jan 16 09:07:56 tor-nagios-02 snmptt-sys[504]: Can not open log file /var/log/snmptt.log
Jan 16 09:07:56 tor-nagios-02 snmptt-sys[504]: Unable to delete trap file #snmptt-trap-1326722864817018 from spool dir

So I said, why would snmptt-sys not be able to open that log file, or delete the trap from the spool?

Permissions, of course! The /var/spool/snmptt directory was still owned by root (should be owned by snmptt:nagios), and needed different access rights.

Anyways, that's the theory. Will update when another trap comes through.

[globalive.nagios]

Yep, that did it! Just permissions.

Last issue to deal with is clearing the alerts back to green. Will be going off of this to start: http://snmptt.sourceforge.net/docs/snmptt.shtml