SNMPTT V1.4 has sudenly stopped working - Help !

[vijilants]

System :Centos 6.4 NSTI RC1.4
2012R2.9
# rpm -qa | grep snmptt
snmptt-1.4-0.9.beta2.el6.noarch
#

Hi ,

I have noticed that SNMPTT v1.4 has suddenly stopped working a few months ago and since then no traps have been reported in NSTI.

Ive tried restating the snmptt service, which appears to be running, but no incoming traps are appearing in the snmptt.log file nor in the snmpttunknown.log file.

Traps are arriving on the NMS as I can see then on syslog.

Looks like the thing broke down the day I went to upgrade from 2012R2.2 to 2012R2.9. A yum update had to be done on that day due to failures when trying to upgrade. Not sure if the updates broke the SNMPTT process. Somewhere along the way it would appear that system was upgraded to snmptt-1.4-0.9.beta2.el6.noarch. Not sure how or why.

Desperately need to get this going

Can you please advise on how to get this going again ?

Thanks

[sreinhardt]

Couple questions, then some system commands for you to run. Is this an XI or NSTI system or both? You have posted in the XI forum, but stated that NSTI has stopped receiving traps. I highly doubt that an XI update would cease snmptt from working, considering that aside from an snmptt update it likely should not be touching anything regarding that. Anyway, if you could also run the following commands.

Code: Select all

 Versions installed

    rpm -qa | grep snmp

Looking for bins:

    ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
    ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
    ls -lva /usr/sbin | grep -i 'snmp\|addmib'

snmptt\trapd settings and user perms:

    grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
    grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
    grep -i 'nag' /etc/group
    grep -i 'snmp' /etc/group

Checking log and spool dirs:

    ll /var/log/snmptt/
    ll -d /var/log/snmptt/
    ll /var/spool/snmptt
    ll -d /var/spool/snmptt

[vijilants]

Hi,

The system runs both Nagios XI and NSTI. However when the upgrade took place I had some issues with it, and I had to do a "yum update" as advised on this forum and that allowed the upgrade. However, it also looks like at that point it would appear that system was upgraded to snmptt-1.4-0.9.beta2 as it was running 1.3 previously. My snmptt.ini file is showing v 1.3.

However it looks like another thread had a similar issue:

http://support.nagios.com/forum/viewtop ... =22&t=9369

If I do need to downgarde this, to v1.3 how do I go about it ?

here is the info you requested:

Versions installed

Code: Select all

# rpm -qa | grep snmp
net-snmp-libs-5.5-49.el6.i686
snmptt-1.4-0.9.beta2.el6.noarch
net-snmp-5.5-49.el6.i686
net-snmp-utils-5.5-49.el6.i686
php-snmp-5.3.3-27.el6_5.i686
net-snmp-perl-5.5-49.el6.i686

Looking for bins:

Code: Select all

# ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
# 

Code: Select all

#ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
# 

Code: Select all

# ls -lva /usr/sbin | grep -i 'snmp\|addmib'
-rwxr-xr-x   1 root root        25972 Nov 22  2013 snmpd
-rwxr-xr-x   1 root root        25992 Nov 22  2013 snmptrapd
-rwxr-xr-x   1 root root       177466 Oct 22  2012 snmptt
-rwxr-xr-x   1 root root         6493 Oct 22  2012 snmptthandler

snmptt\trapd settings and user perms:

Code: Select all

grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
mode = standalone
description_mode = 0
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt

Code: Select all

# grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An indication that the agent is in the process of being shut down. $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An indication that the agent has been restarted. $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An example notification, used to illustrate the $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "The SNMP trap that is generated when an alarm $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "The SNMP trap that is generated when an alarm $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A coldStart trap signifies that the SNMP entity, $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A warmStart trap signifies that the SNMP entity, $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An authenticationFailure trap signifies that the SNMP $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "This trap could in principle be sent when the agent start $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "This trap is sent when the agent terminates $*"
#

Code: Select all

# grep -i 'nag' /etc/group
nagios:x:500:nagios,apache
nagcmd:x:501:nagios,apache
vij:x:503:nagios,root

Code: Select all

#  grep -i 'snmp' /etc/group
snmptt:x:487:

Checking log and spool dirs:

Code: Select all

# ll /var/log/snmptt/
total 6136
-rw-rw-r-- 1 snmptt snmptt       0 Mar  2 03:23 snmptt.log
-rw-rw-r-- 1 snmptt snmptt  592795 Feb  9 02:19 snmptt.log-20140209
-rw-rw-r-- 1 snmptt snmptt  436067 Feb 16 02:21 snmptt.log-20140216
-rw-rw-r-- 1 snmptt snmptt  511027 Feb 22 23:18 snmptt.log-20140223
-rw-rw-r-- 1 snmptt snmptt   68499 Feb 24 17:16 snmptt.log-20140302
-rw-rw-r-- 1 snmptt snmptt    3588 Jun  2 15:36 snmpttsystem.log
-rw-rw-r-- 1 snmptt snmptt    1452 Jan 22 13:49 snmpttsystem.log-20140123
-rw-rw-r-- 1 snmptt snmptt     875 Feb 19 09:06 snmpttsystem.log-20140220
-rw-rw-r-- 1 snmptt snmptt     170 Feb 24 19:32 snmpttsystem.log-20140225
-rw-rw-r-- 1 snmptt snmptt    1368 Feb 27 14:06 snmpttsystem.log-20140302
-rw-rw-r-- 1 snmptt snmptt       0 Mar  2 03:23 snmpttunknown.log
-rw-rw-r-- 1 snmptt snmptt 2273174 Feb  9 03:34 snmpttunknown.log-20140209
-rw-rw-r-- 1 snmptt snmptt 2038921 Feb 15 21:51 snmpttunknown.log-20140216
-rw-rw-r-- 1 snmptt snmptt  292038 Feb 22 14:03 snmpttunknown.log-20140223
-rw-rw-r-- 1 snmptt snmptt     994 Feb 24 05:00 snmpttunknown.log-20140302

Code: Select all

# ll -d /var/log/snmptt/
drwxrwxr-x 2 snmptt snmptt 4096 Mar  2 03:23 /var/log/snmptt/ 

Code: Select all

# ll /var/spool/snmptt
total 0

Code: Select all

ll -d /var/spool/snmptt
drwxr-xr-x 2 snmptt snmptt 12288 Jan 18 18:44 /var/spool/snmptt

[sreinhardt]

That forum post is more than a year old. I would actually highly suggest staying with 1.4. However I would also suggest that you run the script referenced in this document: http://assets.nagios.com/downloads/nagi ... ios_XI.pdf

Once that is done. Most likely we just need to make sure the mysql information for snmptt is still being updated. It certainly seems that your traps are being reaped, but you have some user\group issues that are likely preventing snmptt from submitting to XI, and potentially due to the upgrade still some issues with mysql settings getting reverted.

[vijilants]

Thank you for your prompt replies and help.

Running the script worked !!!!! Well done !!

The NSTI Web page is now showing incoming traps and the snmptt.log is now showing incoming traps.....does that also mean that the XI Traps area will also work ?

Also for my own knowledge, can you please advise me on how from the above info that I gave you, you could see that Traps were being harvested ?

Many Thanks

[sreinhardt]

Running the script worked !!!!! Well done !!
The NSTI Web page is now showing incoming traps and the snmptt.log is now showing incoming traps.....does that also mean that the XI Traps area will also work ?

Thanks! yes that means that XI and NSTI traps should work, the only notable difference with XI is new devices\traps and making sure the configuration is correct. But if you have done that before I'm sure you can handle it for any new devices. Existing ones should start showing up from now on.

Also for my own knowledge, can you please advise me on how from the above info that I gave you, you could see that Traps were being harvested ?

Sure, in this particular case, I noted that you did not have any spooled traps, which is generally a big giveaway that they are not processing. I could not be 100% certain since your logs were somewhat old, but the permissions looked correct, so I had to assume that they might have been disabled in favor of NSTI. The other big part, was that all of your exec lines shown are the way they should be, so there were many less options as to what might be the issue. Just a matter of narrowing down the options!

[vijilants]

Thank you for your support !