Page 1 of 6
recieving SNMP traps
Posted: Mon Aug 19, 2013 2:03 pm
by benhank
Ok fellas I have tried to sort this one out myself, with my typical results : epic fail...
The initial issue, recorded in my award winning (#1 has no clue sys admin) post:
Code: Select all
http://support.nagios.com/forum/viewtopic.php?f=16&t=11418
have all been resolved. I had to reinstall the net-snmp and all the other stuff.
then I followed the instructions detailed in
Code: Select all
Nagios XI – Integrating SNMP Traps
AND the NSTI install document.
Every thing seems to have set up correctly as I had zero issues with installing my mibs, and restarting the snmptt snmptrapd services. The issue is i set up 2 sending servers, on is a nagios box and the other a windows box.
Prior to my reinstall the snmpunknown.log and the snmptt.log both had recorded data.
Now, I get nuthin' the log files aren't even created.
I disabled my iptables just in case,.. but my server just somehow isnt receiving the traps like it used to. I know I must be missing a step, I just cant figure out what it is....
Re: recieving SNMP traps
Posted: Mon Aug 19, 2013 2:21 pm
by abrist
First question:
I presume that snmptt is running?
Are the traps coming in as unconfigured objects the XI interface?
Have you checked to make sure that you nagios server did not get a different ip lease when you reinstalled the server? if so, did you change the remote devices to send to the new ip?
Re: recieving SNMP traps
Posted: Mon Aug 19, 2013 2:38 pm
by benhank
Code: Select all
[root@LkennagiosT01 ~]# service snmptt status
snmptt (pid 2919) is running...
[root@LkennagiosT01 ~]#
the IP is still the same, and the service check is not in unconfigured objects.
here is the service setup for the server:
Code: Select all
define service {
host_name wkensimp02.healthone.org
service_description Passive Service
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts nagiosadmin
stalking_options n
_xiwizard passivecheck
register 1
}
define service {
host_name wkensimp02.healthone.org
service_description SNMP Traps
use xiwizard_snmptrap_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 1
notification_period xi_timeperiod_24x7
contacts nagiosadmin
icon_image snmptrap.png
_xiwizard snmp_trap
register 1
}
define service {
host_name wkensimp02.healthone.org
service_description snmptrapd
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts nagiosadmin
stalking_options n
_xiwizard passivecheck
register 1
}
###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################
and the other:
Code: Select all
define service {
host_name lkennagiosp03.healthone.org
service_description Passive Service
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts nagiosadmin
stalking_options n
_xiwizard passivecheck
register 1
}
define service {
host_name lkennagiosp03.healthone.org
service_description SNMP Traps
use xiwizard_snmptrap_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 1
notification_period xi_timeperiod_24x7
contacts nagiosadmin
icon_image snmptrap.png
_xiwizard snmp_trap
register 1
}
define service {
host_name lkennagiosp03.healthone.org
service_description snmptrapd
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts nagiosadmin
stalking_options n
_xiwizard passivecheck
register 1
}
define service {
host_name lkennagiosp03.healthone.org
service_description snmptt
use xiwizard_passive_service
max_check_attempts 1
check_interval 1
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts nagiosadmin
stalking_options n
_xiwizard passivecheck
register 1
}
###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################
Re: recieving SNMP traps
Posted: Mon Aug 19, 2013 4:04 pm
by sreinhardt
You should definitely be able to see logging in both flat files and the database for nsti. As a check, you are not seeing them via nsti either correct? Another point, is that snmp traps rely on three services, snmpd, snmptt, and snmptrapd. If you have imported mibs since this stopped working, lets make sure all are working correctly.
Code: Select all
service snmpd restart
service snmptt restart
service snmptrapd restart
I always restart them in this order, that is not to say that it makes a huge difference. Two other things we can check, is to nmap from another device and to tcp dump on the nagios server.
Code: Select all
nmap -sU -p 162 [nagios server IP]
tcpdump -i eth0 dst port 162
Re: recieving SNMP traps
Posted: Tue Aug 20, 2013 10:28 am
by benhank
I am not using nsti, i am using the snmptrap wiz and the passive check wiz. however:
Son of a Biskit! i did what you said and everything restarted ok:
Code: Select all
Last login: Mon Aug 19 15:32:47 2013 from 172.26.68.146
[root@LkennagiosT01 ~]# service snmpd restart
Stopping snmpd: [ OK ]
Starting snmpd: [ OK ]
[root@LkennagiosT01 ~]# service snmptt restart
Stopping snmptt: [ OK ]
Starting snmptt: [ OK ]
[root@LkennagiosT01 ~]# service snmptrapd restart
Stopping snmptrapd: [ OK ]
Starting snmptrapd: [ OK ]
[root@LkennagiosT01 ~]#
but still no logs =(
Then I noticed on my receiving server nagios displays the following on localhost:
Code: Select all
localhost HTTP
Warning 12d 23h 49m 28s 4/4 2013-08-20 11:17:50 HTTP WARNING: HTTP/1.1 403 Forbidden
on a whim I tailed the error log and got this:
Code: Select all
[Tue Aug 20 11:13:41 2013] [error] [client 172.22.3.18] Directory index forbidden by Options directive: /usr/local/nagiosti/nsti/
I'd tried to install NSTI and ran into the django error as recorded here:
Code: Select all
http://support.nagios.com/forum/viewtopic.php?f=22&t=11554
Any suggestions bro?
is it possible to uninstall nsti? or is there another way to fix this?
Re: recieving SNMP traps
Posted: Tue Aug 20, 2013 10:53 am
by sreinhardt
You can certainly remove the /usr/local/nagiosti or /usr/local/nsti, depending on which version was installed. Otherwise it truely doesn't matter much as long as the daemon isn't running. Could you tar up the /etc/snmp folder and all contents, and pm me. It likely contains some private info that I wouldn't want widely posted.
Re: recieving SNMP traps
Posted: Tue Aug 20, 2013 11:06 am
by benhank
sure !
Re: recieving SNMP traps
Posted: Tue Aug 20, 2013 12:57 pm
by sreinhardt
Couple changes that need to be made:
Code: Select all
snmptt.ini:
mode = standalone TO mode = daemon
Might also turn on debugging to help
DEBUGGING = 1
DEBUGGING_FILE = /var/log/snmptt/snmptt-debug.log
Of course, restart all three snmp services after these changes.
Also lets check permissions on a few things:
Code: Select all
ll /var/spool/snmptt/
ll /etc/snmp/
Re: recieving SNMP traps
Posted: Tue Aug 20, 2013 1:16 pm
by benhank
ok so there was a difference this time
Code: Select all
Last login: Tue Aug 20 14:04:06 2013 from 172.26.68.146
[root@LkennagiosT01 ~]# service snmpd restart
Stopping snmpd: [ OK ]
Starting snmpd: [ OK ]
[root@LkennagiosT01 ~]# service snmptt restart
Stopping snmptt: [ OK ]
Starting snmptt: PID file: /var/run/snmptt.pid <<-----this is different than before
[ OK ]
[root@LkennagiosT01 ~]# service snmptrapd restart
Stopping snmptrapd: [ OK ]
Starting snmptrapd: [ OK ]
[root@LkennagiosT01 ~]#
:
and
returned (a hundred or so, I just took a few.
Code: Select all
-rw-r--r-- 1 root root 304 Aug 20 12:19 #snmptt-trap-1377015546835887
-rw-r--r-- 1 root root 828 Aug 20 13:01 #snmptt-trap-1377018082888443
-rw-r--r-- 1 root root 306 Aug 20 14:01 #snmptt-trap-1377021679597269
[root@LkennagiosT01 ~]#
and finally:
Code: Select all
[root@LkennagiosT01 ~]# ll /etc/snmp/
total 2836
-rw-r--r-- 1 root root 18861 Aug 12 10:46 snmpd.conf
-rw-r--r-- 1 root root 74 Aug 13 16:06 snmptrapd.conf
-rw-r--r-- 1 root root 67 Aug 7 12:38 snmptrapd.conf.rpmsave
-rw-r--r-- 1 root root 3909 Aug 19 12:45 snmptt.conf
-rw-r--r--. 1 root root 2303 Mar 24 2003 snmptt.conf.rpmnew
-rw-r--r-- 1 root root 1216001 Aug 7 12:37 snmptt.conf.rpmorig
-rw-r--r--. 1 root root 1583817 Aug 6 13:16 snmptt.conf.rpmsave
-rw-rw-r-- 1 apache nagios 26025 Aug 20 14:09 snmptt.ini
-rw-rw-r--. 1 apache nagios 26690 Aug 6 13:16 snmptt.ini.rpmsave
Re: recieving SNMP traps
Posted: Tue Aug 20, 2013 1:25 pm
by sreinhardt
Future trap spools should be created as snmp user, however for now this is also going to be a part of the issue. (don't worry, snmp is certainly anything but "simple") Let's try changing permissions and see if that changes anything for getting things rolling. Fair warning, you may have to simply delete those current spools if this doesn't work.
Code: Select all
chmod -R root.snmptt /var/spool/snmptt/
restart services, agian... yes this has to be done just about any time anything changes with snmp stuff.
Check logs
Code: Select all
tail -f /var/log/snmp/snmptt.log
tail -f /var/log/snmp/snmpttunknown.log
Also since I forgot to add it, lets check your log permissions just to be sure: