recieving SNMP traps

[benhank]

Ok fellas I have tried to sort this one out myself, with my typical results : epic fail...
The initial issue, recorded in my award winning (#1 has no clue sys admin) post:

Code: Select all

http://support.nagios.com/forum/viewtopic.php?f=16&t=11418

have all been resolved. I had to reinstall the net-snmp and all the other stuff.

then I followed the instructions detailed in

Code: Select all

Nagios XI – Integrating SNMP Traps

AND the NSTI install document.
Every thing seems to have set up correctly as I had zero issues with installing my mibs, and restarting the snmptt snmptrapd services. The issue is i set up 2 sending servers, on is a nagios box and the other a windows box.

Prior to my reinstall the snmpunknown.log and the snmptt.log both had recorded data.
Now, I get nuthin' the log files aren't even created.

I disabled my iptables just in case,.. but my server just somehow isnt receiving the traps like it used to. I know I must be missing a step, I just cant figure out what it is....

[abrist]

First question:
I presume that snmptt is running?

Code: Select all

service snmptt status

Are the traps coming in as unconfigured objects the XI interface?
Have you checked to make sure that you nagios server did not get a different ip lease when you reinstalled the server? if so, did you change the remote devices to send to the new ip?

[benhank]

Code: Select all

[root@LkennagiosT01 ~]# service snmptt status
snmptt (pid  2919) is running...
[root@LkennagiosT01 ~]#

the IP is still the same, and the service check is not in unconfigured objects.
here is the service setup for the server:

Code: Select all

define service {
	host_name			wkensimp02.healthone.org
	service_description		Passive Service
	use				xiwizard_passive_service
	max_check_attempts		1
	check_interval			1
	retry_interval			1
	check_period			xi_timeperiod_24x7
	notification_interval		60
	notification_period		xi_timeperiod_24x7
	contacts			nagiosadmin
	stalking_options		n
	_xiwizard			passivecheck
	register			1
	}	

define service {
	host_name			wkensimp02.healthone.org
	service_description		SNMP Traps
	use				xiwizard_snmptrap_service
	max_check_attempts		1
	check_interval			1
	retry_interval			1
	check_period			xi_timeperiod_24x7
	notification_interval		1
	notification_period		xi_timeperiod_24x7
	contacts			nagiosadmin
	icon_image			snmptrap.png
	_xiwizard			snmp_trap
	register			1
	}	

define service {
	host_name			wkensimp02.healthone.org
	service_description		snmptrapd
	use				xiwizard_passive_service
	max_check_attempts		1
	check_interval			1
	retry_interval			1
	check_period			xi_timeperiod_24x7
	notification_interval		60
	notification_period		xi_timeperiod_24x7
	contacts			nagiosadmin
	stalking_options		n
	_xiwizard			passivecheck
	register			1
	}	

###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################

and the other:

Code: Select all

define service {
	host_name			lkennagiosp03.healthone.org
	service_description		Passive Service
	use				xiwizard_passive_service
	max_check_attempts		1
	check_interval			1
	retry_interval			1
	check_period			xi_timeperiod_24x7
	notification_interval		60
	notification_period		xi_timeperiod_24x7
	contacts			nagiosadmin
	stalking_options		n
	_xiwizard			passivecheck
	register			1
	}	

define service {
	host_name			lkennagiosp03.healthone.org
	service_description		SNMP Traps
	use				xiwizard_snmptrap_service
	max_check_attempts		1
	check_interval			1
	retry_interval			1
	check_period			xi_timeperiod_24x7
	notification_interval		1
	notification_period		xi_timeperiod_24x7
	contacts			nagiosadmin
	icon_image			snmptrap.png
	_xiwizard			snmp_trap
	register			1
	}	

define service {
	host_name			lkennagiosp03.healthone.org
	service_description		snmptrapd
	use				xiwizard_passive_service
	max_check_attempts		1
	check_interval			1
	retry_interval			1
	check_period			xi_timeperiod_24x7
	notification_interval		60
	notification_period		xi_timeperiod_24x7
	contacts			nagiosadmin
	stalking_options		n
	_xiwizard			passivecheck
	register			1
	}	

define service {
	host_name			lkennagiosp03.healthone.org
	service_description		snmptt
	use				xiwizard_passive_service
	max_check_attempts		1
	check_interval			1
	retry_interval			1
	check_period			xi_timeperiod_24x7
	notification_interval		60
	notification_period		xi_timeperiod_24x7
	contacts			nagiosadmin
	stalking_options		n
	_xiwizard			passivecheck
	register			1
	}	

###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################

[sreinhardt]

You should definitely be able to see logging in both flat files and the database for nsti. As a check, you are not seeing them via nsti either correct? Another point, is that snmp traps rely on three services, snmpd, snmptt, and snmptrapd. If you have imported mibs since this stopped working, lets make sure all are working correctly.

Code: Select all

service snmpd restart
service snmptt restart
service snmptrapd restart

I always restart them in this order, that is not to say that it makes a huge difference. Two other things we can check, is to nmap from another device and to tcp dump on the nagios server.

Code: Select all

nmap -sU -p 162 [nagios server IP]
tcpdump -i eth0 dst port 162

[benhank]

I am not using nsti, i am using the snmptrap wiz and the passive check wiz. however:

Son of a Biskit! i did what you said and everything restarted ok:

Code: Select all

Last login: Mon Aug 19 15:32:47 2013 from 172.26.68.146
[root@LkennagiosT01 ~]# service snmpd restart
Stopping snmpd:                                            [  OK  ]
Starting snmpd:                                            [  OK  ]
[root@LkennagiosT01 ~]# service snmptt restart
Stopping snmptt:                                           [  OK  ]
Starting snmptt:                                           [  OK  ]
[root@LkennagiosT01 ~]# service snmptrapd restart
Stopping snmptrapd:                                        [  OK  ]
Starting snmptrapd:                                        [  OK  ]
[root@LkennagiosT01 ~]#

but still no logs =(

Then I noticed on my receiving server nagios displays the following on localhost:

Code: Select all

localhost  HTTP
Warning 	12d 23h 49m 28s 	4/4 	2013-08-20 11:17:50 	HTTP WARNING: HTTP/1.1 403 Forbidden

on a whim I tailed the error log and got this:

Code: Select all

[Tue Aug 20 11:13:41 2013] [error] [client 172.22.3.18] Directory index forbidden by Options directive: /usr/local/nagiosti/nsti/

I'd tried to install NSTI and ran into the django error as recorded here:

Code: Select all

http://support.nagios.com/forum/viewtopic.php?f=22&t=11554

Any suggestions bro?
is it possible to uninstall nsti? or is there another way to fix this?

[sreinhardt]

You can certainly remove the /usr/local/nagiosti or /usr/local/nsti, depending on which version was installed. Otherwise it truely doesn't matter much as long as the daemon isn't running. Could you tar up the /etc/snmp folder and all contents, and pm me. It likely contains some private info that I wouldn't want widely posted.

[benhank]

sure !

[sreinhardt]

Couple changes that need to be made:

Code: Select all

snmptt.ini:
mode = standalone TO mode = daemon
Might also turn on debugging to help
DEBUGGING = 1
DEBUGGING_FILE = /var/log/snmptt/snmptt-debug.log

Of course, restart all three snmp services after these changes.

Also lets check permissions on a few things:

Code: Select all

ll /var/spool/snmptt/
ll /etc/snmp/

[benhank]

ok so there was a difference this time

Code: Select all

Last login: Tue Aug 20 14:04:06 2013 from 172.26.68.146
[root@LkennagiosT01 ~]# service snmpd restart
Stopping snmpd:                                            [  OK  ]
Starting snmpd:                                            [  OK  ]
[root@LkennagiosT01 ~]# service snmptt restart
Stopping snmptt:                                           [  OK  ]
Starting snmptt: PID file: /var/run/snmptt.pid   <<-----this is different than before
                                                           [  OK  ]
[root@LkennagiosT01 ~]# service snmptrapd restart
Stopping snmptrapd:                                        [  OK  ]
Starting snmptrapd:                                        [  OK  ]
[root@LkennagiosT01 ~]#

:
and

Code: Select all

ll /var/spool/snmptt/

returned (a hundred or so, I just took a few.

Code: Select all

-rw-r--r-- 1 root root 304 Aug 20 12:19 #snmptt-trap-1377015546835887
-rw-r--r-- 1 root root 828 Aug 20 13:01 #snmptt-trap-1377018082888443
-rw-r--r-- 1 root root 306 Aug 20 14:01 #snmptt-trap-1377021679597269
[root@LkennagiosT01 ~]#

and finally:

Code: Select all

[root@LkennagiosT01 ~]# ll /etc/snmp/
total 2836
-rw-r--r--  1 root   root     18861 Aug 12 10:46 snmpd.conf
-rw-r--r--  1 root   root        74 Aug 13 16:06 snmptrapd.conf
-rw-r--r--  1 root   root        67 Aug  7 12:38 snmptrapd.conf.rpmsave
-rw-r--r--  1 root   root      3909 Aug 19 12:45 snmptt.conf
-rw-r--r--. 1 root   root      2303 Mar 24  2003 snmptt.conf.rpmnew
-rw-r--r--  1 root   root   1216001 Aug  7 12:37 snmptt.conf.rpmorig
-rw-r--r--. 1 root   root   1583817 Aug  6 13:16 snmptt.conf.rpmsave
-rw-rw-r--  1 apache nagios   26025 Aug 20 14:09 snmptt.ini
-rw-rw-r--. 1 apache nagios   26690 Aug  6 13:16 snmptt.ini.rpmsave

[sreinhardt]

Future trap spools should be created as snmp user, however for now this is also going to be a part of the issue. (don't worry, snmp is certainly anything but "simple") Let's try changing permissions and see if that changes anything for getting things rolling. Fair warning, you may have to simply delete those current spools if this doesn't work.

Code: Select all

chmod -R root.snmptt /var/spool/snmptt/

restart services, agian... yes this has to be done just about any time anything changes with snmp stuff.
Check logs

Code: Select all

tail -f /var/log/snmp/snmptt.log
tail -f /var/log/snmp/snmpttunknown.log

Also since I forgot to add it, lets check your log permissions just to be sure:

Code: Select all

ll /var/log/snmptt/