Traps & eventman.log

[oslec]

Hi,
I have problems to send snmptraps, I don't see any event in eventman.log
Nagios 5.7.4 - Centos 7.8

[root@mxtulm01nagiosxi var]# tail -f eventman.log
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
..................

In apache logs, see

[Wed Oct 21 16:24:25.182169 2020] [:error] [pid 5697] [client 10.32.1.175:57992] PHP Notice: Undefined index: v3_auth_proto in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 338, referer: https://10.150.57.40/nagiosxi/admin/com ... trapsender
[Wed Oct 21 16:24:25.182172 2020] [:error] [pid 5697] [client 10.32.1.175:57992] PHP Notice: Undefined index: v3_auth_proto in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 339, referer: https://10.150.57.40/nagiosxi/admin/com ... trapsender
[Wed Oct 21 16:24:25.182176 2020] [:error] [pid 5697] [client 10.32.1.175:57992] PHP Notice: Undefined index: v3_priv_proto in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 344, referer: https://10.150.57.40/nagiosxi/admin/com ... trapsender
[Wed Oct 21 16:24:25.182180 2020] [:error] [pid 5697] [client 10.32.1.175:57992] PHP Notice: Undefined index: v3_priv_proto in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 345, referer: https://10.150.57.40/nagiosxi/admin/com ... trapsender
[Wed Oct 21 16:24:25.182183 2020] [:error] [pid 5697] [client 10.32.1.175:57992] PHP Notice: Undefined index: v3_priv_proto in /usr/local/nagiosxi/html/includes/components/snmptrapsender/snmptrapsender.inc.php on line 346, referer: https://10.150.57.40/nagiosxi/admin/com ... trapsender

Regards,

[tgriep]

Go to the Admin > Manage Components menu and edit the SNMP Trap sender.
It is still enabled?
Is the host that you are sending the traps to receiving the traps?

Try this, disable the trap sender and Apply the Settings, re-enable, verify that the settings are still valid, and see if they start to log.
Also, enable the Debug logging and check to see if they are logged there.

[oslec]

Hi,

Thank you,

Steps Disable Module (TrapSender) - Apply Settings - Re-enable and no events:

[root@mxtulm01nagiosxi var]# tail -f event_handler.log
LOCKFILE '/usr/local/nagiosxi/var/event_handler.lock' CREATED
DELETED LOCKFILE '/usr/local/nagiosxi/var/event_handler.lock'
EVENT HANDLER EXITING
LOCKFILE '/usr/local/nagiosxi/var/event_handler.lock' CREATED
DELETED LOCKFILE '/usr/local/nagiosxi/var/event_handler.lock'
EVENT HANDLER EXITING
LOCKFILE '/usr/local/nagiosxi/var/event_handler.lock' CREATED
DELETED LOCKFILE '/usr/local/nagiosxi/var/event_handler.lock'
EVENT HANDLER EXITING
LOCKFILE '/usr/local/nagiosxi/var/event_handler.lock' CREATED
^X^C
[root@mxtulm01nagiosxi var]# tail -f eventman.log
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS

Regards,

[tgriep]

Can you run the following command as root and post the output so I can check the settings for the SNMPTrap sender component?

Code: Select all

echo ' select * from xi_options;' |mysql -t -u root -pnagiosxi nagiosxi |grep snmp

Did you enable the Debug logging for the SNMPTrap sender and check to see if they are logged in this file?

Code: Select all

/usr/local/nagiosxi/var/components/snmptrapsender.log

Is the device that is setup to receive the traps receiving them?

[oslec]

Hi,

The module is enable but I can't see any traps.
❯ ssh -l root 10.150.57.40
[email protected]'s password:
Last login: Thu Oct 22 18:11:23 2020 from 10.32.1.148
-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
[root@mxtulm01nagiosxi ~]# echo ' select * from xi_options;' |mysql -t -u root -pnagiosxi nagiosxi |grep snmp
| 215 | snmptrapsender_component_options | a:4:{s:7:"enabled";i:1;s:10:"trap_hosts";a:1:{i:0;a:7:{s:7:"address";s:12:"10.150.10.73";s:4:"port";s:3:"162";s:11:"hoststateid";s:1:"0";s:14:"servicestateid";s:1:"0";s:9:"statetype";s:4:"BOTH";s:11:"snmpversion";s:2:"2c";s:9:"community";s:6:"public";}}s:5:"debug";s:2:"on";s:8:"log_file";s:53:"/usr/local/nagiosxi/var/components/snmptrapsender.log";} |
[root@mxtulm01nagiosxi ~]# tail -f /usr/local/nagiosxi/var/components/snmptrapsender.log
tail: cannot open '/usr/local/nagiosxi/var/components/snmptrapsender.log' for reading: No such file or directory
tail: no files remaining
[root@mxtulm01nagiosxi ~]#

[oslec]

Hi,

Any news ?

Regards!

[tgriep]

I see some permission errors on the system so can you run the following as root and post the output?

Code: Select all

chage -l nagios
grep nag /etc/group
ls -l /usr/local/nagiosxi
ls -l /usr/local/nagiosxi/var
ls -l /usr/local/nagiosxi/var/components

Get the following files from the server and attach them to the post.

Code: Select all

/etc/php.ini
/usr/local/nagios/var/nagios.log

Thanks.

[oslec]

Hi,

Thank you.
If you need remote access to the Nagios XI..let me know.

Is strange..I have traps in the morning...

"[root@mxtulm01nagiosxi ~]# tail -f /usr/local/nagiosxi/var/components/snmptrapsender.log
2020-10-26 00:55:42 - /usr/bin/snmptrap -v 2c -c public 10.150.10.73:162 '' NAGIOS-NOTIFY-MIB::nHostEvent nHostname s "mxtijm01rtdclfisp01" nHostStateID i 1 nHostOutput s "CRITICAL - 10.225.236.136: Net unreachable @ 192.168.30.1. rta nan, lost 100%" nHostAlias s "mxtijm01rtdclfisp01"
2020-10-26 00:56:30 - /usr/bin/snmptrap -v 2c -c public 10.150.10.73:162 '' NAGIOS-NOTIFY-MIB::nHostEvent nHostname s "mxtijm01rtdclfisp02" nHostStateID i 1 nHostOutput s "CRITICAL - 10.225.236.137: rta nan, lost 100%" nHostAlias s "mxtijm01rtdclfisp02"
2020-10-26 00:57:08 - /usr/bin/snmptrap -v 2c -c public 10.150.10.73:162 '' NAGIOS-NOTIFY-MIB::nSvcEvent nSvcHostname s "chi-r9-ea" nSvcDesc s "Conectividad" nSvcStateID i 2 nSvcOutput s "CRITICAL - 10.190.1.4: Net unreachable @ 192.168.30.1. rta nan, lost 100%" nHostAlias s "chi-r9-ea"
2020-10-26 00:58:11 - /usr/bin/snmptrap -v 2c -c public 10.150.10.73:162 '' NAGIOS-NOTIFY-MIB::nSvcEvent nSvcHostname s "MXTAMALT1006RTBHCSRMOB02" nSvcDesc s "Check remote TEF_LTE 201.162.238.110" nSvcStateID i 2 nSvcOutput s "(Service check timed out after 120.01 seconds)" nHostAlias s "MXTAMALT1006RTBHCSRMOB02""

Today I forced route null one IP and see the Critical in Logs, but never an event in eventman or snmptrapsender.log...

Look the logs..

[1603728505] wproc: Core Worker 3976: job 46 (pid=11632): Dormant child reaped
[1603728532] HOST NOTIFICATION: ac330k;mxcmxm01rtdclfisp02;DOWN;xi_host_notification_handler;CRITICAL - 10.225.118.9: rta nan, lost 100%
[1603728532] HOST NOTIFICATION: nagiosadmin;mxcmxm01rtdclfisp02;DOWN;xi_host_notification_handler;CRITICAL - 10.225.118.9: rta nan, lost 100%
[1603728532] HOST ALERT: mxcmxm01rtdclfisp02;DOWN;HARD;5;CRITICAL - 10.225.118.9: rta nan, lost 100%
[1603728532] GLOBAL HOST EVENT HANDLER: mxcmxm01rtdclfisp02;DOWN;HARD;5;xi_host_event_handler
[1603728562] wproc: Core Worker 3973: job 49 (pid=11928) timed out. Killing it
[1603728562] wproc: NOTIFY job 49 from worker Core Worker 3973 timed out after 30.01s
[1603728562] wproc: host=mxcmxm01rtdclfisp02; service=(none); contact=ac330k
[1603728562] wproc: early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1603728562] Warning: Notifying contact 'ac330k' of host 'mxcmxm01rtdclfisp02' by command '/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=host --contact="ac330k" --contactemail="[email protected]" --type=PROBLEM --escalated="0" --author="" --comments="" --host="mxcmxm01rtdclfisp02" --hostaddress="10.225.118.9" --hostalias="Dalia" --hostdisplayname="mxcmxm01rtdclfisp02" --hoststate=DOWN --hoststateid=1 --lasthoststate=DOWN --lasthoststateid=1 --hoststatetype=HARD --currentattempt=5 --maxattempts=5 --hosteventid=5075339 --hostproblemid=2470842 --hostoutput="CRITICAL - 10.225.118.9: rta nan, lost 100%" --longhostoutput="" --datetime="Mon Oct 26 10:08:52 CST 2020"' timed out after 0.00 seconds
[1603728562] wproc: Core Worker 3976: job 49 (pid=11930) timed out. Killing it
[1603728562] wproc: GLOBAL HOST EVENTHANDLER job 49 from worker Core Worker 3976 timed out after 30.01s
[1603728562] wproc: early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1603728562] Warning: Global host event handler command '/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_event.php --handler-type=host --host="mxcmxm01rtdclfisp02" --hostaddress="10.225.118.9" --hoststate=DOWN --hoststateid=1 --lasthoststate=DOWN --lasthoststateid=1 --hoststatetype=HARD --currentattempt=5 --maxattempts=5 --hosteventid=5075339 --hostproblemid=2470842 --hostoutput="CRITICAL - 10.225.118.9: rta nan, lost 100%" --longhostoutput="" --hostdowntime=0' timed out after 0.00 seconds
[1603728562] wproc: Core Worker 3974: job 49 (pid=11929) timed out. Killing it
[1603728562] wproc: NOTIFY job 49 from worker Core Worker 3974 timed out after 30.02s
[1603728562] wproc: host=mxcmxm01rtdclfisp02; service=(none); contact=nagiosadmin
[1603728562] wproc: early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1603728562] Warning: Notifying contact 'nagiosadmin' of host 'mxcmxm01rtdclfisp02' by command '/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=host --contact="nagiosadmin" --contactemail="@att.com" --type=PROBLEM --escalated="0" --author="" --comments="" --host="mxcmxm01rtdclfisp02" --hostaddress="10.225.118.9" --hostalias="Dalia" --hostdisplayname="mxcmxm01rtdclfisp02" --hoststate=DOWN --hoststateid=1 --lasthoststate=DOWN --lasthoststateid=1 --hoststatetype=HARD --currentattempt=5 --maxattempts=5 --hosteventid=5075339 --hostproblemid=2470842 --hostoutput="CRITICAL - 10.225.118.9: rta nan, lost 100%" --longhostoutput="" --datetime="Mon Oct 26 10:08:52 CST 2020"' timed out after 0.00 seconds
[1603728563] wproc: Core Worker 3973: job 49 (pid=11928): Dormant child reaped
[1603728563] wproc: Core Worker 3976: job 49 (pid=11930): Dormant child reaped
[1603728563] wproc: Core Worker 3974: job 49 (pid=11929): Dormant child reaped
[root@mxtulm01nagiosxi var]#


This is the info:


[root@mxtulm01nagiosxi ~]# chage -l nagios
Last password change : Jul 01, 2020
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
[root@mxtulm01nagiosxi ~]#


grep nag /etc/group
nagios1001:nagios,apache,snmptt
nagcmd1002:nagios,apache,snmptt
[root@mxtulm01nagiosxi ~]#

[root@mxtulm01nagiosxi ~]# ls -l /usr/local/nagiosxi
total 20
drwxr-xr-x. 2 root nagios 4096 Jul 12 2019 cron
drwxr-xr-x. 4 root nagios 67 Jul 12 2019 etc
drwxr-xr-x. 19 root nagios 4096 Sep 6 20:42 html
drwxr-xr-x. 3 root nagios 25 Jan 24 2014 nom
drwxr-xr-x. 4 root nagios 4096 Oct 23 19:06 scripts
drwsrwsr-x. 2 root nagios 254 Oct 26 09:44 tmp
drwxr-xr-x. 2 root nagios 4096 Sep 28 12:39 tools
drwxrwxr-x. 7 nagios nagios 4096 Oct 26 09:56 var
[root@mxtulm01nagiosxi ~]#

[tgriep]

That is strange that there were log entries from earlier. Maybe something is cached on the server ot that there is a corrupted MYSQL table.

Run these commands to stop the processes, clean and repair the SQL database and to restart the processes. Run them all as root.

Code: Select all

systemctl stop npcd
systemctl stop nagios
systemctl stop crond
pkill -9 -u nagios
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm
systemctl restart mariadb
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /var/lib/mrtg/mrtg_l
rm -f /usr/local/nagiosxi/var/*.lock
pkill python
rm -f /var/lib/php/session/*
systemctl restart httpd
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

Hopefully this will reload everything and allow the traps to be sent with new Alerts.

[oslec]

Hi,

Still no traps..and no events en eventman.log =(

PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................
PROCESSED 0 EVENTS
....................
PROCESSED 0 EVENTS
.....................
PROCESSED 0 EVENTS
...................

Forced down 1 host ..see the log in nagios.log

Oct 28 19:02:53 mxtulm01nagiosxi nagios: HOST NOTIFICATION: ac330k;MXCMXM01RTDCLFISP02;DOWN;xi_host_notification_handler;CRITICAL - 10.225.118.9: rta nan, lost 100%
Oct 28 19:02:53 mxtulm01nagiosxi nagios: HOST NOTIFICATION: nagiosadmin;MXCMXM01RTDCLFISP02;DOWN;xi_host_notification_handler;CRITICAL - 10.225.118.9: rta nan, lost 100%

Oct 28 19:03:23 mxtulm01nagiosxi nagios: job 159 (pid=7889): read() returned error 11
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: Core Worker 12740: job 159 (pid=7889) timed out. Killing it
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: NOTIFY job 159 from worker Core Worker 12740 timed out after 30.01s
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: host=MXCMXM01RTDCLFISP02; service=(none); contact=nagiosadmin
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Oct 28 19:03:23 mxtulm01nagiosxi nagios: Warning: Notifying contact 'nagiosadmin' of host 'MXCMXM01RTDCLFISP02' by command '/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=host --contact="nagiosadmin" --contactemail="@att.com" --type=PROBLEM --escalated="0" --author="" --comments="" --host="MXCMXM01RTDCLFISP02" --hostaddress="10.225.118.9" --hostalias="DALIA" --hostdisplayname="MXCMXM01RTDCLFISP02" --hoststate=DOWN --hoststateid=1 --lasthoststate=DOWN --lasthoststateid=1 --hoststatetype=HARD --currentattempt=5 --maxattempts=5 --hosteventid=5075347 --hostproblemid=2470846 --hostoutput="CRITICAL - 10.225.118.9: rta nan, lost 100%" --longhostoutput="" --datetime="Wed Oct 28 19:02:53 CST 2020"' timed out after 0.00 seconds
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: Core Worker 12745: job 158 (pid=7888) timed out. Killing it
Oct 28 19:03:23 mxtulm01nagiosxi nagios: job 158 (pid=7888): read() returned error 11
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: NOTIFY job 158 from worker Core Worker 12745 timed out after 30.01s
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: host=MXCMXM01RTDCLFISP02; service=(none); contact=ac330k
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
Oct 28 19:03:23 mxtulm01nagiosxi nagios: Warning: Notifying contact 'ac330k' of host 'MXCMXM01RTDCLFISP02' by command '/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=host --contact="ac330k" --contactemail="[email protected]" --type=PROBLEM --escalated="0" --author="" --comments="" --host="MXCMXM01RTDCLFISP02" --hostaddress="10.225.118.9" --hostalias="DALIA" --hostdisplayname="MXCMXM01RTDCLFISP02" --hoststate=DOWN --hoststateid=1 --lasthoststate=DOWN --lasthoststateid=1 --hoststatetype=HARD --currentattempt=5 --maxattempts=5 --hosteventid=5075347 --hostproblemid=2470846 --hostoutput="CRITICAL - 10.225.118.9: rta nan, lost 100%" --longhostoutput="" --datetime="Wed Oct 28 19:02:53 CST 2020"' timed out after 0.00 seconds
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: Core Worker 12740: job 159 (pid=7889): Dormant child reaped
Oct 28 19:03:23 mxtulm01nagiosxi nagios: wproc: Core Worker 12745: job 158 (pid=7888): Dormant child reaped
Oct 28 19:04:01 mxtulm01nagiosxi systemd: Started Session 120378 of user nagios.

Can you help me ?

Regards,