NagiosXI sending multiple SNMP traps

[Pselvin]

Hi Team,

We are experiencing issue with NagiosXI after the migration from CENT OS to RHEL 7. Nagios is sending muliple(approx 5000) SNMP trap events to our ticketing tool. We have looked at the console and the monitors/checks are green but, we are not sure why this is generating vast events.

Version 5.5.1

Here are the sample events

Code: Select all

losatlngp01-m.ec.lowes.com.52227 > bppmsnmpqa2.lowes.com.snmptrap: [bad udp cksum 0x7718 -> 0x9b3b!]  { SNMPv2c { V2Trap(259) R=58132531  system.sysUpTime.0=3541006 S:1.1.4.1.0=E:20006.1.7 E:20006.1.3.1.2="losatlapp45" E:20006.1.3.1.6="Lowes7OnlineB Node Service" E:20006.1.3.1.7=2 E:20006.1.3.1.17="(No output on stdout) stderr: connect to address 172.16.72.73 port 5666: Connection refused" } }
05:00:05.143127 IP (tos 0x0, ttl 64, id 7901, offset 0, flags [DF], proto UDP (17), length 325)
    losatlngp01-m.ec.lowes.com.47437 > bppmsnmpqa2.lowes.com.snmptrap: [bad udp cksum 0x772b -> 0x1ad5!]  { SNMPv2c { V2Trap(278) R=1002862805  system.sysUpTime.0=3541029 S:1.1.4.1.0=E:20006.1.7 E:20006.1.3.1.2="losatlwbp12" E:20006.1.3.1.6="MobileWebIHS_B_httpd process/cpu utilization" E:20006.1.3.1.7=2 E:20006.1.3.1.17="(No output on stdout) stderr: connect to address 172.16.68.14 port 5666: Connection refused" } }

Code: Select all

losatlngp01-m.ec.lowes.com.37655 > bppmsnmpqa2.lowes.com.snmptrap: [bad udp cksum 0x76d4 -> 0x6c44!]  { SNMPv2c { V2Trap(193) R=15726336  system.sysUpTime.0=3532934 S:1.1.4.1.0=E:20006.1.7 E:20006.1.3.1.2="losatlapp28" E:20006.1.3.1.6="Check Server Total Processes" E:20006.1.3.1.7=0 E:20006.1.3.1.17="PROCS OK: 275 processes" } }
04:58:44.266157 IP (tos 0x0, ttl 64, id 32906, offset 0, flags [DF], proto UDP (17), length 239)
    losatlngp01-m.ec.lowes.com.34610 > bppmsnmpqa2.lowes.com.snmptrap: [bad udp cksum 0x76d5 -> 0xcd74!]  { SNMPv2c { V2Trap(194) R=1307315145  system.sysUpTime.0=3532941 S:1.1.4.1.0=E:20006.1.7 E:20006.1.3.1.2="losatlqpp04" E:20006.1.3.1.6="Check Server CPU Status - Linux" E:20006.1.3.1.7=0 E:20006.1.3.1.17="OK: CPU Idle = 98.81%" } }

[Pselvin]

Code: Select all

*** GLOBAL HANDLER (snmptrapsender)...
Array
(
    [event_id] => 7021949
    [event_source] => 2
    [event_type] => 1
    [event_time] => 2019-02-22 09:24:31.363768
    [event_meta] => Array
        (
            [handler-type] => service
            [host] => losatlesp102
            [service] => File System:Permission Status:Read-Only Filesystems
            [hostaddress] => 172.16.72.111
            [hoststate] => UP
            [hoststateid] => 0
            [hosteventid] => 878008
            [hostproblemid] => 0
            [servicestate] => CRITICAL
            [servicestateid] => 2
            [lastservicestate] => CRITICAL
            [lastservicestateid] => 2
            [servicestatetype] => HARD
            [currentattempt] => 5
            [maxattempts] => 5
            [serviceeventid] => 902844
            [serviceproblemid] => 444021
            [serviceoutput] => (No output on stdout) stderr: connect to address 172.16.72.111 port 5666: Connection refused
            [longserviceoutput] => connect to host 172.16.72.111 port 5666: Connection refused
            [servicedowntime] => 0
        )

    [logging_enabled] => 1
)
PROCESSING:

[cdienger]

You may be running into a known bug with the xi_events table. Follow the below to clear out this table(along with a couple more that can have problems) and restart services:

*note* use mariadb instead of mysql depending on what is installed. Also, gearman may not be on this server so these may fail.

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop mod-gearman2-worker
systemctl stop gearmand
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl stop mysqld
systemctl start mysqld
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
systemctl start gearmand
systemctl start mod-gearman2-worker
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl start httpd

[Pselvin]

Tried running mentioned steps, got below error while executing this step - echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi

error 1045 (28000): access denied for user 'root'@'localhost' (using password: yes)

systemctl stop mod-gearman2-worker - no such service
systemctl stop gearmand - no such service

rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock - file is not exists.

DB : Mariadb

[cdienger]

The database is likely on a different machine(https://assets.nagios.com/downloads/nag ... Server.pdf) so the command would need to be changed:

mysql -h remote_db_IP -u username -ppassword -D nagiosxi

[Pselvin]

Hi There,

The database is on the same system. Could you please advise.

Thanks,
Prince

[ssax]

I've updated the command, you likely changed your root mysql password, try these:

Code: Select all

systemctl stop crond
systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop mod-gearman2-worker
systemctl stop gearmand
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
systemctl stop mysqld
systemctl start mysqld
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -unagiosxi -p'n@gweb' nagiosxi
systemctl start gearmand
systemctl start mod-gearman2-worker
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond
systemctl start httpd

[Pselvin]

We have ran all mentioned commands, but no luck.. Still mutiple SNMP traps are sending to BPPM.


losatlngp01.56821 > bppmsnmpqa2.snmptrap: [bad udp cksum 0x7712 -> 0xc372!] { SNMPv2c { V2Trap(254) R=1545387054 system.sysUpTime.0=170352 S:1.1.4.1.0=E:20006.1.7 E:20006.1.3.1.2="losatlnlp22" E:20006.1.3.1.6="NodeJS Port 3001 check" E:20006.1.3.1.7=2 E:20006.1.3.1.17="connect to address losatlnlp22 and port 3001: Connection refused" } }
10:48:29.533920 IP (tos 0x0, ttl 64, id 58206, offset 0, flags [DF], proto UDP (17), length 263)
losatlngp01.37266 > bppmsnmpqa2: [bad udp cksum 0x76ed -> 0x9cfa!] { SNMPv2c { V2Trap(218) R=496973530 system.sysUpTime.0=170369 S:1.1.4.1.0=E:20006.1.7 E:20006.1.3.1.2="losatlapp50" E:20006.1.3.1.6="SSH - Port 22" E:20006.1.3.1.7=2 E:20006.1.3.1.17="connect to address and losatlnlp22 port 22: Connection refused" } }

*** GLOBAL HANDLER...
Array
(
[event_id] => 7052651
[event_source] => 2
[event_type] => 1
[event_time] => 2019-02-22 09:26:42.456104
[event_meta] => Array
(
[handler-type] => service
[host] => losatldbp42
[service] => Memory Usage
[hostaddress] => losatldbp42
[hoststate] => UP
[hoststateid] => 0
[hosteventid] => 858250
[hostproblemid] => 0
[servicestate] => CRITICAL
[servicestateid] => 2
[lastservicestate] => CRITICAL
[lastservicestateid] => 2
[servicestatetype] => SOFT
[currentattempt] => 4
[maxattempts] => 5
[serviceeventid] => 904555
[serviceproblemid] => 445728
[serviceoutput] => (No output on stdout) stderr: connect to address losatldbp42 port 5666: Connection refused
[longserviceoutput] => connect to host losatldbp42.ec.lowes.com port 5666: Connection refused
[servicedowntime] => 0
)

[logging_enabled] => 1
)

[ssax]

Since you did a migration, did the IP address of the XI server change? If so, did you update all of your agents to allow from that new IP like you had to do with the old IP?