Page 1 of 2
snmptrap on Nagios 5.7.3
Posted: Mon Sep 21, 2020 4:24 pm
by dhatha03
Hello,
I recently made minor interface configuration changes to my Cisco router. During the course of this it generated a few snmp traps and Nagios properly handled them. However, these traps seem to be stuck in a constant loop within /usr/local/nagios/var/nagios.log:
[1600722478] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;CRITICAL;HARD;1;SNMP Trap Received at 1600696975 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:administratively down
[1600722479] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600696980 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up
..
..
..
[1600722493] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;CRITICAL;HARD;1;SNMP Trap Received at 1600696975 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:administratively down
[1600722494] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600696980 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up
As you can see the trap received time stamp within the messages match.
I have logging enabled for snmptrapd.log and see no traps, as it should be, but nagios.log is still filling up with the messages above. I have also disconnected the network gear and still saw the above messages in nagios.log.
Thank you in advance.
Re: snmptrap on Nagios 5.7.3
Posted: Mon Sep 21, 2020 4:37 pm
by dhatha03
I deleted all files within /var/spool/snmptt/ and the spamming of the repeated traps stopped within nagios.log. I conducted a quick test and disabled a Cisco interface and nagios.log correctly showed one trap. However, when I enabled the port the spam messaging started up again with saying the port is up constantly and with the same time stamp.
[1600724503] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;CRITICAL;HARD;1;SNMP Trap Received at 1600724499 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:administratively down
[1600724530] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600724527 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up
[1600724535] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;CRITICAL;HARD;1;SNMP Trap Received at 1600724499 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:administratively down
[1600724536] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600724527 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up
[1600724541] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;CRITICAL;HARD;1;SNMP Trap Received at 1600724499 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:administratively down
[1600724541] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600724527 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up
Re: snmptrap on Nagios 5.7.3
Posted: Tue Sep 22, 2020 3:36 pm
by ssax
Please PM a copy of your profile, you can download it from
Admin > System Profile > Download Profile.
Then run this command (as root) and PM me the resulting /tmp/SNMPFILES.zip file:
Code: Select all
zip -r /tmp/SNMPFILES.zip /etc/snmp
Additionally, what is the output of these commands?
Code: Select all
ls -ld /var/spool/snmptt
ls -ltr /var/spool/snmptt | head -10 # This is to check if they are even being deleted
Do you see anything else that could be related in your
/var/log/snmptt/* logs?
Re: snmptrap on Nagios 5.7.3
Posted: Wed Sep 23, 2020 6:59 am
by dhatha03
Files have been sent to you. I have the following output.
[root@ovm-nagios01 ~]# ls -ld /var/spool/snmptt
drwxrwxr-x 2 snmptt snmptt 4096 Sep 22 20:51 /var/spool/snmptt
[root@ovm-nagios01 ~]# ls -ltr /var/spool/snmptt | head -10
total 0
I have had to manually delete the files in /var/spool/snmptt when the problem occurs
Output for /var/log/snmptt/*
-rw-rw-r-- 1 root root 0 Sep 20 03:29 /var/log/snmptt/snmptt.log
-rw-rw-r-- 1 root root 4293 Aug 24 14:21 /var/log/snmptt/snmptt.log-20200830
-rw-rw-r-- 1 root root 13520 Sep 3 20:21 /var/log/snmptt/snmptt.log-20200906
-rw-rw-r-- 1 root root 494 Sep 8 20:25 /var/log/snmptt/snmptt.log-20200913
-rw-rw-r-- 1 root root 494 Sep 14 14:28 /var/log/snmptt/snmptt.log-20200920
-rw-r--r-- 1 root root 0 Sep 22 03:30 /var/log/snmptt/snmpttsystem.log
-rw-r--r-- 1 root root 382 Sep 1 15:27 /var/log/snmptt/snmpttsystem.log-20200902
-rw-r--r-- 1 root root 1146 Sep 3 20:02 /var/log/snmptt/snmpttsystem.log-20200906
-rw-r--r-- 1 root root 419 Sep 16 12:21 /var/log/snmptt/snmpttsystem.log-20200917
-rw-r--r-- 1 root root 2514 Sep 21 21:28 /var/log/snmptt/snmpttsystem.log-20200922
-rw-rw-r-- 1 root root 0 Jun 2 03:27 /var/log/snmptt/snmpttunknown.log
-rw-rw-r-- 1 root root 1971 Aug 27 2019 /var/log/snmptt/snmpttunknown.log-20190828
-rw-rw-r-- 1 root root 740 May 12 14:12 /var/log/snmptt/snmpttunknown.log-20200513
-rw-rw-r-- 1 root root 2174 May 13 17:50 /var/log/snmptt/snmpttunknown.log-20200517
-rw-rw-r-- 1 root root 371 Jun 1 17:45 /var/log/snmptt/snmpttunknown.log-20200602
Re: snmptrap on Nagios 5.7.3
Posted: Wed Sep 23, 2020 9:08 am
by ssax
I do not see these files, please resend this part as it must've gotten missed:
Then run this command (as root) and PM me the resulting
/tmp/SNMPFILES.zip file:
Code: Select all
zip -r /tmp/SNMPFILES.zip /etc/snmp
Re: snmptrap on Nagios 5.7.3
Posted: Wed Sep 23, 2020 9:22 am
by dhatha03
File sent.
Results of executing the zip command:
[root@ovm-nagios01 ~]# zip -r /tmp/SNMPFILES.zip /etc/snmp
adding: etc/snmp/ (stored 0%)
adding: etc/snmp/snmptt.ini (deflated 71%)
adding: etc/snmp/tmp/ (stored 0%)
adding: etc/snmp/tmp/SCTP-MIB.txt (deflated 13%)
adding: etc/snmp/tmp/NET-SNMP-EXTEND-MIB.txt (deflated 17%)
adding: etc/snmp/tmp/MTA-MIB.txt (deflated 12%)
adding: etc/snmp/tmp/SNMP-VIEW-BASED-ACM-MIB.txt (deflated 19%)
adding: etc/snmp/tmp/CISCO-TC.txt (deflated 12%)
adding: etc/snmp/tmp/LM-SENSORS-MIB.txt (deflated 16%)
adding: etc/snmp/tmp/IPV6-ICMP-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/SNMP-USER-BASED-SM-MIB.txt (deflated 19%)
adding: etc/snmp/tmp/NET-SNMP-PASS-MIB.txt (deflated 17%)
adding: etc/snmp/tmp/TCP-MIB.txt (deflated 12%)
adding: etc/snmp/tmp/HOST-RESOURCES-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/UCD-IPFWACC-MIB.txt (deflated 16%)
adding: etc/snmp/tmp/NET-SNMP-EXAMPLES-MIB.txt (deflated 41%)
adding: etc/snmp/tmp/EtherLike-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/AGENTX-MIB.txt (deflated 14%)
adding: etc/snmp/tmp/RFC-1215.txt (deflated 12%)
adding: etc/snmp/tmp/SNMP-COMMUNITY-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/NET-SNMP-AGENT-MIB.txt (deflated 59%)
adding: etc/snmp/tmp/SNMP-MPD-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/IF-INVERTED-STACK-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/INET-ADDRESS-MIB.txt (deflated 17%)
adding: etc/snmp/tmp/IP-MIB.txt (deflated 12%)
adding: etc/snmp/tmp/TRANSPORT-ADDRESS-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/SNMPv2-CONF.txt (deflated 13%)
adding: etc/snmp/tmp/UDP-MIB.txt (deflated 12%)
adding: etc/snmp/tmp/HCNUM-TC.txt (deflated 12%)
adding: etc/snmp/tmp/RFC1213-MIB.txt (deflated 14%)
adding: etc/snmp/tmp/IF-MIB.txt (deflated 61%)
adding: etc/snmp/tmp/RFC1155-SMI.txt (deflated 13%)
adding: etc/snmp/tmp/IPV6-MIB.txt (deflated 40%)
adding: etc/snmp/tmp/CISCO-SMI.txt (deflated 13%)
adding: etc/snmp/tmp/IP-FORWARD-MIB.txt (deflated 16%)
adding: etc/snmp/tmp/UCD-DEMO-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/UCD-SNMP-MIB.txt (deflated 51%)
adding: etc/snmp/tmp/SNMP-USM-DH-OBJECTS-MIB.txt (deflated 19%)
adding: etc/snmp/tmp/CISCO-ENVMON-MIB.txt (deflated 82%)
adding: etc/snmp/tmp/SNMP-FRAMEWORK-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/CISCO-PROCESS-MIB.txt (deflated 63%)
adding: etc/snmp/tmp/NET-SNMP-VACM-MIB.txt (deflated 17%)
adding: etc/snmp/tmp/SNMP-USM-AES-MIB.txt (deflated 17%)
adding: etc/snmp/tmp/IANA-RTPROTO-MIB.txt (deflated 17%)
adding: etc/snmp/tmp/NETWORK-SERVICES-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/IPV6-UDP-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/IPV6-TC.txt (deflated 11%)
adding: etc/snmp/tmp/IANA-ADDRESS-FAMILY-NUMBERS-MIB.txt (deflated 22%)
adding: etc/snmp/tmp/DISMAN-SCHEDULE-MIB.txt (deflated 39%)
adding: etc/snmp/tmp/SNMPv2-TC.txt (deflated 13%)
adding: etc/snmp/tmp/NET-SNMP-TC.txt (deflated 13%)
adding: etc/snmp/tmp/SNMP-NOTIFICATION-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/SMUX-MIB.txt (deflated 13%)
adding: etc/snmp/tmp/SNMPv2-MIB.txt (deflated 60%)
adding: etc/snmp/tmp/HOST-RESOURCES-TYPES.txt (deflated 17%)
adding: etc/snmp/tmp/IANA-LANGUAGE-MIB.txt (deflated 17%)
adding: etc/snmp/tmp/NET-SNMP-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/UCD-DLMOD-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/SNMPv2-SMI.txt (deflated 13%)
adding: etc/snmp/tmp/IPV6-TCP-MIB.txt (deflated 15%)
adding: etc/snmp/tmp/NOTIFICATION-LOG-MIB.txt (deflated 18%)
adding: etc/snmp/tmp/IANAifType-MIB.txt (deflated 16%)
adding: etc/snmp/tmp/DISMAN-SCRIPT-MIB.txt (deflated 65%)
adding: etc/snmp/tmp/RMON-MIB.txt (deflated 63%)
adding: etc/snmp/tmp/SNMPv2-TM.txt (deflated 13%)
adding: etc/snmp/tmp/DISMAN-EVENT-MIB.txt (deflated 74%)
adding: etc/snmp/tmp/UCD-DISKIO-MIB.txt (deflated 16%)
adding: etc/snmp/tmp/SNMP-TARGET-MIB.txt (deflated 16%)
adding: etc/snmp/tmp/BRIDGE-MIB.txt (deflated 53%)
adding: etc/snmp/tmp/SNMP-PROXY-MIB.txt (deflated 16%)
adding: etc/snmp/snmptt.conf.nxti (deflated 89%)
adding: etc/snmp/snmptt_nxti.bak (deflated 89%)
adding: etc/snmp/snmpd.conf (deflated 6%)
adding: etc/snmp/nagios-check-storage (stored 0%)
adding: etc/snmp/backup/ (stored 0%)
adding: etc/snmp/backup/snmptt.ini (deflated 71%)
adding: etc/snmp/backup/snmptt.conf.nxti (deflated 89%)
adding: etc/snmp/backup/snmptt_nxti.bak (deflated 89%)
adding: etc/snmp/backup/snmpd.conf (deflated 13%)
adding: etc/snmp/backup/snmptrapd.conf (deflated 68%)
adding: etc/snmp/backup/snmptt.conf (stored 0%)
adding: etc/snmp/before_upgrade/ (stored 0%)
adding: etc/snmp/before_upgrade/snmptt.ini (deflated 71%)
adding: etc/snmp/before_upgrade/snmptt.conf.nxti (deflated 89%)
adding: etc/snmp/before_upgrade/snmptt_nxti.bak (deflated 88%)
adding: etc/snmp/before_upgrade/snmpd.conf (deflated 13%)
adding: etc/snmp/before_upgrade/snmptrapd.conf (deflated 58%)
adding: etc/snmp/before_upgrade/snmptt.conf (stored 0%)
adding: etc/snmp/original/ (stored 0%)
adding: etc/snmp/original/snmptt.conf.nxti (deflated 87%)
adding: etc/snmp/original/snmptt.conf (deflated 70%)
adding: etc/snmp/snmptrapd.conf (deflated 68%)
adding: etc/snmp/snmptt.conf (stored 0%)
adding: etc/snmp/after_upgrade/ (stored 0%)
adding: etc/snmp/after_upgrade/snmptt.ini (deflated 71%)
adding: etc/snmp/after_upgrade/snmptt.conf.nxti (deflated 89%)
adding: etc/snmp/after_upgrade/snmptt_nxti.bak (deflated 88%)
adding: etc/snmp/after_upgrade/snmpd.conf (deflated 13%)
adding: etc/snmp/after_upgrade/snmptrapd.conf (deflated 58%)
adding: etc/snmp/after_upgrade/snmptt.conf (stored 0%)
Re: snmptrap on Nagios 5.7.3
Posted: Thu Sep 24, 2020 5:18 pm
by ssax
If this happens again, please get us this output before fixing:
Try editing your
/etc/snmp/snmptt.ini and change these:
Code: Select all
threads_enable = 0
threads_max = 10
To these:
Code: Select all
threads_enable = 1
threads_max = 20
Then restart SNMPTT:
See if that alleviates it.
What is the output of this command?
Code: Select all
grep "alarm\|FIFO" /usr/local/bin/snmptraphandling.py
Re: snmptrap on Nagios 5.7.3
Posted: Mon Sep 28, 2020 6:48 am
by dhatha03
Followed the changes recommended and still experiencing the issue.
For ps aux:
Code: Select all
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 33664 1424 ? Ss Sep21 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S Sep21 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Sep21 0:36 [migration/0]
root 4 0.0 0.0 0 0 ? S Sep21 0:03 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Sep21 0:00 [stopper/0]
root 6 0.0 0.0 0 0 ? S Sep21 0:01 [watchdog/0]
root 7 0.0 0.0 0 0 ? S Sep21 0:35 [migration/1]
root 8 0.0 0.0 0 0 ? S Sep21 0:00 [stopper/1]
root 9 0.0 0.0 0 0 ? S Sep21 0:03 [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S Sep21 0:01 [watchdog/1]
root 11 0.0 0.0 0 0 ? S Sep21 0:57 [events/0]
root 12 0.0 0.0 0 0 ? S Sep21 1:06 [events/1]
root 13 0.0 0.0 0 0 ? S Sep21 0:00 [events/0]
root 14 0.0 0.0 0 0 ? S Sep21 0:00 [events/1]
root 15 0.0 0.0 0 0 ? S Sep21 0:00 [events_long/0]
root 16 0.0 0.0 0 0 ? S Sep21 0:00 [events_long/1]
root 17 0.0 0.0 0 0 ? S Sep21 0:00 [events_power_ef]
root 18 0.0 0.0 0 0 ? S Sep21 0:00 [events_power_ef]
root 19 0.0 0.0 0 0 ? S Sep21 0:00 [cgroup]
root 20 0.0 0.0 0 0 ? S Sep21 0:00 [khelper]
root 21 0.0 0.0 0 0 ? S Sep21 0:00 [netns]
root 22 0.0 0.0 0 0 ? S Sep21 0:00 [async/mgr]
root 23 0.0 0.0 0 0 ? S Sep21 0:00 [pm]
root 24 0.0 0.0 0 0 ? S Sep21 0:02 [sync_supers]
root 25 0.0 0.0 0 0 ? S Sep21 0:00 [bdi-default]
root 26 0.0 0.0 0 0 ? S Sep21 0:00 [kintegrityd/0]
root 27 0.0 0.0 0 0 ? S Sep21 0:00 [kintegrityd/1]
root 28 0.0 0.0 0 0 ? S Sep21 0:01 [kblockd/0]
root 29 0.0 0.0 0 0 ? S Sep21 3:31 [kblockd/1]
root 30 0.0 0.0 0 0 ? S Sep21 0:00 [kacpid]
root 31 0.0 0.0 0 0 ? S Sep21 0:00 [kacpi_notify]
root 32 0.0 0.0 0 0 ? S Sep21 0:00 [kacpi_hotplug]
root 33 0.0 0.0 0 0 ? S Sep21 0:00 [ata_aux]
root 34 0.0 0.0 0 0 ? S Sep21 0:00 [ata_sff/0]
root 35 0.0 0.0 0 0 ? S Sep21 0:00 [ata_sff/1]
root 36 0.0 0.0 0 0 ? S Sep21 0:00 [ksuspend_usbd]
root 37 0.0 0.0 0 0 ? S Sep21 0:00 [khubd]
root 38 0.0 0.0 0 0 ? S Sep21 0:00 [kseriod]
root 39 0.0 0.0 0 0 ? S Sep21 0:00 [md/0]
root 40 0.0 0.0 0 0 ? S Sep21 0:00 [md/1]
root 41 0.0 0.0 0 0 ? S Sep21 0:00 [md_misc/0]
root 42 0.0 0.0 0 0 ? S Sep21 0:00 [md_misc/1]
root 43 0.0 0.0 0 0 ? S Sep21 0:00 [linkwatch]
root 44 0.0 0.0 0 0 ? S Sep21 0:00 [khungtaskd]
root 45 0.0 0.0 0 0 ? S Sep21 0:00 [lru-add-drain/0]
root 46 0.0 0.0 0 0 ? S Sep21 0:00 [lru-add-drain/1]
root 47 0.0 0.0 0 0 ? S Sep21 0:06 [kswapd0]
root 48 0.0 0.0 0 0 ? SN Sep21 0:00 [ksmd]
root 49 0.0 0.0 0 0 ? SN Sep21 5:29 [khugepaged]
root 50 0.0 0.0 0 0 ? S Sep21 0:00 [aio/0]
root 51 0.0 0.0 0 0 ? S Sep21 0:00 [aio/1]
root 52 0.0 0.0 0 0 ? S Sep21 0:00 [crypto/0]
root 53 0.0 0.0 0 0 ? S Sep21 0:00 [crypto/1]
root 60 0.0 0.0 0 0 ? S Sep21 0:00 [kthrotld/0]
root 61 0.0 0.0 0 0 ? S Sep21 0:00 [kthrotld/1]
root 63 0.0 0.0 0 0 ? S Sep21 0:00 [kpsmoused]
root 64 0.0 0.0 0 0 ? S Sep21 0:00 [usbhid_resumer]
root 65 0.0 0.0 0 0 ? S Sep21 0:00 [deferwq]
root 98 0.0 0.0 0 0 ? S Sep21 0:00 [kdmremove]
root 99 0.0 0.0 0 0 ? S Sep21 0:00 [kstriped]
root 270 0.0 0.0 0 0 ? S Sep21 0:00 [scsi_eh_0]
root 271 0.0 0.0 0 0 ? S Sep21 0:00 [scsi_eh_1]
root 390 0.0 0.0 0 0 ? S Sep21 0:00 [virtio-blk]
root 418 0.0 0.0 0 0 ? S Sep21 0:00 [kdmflush]
root 420 0.0 0.0 0 0 ? S Sep21 0:03 [kdmflush]
root 486 0.0 0.0 0 0 ? S Sep21 0:41 [jbd2/dm-1-8]
root 487 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 584 0.0 0.0 10944 520 ? S<s Sep21 0:01 /sbin/udevd -d
root 753 0.0 0.0 0 0 ? S Sep21 0:00 [hd-audio0]
root 791 0.0 0.0 0 0 ? S Sep21 0:00 [virtio-net]
root 797 0.0 0.0 0 0 ? S Sep21 0:00 [vballoon]
root 1089 0.0 0.0 0 0 ? S Sep21 0:45 [kdmflush]
root 1091 0.0 0.0 0 0 ? S Sep21 0:02 [kdmflush]
root 1093 0.0 0.0 0 0 ? S Sep21 0:48 [kdmflush]
root 1097 0.0 0.0 0 0 ? S Sep21 0:00 [kdmflush]
root 1101 0.0 0.0 0 0 ? S Sep21 0:00 [kdmflush]
root 1102 0.0 0.0 10940 532 ? S< Sep21 0:03 /sbin/udevd -d
root 1109 0.0 0.0 0 0 ? S Sep21 0:00 [kdmflush]
root 1110 0.0 0.0 10940 512 ? S< Sep21 0:00 /sbin/udevd -d
root 1209 0.0 0.0 0 0 ? S Sep21 0:00 [jbd2/vda1-8]
root 1210 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 1211 0.0 0.0 0 0 ? S Sep21 0:00 [jbd2/dm-6-8]
root 1212 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 1213 0.0 0.0 0 0 ? S Sep21 0:00 [jbd2/dm-7-8]
root 1214 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 1215 0.0 0.0 0 0 ? S Sep21 0:04 [jbd2/dm-5-8]
root 1216 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 1217 0.0 0.0 0 0 ? S Sep21 3:11 [jbd2/dm-2-8]
root 1218 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 1219 0.0 0.0 0 0 ? S Sep21 0:15 [jbd2/dm-3-8]
root 1220 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 1221 0.0 0.0 0 0 ? S Sep21 3:20 [jbd2/dm-4-8]
root 1222 0.0 0.0 0 0 ? S Sep21 0:00 [ext4-dio-unwrit]
root 1264 0.0 0.0 0 0 ? S Sep21 0:28 [kauditd]
root 1323 0.0 0.0 0 0 ? S Sep21 0:05 [ib_addr]
root 1328 0.0 0.0 0 0 ? S Sep21 0:00 [infiniband/0]
root 1329 0.0 0.0 0 0 ? S Sep21 0:00 [infiniband/1]
root 1338 0.0 0.0 0 0 ? S Sep21 0:00 [ib_mcast]
root 1343 0.0 0.0 0 0 ? S Sep21 0:00 [iw_cm_wq]
root 1348 0.0 0.0 0 0 ? S Sep21 0:00 [ib_cm/0]
root 1349 0.0 0.0 0 0 ? S Sep21 0:00 [ib_cm/1]
root 1354 0.0 0.0 0 0 ? S Sep21 0:00 [rdma_cm]
root 1375 0.0 0.0 0 0 ? S Sep21 0:00 [ipoib_flush]
root 1501 0.0 0.0 0 0 ? S Sep21 0:53 [flush-253:1]
root 1502 0.0 0.0 0 0 ? S Sep21 0:30 [flush-253:2]
root 1503 0.0 0.0 0 0 ? S Sep21 0:07 [flush-253:3]
root 1504 0.0 0.0 0 0 ? S Sep21 0:03 [flush-253:4]
root 1505 0.0 0.0 0 0 ? S Sep21 0:01 [flush-253:5]
root 1565 0.0 0.0 29764 680 ? S<sl Sep21 4:03 auditd
root 1567 0.0 0.0 84216 896 ? S<sl Sep21 2:56 /sbin/audispd
root 1569 0.0 0.0 12492 876 ? S< Sep21 3:05 /sbin/audisp-remote
root 1601 0.0 0.2 262456 8040 ? Sl Sep21 1:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
root 1635 0.0 0.0 18252 660 ? Ss Sep21 0:24 irqbalance --pid=/var/run/irqbalance.pid
rpc 1657 0.0 0.0 18980 856 ? Ss Sep21 0:01 rpcbind
rpcuser 1681 0.0 0.0 25432 1332 ? Ss Sep21 0:00 rpc.statd
dbus 1719 0.0 0.0 34096 1216 ? Ss Sep21 0:00 dbus-daemon --system
root 1749 0.0 0.0 189140 2892 ? Ss Sep21 0:04 cupsd -C /etc/cups/cupsd.conf
root 1784 0.0 0.0 4076 612 ? Ss Sep21 0:00 /usr/sbin/acpid
68 1796 0.0 0.0 53032 2428 ? Ssl Sep21 0:05 hald
root 1797 0.0 0.0 20396 1116 ? S Sep21 0:00 hald-runner
root 1840 0.0 0.0 22516 1012 ? S Sep21 0:00 hald-addon-input: Listening on /dev/input/event2 /dev/input/event0
68 1844 0.0 0.0 18004 980 ? S Sep21 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root 1859 0.0 0.0 223760 2208 ? Ss Sep21 0:30 /usr/sbin/sssd -f -D
root 1860 0.0 0.1 258460 6660 ? S Sep21 1:24 /usr/libexec/sssd/sssd_be --domain default --uid 0 --gid 0 --debug-to-files
root 1861 0.0 0.0 227088 2996 ? S Sep21 1:08 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --debug-to-files
root 1862 0.0 0.0 206776 2796 ? S Sep21 1:12 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --debug-to-files
root 1863 0.0 0.0 200152 2396 ? S Sep21 0:07 /usr/libexec/sssd/sssd_autofs --uid 0 --gid 0 --debug-to-files
root 1890 0.0 0.0 91272 1364 ? Ssl Sep21 0:00 pcscd
root 1908 0.0 0.0 554504 3452 ? Ssl Sep21 0:32 automount --pid-file /var/run/autofs.pid
root 2027 0.0 0.1 200196 4804 ? S Sep21 1:53 /usr/sbin/snmpd -LS0-5d -Lf /dev/null -p /var/run/snmpd.pid
root 2071 0.0 0.0 66288 1240 ? Ss Sep21 0:00 /usr/sbin/sshd
root 2084 0.0 0.0 21712 852 ? Ss Sep21 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp 2095 0.0 0.0 38760 1892 ? Ss Sep21 0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root 2134 0.0 0.0 108220 1432 ? S Sep21 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --u
mysql 2237 1.2 2.0 1207764 79972 ? Sl Sep21 116:09 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --
root 2348 0.0 0.0 83092 3424 ? Ss Sep21 0:06 /usr/libexec/postfix/master
root 2354 0.0 0.0 0 0 ? S Sep21 0:16 [rpciod/0]
root 2355 0.0 0.0 0 0 ? S Sep21 0:08 [rpciod/1]
postfix 2362 0.0 0.0 93508 3544 ? S Sep21 0:02 qmgr -l -t fifo -u
root 2372 0.0 0.0 0 0 ? S< Sep21 0:00 [kslowd000]
root 2373 0.0 0.0 0 0 ? S< Sep21 0:00 [kslowd001]
root 2374 0.0 0.0 183092 2060 ? Ss Sep21 0:00 /usr/sbin/abrtd
root 2375 0.0 0.0 0 0 ? S Sep21 0:20 [nfsiod]
root 2412 0.0 0.3 348084 15336 ? Ss Sep21 0:30 /usr/sbin/httpd
root 2424 0.0 0.0 116880 1232 ? Ss Sep21 0:26 crond
nagios 2435 0.0 0.0 368884 960 ? S Sep21 1:21 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
root 2457 0.0 0.0 21104 488 ? Ss Sep21 0:00 /usr/sbin/atd
root 2524 0.0 1.3 388736 51640 ? Ssl Sep21 0:35 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent
root 2554 0.0 0.0 64476 1420 ? Ss Sep21 0:00 /usr/sbin/certmonger -S -p /var/run/certmonger.pid
root 2642 0.0 0.0 134028 1968 ? Ssl Sep21 0:00 /usr/sbin/gdm-binary -nodaemon
root 2648 0.0 0.0 4060 516 tty2 Ss+ Sep21 0:00 /sbin/mingetty /dev/tty2
root 2650 0.0 0.0 4060 516 tty3 Ss+ Sep21 0:00 /sbin/mingetty /dev/tty3
root 2656 0.0 0.0 4060 520 tty4 Ss+ Sep21 0:00 /sbin/mingetty /dev/tty4
root 2658 0.0 0.0 4060 520 tty5 Ss+ Sep21 0:00 /sbin/mingetty /dev/tty5
root 2661 0.0 0.0 4060 520 tty6 Ss+ Sep21 0:00 /sbin/mingetty /dev/tty6
root 2713 0.0 0.0 177148 2360 ? Sl Sep21 0:00 /usr/libexec/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display1
root 2716 0.0 0.3 188052 11912 tty1 Ssl+ Sep21 0:07 /usr/bin/Xorg :0 -br -verbose -audit 4 -auth /var/run/gdm/auth-for-gdm-QIWjeV/database -nolisten tcp vt1
root 2791 0.0 0.0 1036460 2900 ? Sl Sep21 0:00 /usr/sbin/console-kit-daemon --no-daemon
gdm 2861 0.0 0.0 20064 588 ? S Sep21 0:00 /usr/bin/dbus-launch --exit-with-session
gdm 2862 0.0 0.0 33808 504 ? Ss Sep21 0:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
gdm 2863 0.0 0.1 271560 5644 ? Ssl Sep21 0:00 /usr/bin/gnome-session --autostart=/usr/share/gdm/autostart/LoginWindow/
root 2869 0.0 0.0 49980 2172 ? S Sep21 0:00 /usr/libexec/devkit-power-daemon
gdm 2875 0.0 0.1 132260 4024 ? S Sep21 0:02 /usr/libexec/gconfd-2
gdm 2891 0.0 0.0 120748 3800 ? S Sep21 0:02 /usr/libexec/at-spi-registryd
gdm 2892 0.0 0.1 361704 7688 ? Ssl Sep21 0:21 /usr/libexec/gnome-settings-daemon --gconf-prefix=/apps/gdm/simple-greeter/settings-manager-plugins
gdm 2895 0.0 0.0 434244 2208 ? Ssl Sep21 0:00 /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=12
gdm 2909 0.0 0.0 135296 1432 ? S Sep21 0:00 /usr/libexec/gvfsd
gdm 2910 0.0 0.1 289316 5548 ? S Sep21 0:02 metacity
gdm 2911 0.0 0.2 387900 8920 ? S Sep21 0:16 /usr/libexec/gdm-simple-greeter
gdm 2913 0.0 0.1 246284 4376 ? S Sep21 0:00 /usr/libexec/polkit-gnome-authentication-agent-1
root 2918 0.0 0.0 64712 2740 ? S Sep21 0:00 /usr/libexec/polkit-1/polkitd
gdm 2926 0.0 0.0 427356 2088 ? S<sl Sep21 0:00 /usr/bin/pulseaudio --start --log-target=syslog
rtkit 2928 0.0 0.0 168452 964 ? SNl Sep21 0:14 /usr/libexec/rtkit-daemon
root 2934 0.0 0.0 187596 1840 ? S Sep21 0:00 pam: gdm-password
root 2981 0.0 0.0 0 0 ? S Sep21 0:00 [nfsv4.0-svc]
root 3746 0.0 0.1 128796 4144 ? Ss 11:33 0:00 sshd: <username> [priv]
<username> 3865 0.0 0.0 128796 2616 ? S 11:33 0:00 sshd: <username>@pts/0
root 3866 0.0 0.1 128808 4112 ? Ss 11:33 0:00 sshd: <username> [priv]
<username> 3874 0.0 0.0 120944 1988 pts/0 Ss 11:33 0:00 -bash
<username> 3879 0.0 0.0 128808 2020 ? S 11:33 0:00 sshd: <username>@notty
<username> 3883 0.0 0.0 73852 2364 ? Ss 11:33 0:00 /usr/libexec/openssh/sftp-server
root 4058 0.0 0.0 182100 2848 pts/0 S 11:33 0:00 su -
root 4071 0.0 0.0 108488 1824 pts/0 S 11:34 0:00 -bash
root 4257 0.0 0.0 100956 604 pts/0 S+ 11:34 0:00 tail -f /usr/local/nagios/var/nagios.log
postfix 4708 0.0 0.0 93480 3608 ? S 11:35 0:00 cleanup -z -t unix -u
postfix 4710 0.0 0.0 93544 3752 ? S 11:35 0:00 smtp -t unix -u
root 5020 0.0 0.1 128808 4136 ? Ss 11:35 0:00 sshd: <username> [priv]
<username> 5106 0.0 0.0 128808 2632 ? S 11:35 0:00 sshd: <username>@pts/1
root 5107 0.0 0.1 128808 4112 ? Ss 11:35 0:00 sshd: <username> [priv]
<username> 5119 0.0 0.0 120944 1992 pts/1 Ss 11:35 0:00 -bash
<username> 5130 0.0 0.0 128808 2020 ? S 11:35 0:00 sshd: <username>@notty
<username> 5148 0.0 0.0 73852 2364 ? Ss 11:35 0:00 /usr/libexec/openssh/sftp-server
root 5173 0.0 0.0 182100 2848 pts/1 S 11:35 0:00 su -
root 5175 0.0 0.0 114540 2392 ? S 11:35 0:00 /usr/libexec/fprintd
root 5257 0.0 0.0 0 0 ? S 11:35 0:00 [flush-253:6]
root 5283 0.0 0.0 108488 1828 pts/1 S 11:35 0:00 -bash
root 5483 0.0 0.0 160384 1948 ? S 11:36 0:00 CROND
root 5484 0.0 0.0 160384 1948 ? S 11:36 0:00 CROND
root 5485 0.0 0.0 160384 1948 ? S 11:36 0:00 CROND
root 5486 0.0 0.0 160384 1948 ? S 11:36 0:00 CROND
root 5487 0.0 0.0 160384 1948 ? S 11:36 0:00 CROND
root 5488 0.0 0.0 160384 1948 ? S 11:36 0:00 CROND
nagios 5489 0.0 0.0 106112 1124 ? Ss 11:36 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 5490 0.0 0.0 106112 1120 ? Ss 11:36 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 5493 0.0 0.0 106112 1124 ? Ss 11:36 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 5498 0.0 0.0 106112 1124 ? Ss 11:36 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 5499 0.0 0.0 106112 1120 ? Ss 11:36 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 5500 1.7 0.6 333748 24860 ? S 11:36 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 5501 0.0 0.0 106112 1124 ? Ss 11:36 0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1
nagios 5503 1.5 0.6 333492 24532 ? S 11:36 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios 5505 2.6 0.9 344576 36084 ? S 11:36 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 5506 1.9 0.6 333660 24764 ? S 11:36 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 5507 2.7 0.9 345092 36416 ? S 11:36 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 5510 1.6 0.6 333488 24512 ? S 11:36 0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php
root 5889 1.0 0.0 122708 1216 pts/1 R+ 11:36 0:00 ps aux
apache 15555 0.5 1.0 485064 41568 ? S 11:05 0:10 /usr/sbin/httpd
postfix 19435 0.0 0.0 93336 3488 ? S 10:25 0:00 pickup -l -t fifo -u
apache 22177 0.0 1.0 484996 41528 ? S Sep27 0:09 /usr/sbin/httpd
apache 22178 0.0 0.9 480916 37540 ? S Sep27 0:08 /usr/sbin/httpd
apache 22179 0.0 1.0 485284 42228 ? S Sep27 0:09 /usr/sbin/httpd
apache 22180 0.0 1.0 485256 41708 ? S Sep27 0:09 /usr/sbin/httpd
apache 22181 0.0 1.0 484756 41148 ? S Sep27 0:09 /usr/sbin/httpd
apache 22182 0.0 1.0 484756 40896 ? S Sep27 0:09 /usr/sbin/httpd
apache 22183 0.0 1.0 485004 41480 ? S Sep27 0:09 /usr/sbin/httpd
apache 22184 0.0 0.9 478856 35320 ? S Sep27 0:09 /usr/sbin/httpd
nagios 28520 0.1 0.2 170832 10872 ? Ss Sep21 18:29 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 28521 0.0 0.0 10092 956 ? S Sep21 0:52 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 28522 0.0 0.0 10092 952 ? S Sep21 0:52 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 28523 0.0 0.0 10092 960 ? S Sep21 0:53 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 28524 0.0 0.0 10092 952 ? S Sep21 0:53 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 28529 0.0 0.1 179632 6352 ? S Sep21 0:53 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 31258 0.0 0.0 196224 2632 ? Ss Sep21 0:14 /usr/sbin/snmptrapd -Ln -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid
root 31297 0.0 0.2 162240 11468 ? Ss Sep21 0:12 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt 31298 0.1 0.3 166424 12544 ? Ss Sep21 10:11 /usr/bin/perl /usr/sbin/snmptt --daemon
grep "alarm\|FIFO" /usr/local/bin/snmptraphandling.py did not return a value.
Re: snmptrap on Nagios 5.7.3
Posted: Mon Sep 28, 2020 11:25 am
by dhatha03
Also, this started to happen after I upgraded to 5.7.2. I have since upgraded to 5.7.3 in hopes that it resolved it, but it has not.
Re: snmptrap on Nagios 5.7.3
Posted: Mon Sep 28, 2020 6:10 pm
by ssax
Please attach this file:
Code: Select all
/usr/local/bin/snmptraphandling.py
What are the trap OIDs that are being sent in for these? It's not listing the OID and I'm having trouble finding how you're getting duplicates.
Code: Select all
[1600724530] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600724527 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up
[1600724535] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;CRITICAL;HARD;1;SNMP Trap Received at 1600724499 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:administratively down
[1600724536] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600724527 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up
[1600724541] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;CRITICAL;HARD;1;SNMP Trap Received at 1600724499 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:administratively down
[1600724541] SERVICE ALERT: <hostname removed>;Interface: GigabitEthernet0/0/2;OK;HARD;1;SNMP Trap Received at 1600724527 with variables ifIndex.3:3 ifDescr.3:GigabitEthernet0/0/2 ifType.3:ethernetCsmacd enterprises.9.2.2.1.1.20.3:up