NagiosXI Adding MIB's error messages

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: NagiosXI Adding MIB's error messages

Post by tgriep »

Thanks for the nagios.log file. I found these entries for the traps you have sent and the Nagios Process is receiving them.
[1514350800] CURRENT SERVICE STATE: 10.15.98.213;SNMP Traps;OK;HARD;1;The agent card is alive. 12:0:00:44.41 0 1 / sysUpTime (TICKS):12:0:00:44.41 enterprises.476.1.42.3.2.2 ():0 enterprises.476.1.42.2.1.6 ():1
[1514350800] CURRENT SERVICE STATE: 10.15.98.234;SNMP Traps;OK;HARD;1;The agent card is alive. 4:21:35:08.43 0 1 / sysUpTime (TICKS):4:21:35:08.43 enterprises.476.1.42.3.2.2 ():0 enterprises.476.1.42.2.1.6 ():1
[1514350800] CURRENT SERVICE STATE: 10.15.98.236;SNMP Traps;OK;HARD;1;The agent card is alive. 5:0:01:29.10 1 1 / sysUpTime (TICKS):5:0:01:29.10 enterprises.476.1.42.3.2.2 ():1 enterprises.476.1.42.2.1.6 ():1
[1514350800] CURRENT SERVICE STATE: 10.15.98.237;SNMP Traps;OK;HARD;1;The agent card is alive. 5:0:01:28.64 1 1 / sysUpTime (TICKS):5:0:01:28.64 enterprises.476.1.42.3.2.2 ():1 enterprises.476.1.42.2.1.6 ():1
From your screen capture, it looks like Active Checks have been enabled for those services, when that is done, it will submit a OK status and a TRAP RESET message every minute which is overwriting the trap message.
Go to those services and disable Active Checks and the next time a trap is received, it will not get reset by the Active Check.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
rfaraci
Posts: 20
Joined: Wed Dec 06, 2017 10:30 am

Re: NagiosXI Adding MIB's error messages

Post by rfaraci »

I did as you asked, turning off the active checks, nice find on that because that would have been an issue down the road for sure. HOWEVER, still getting same result on test. I have attached the snmptt.log.
snmptt.log.jpg
I'm not understanding how we are expecting to see any result in the Service Status GUI if the log is not showing a correctly processed trap? In the log file, I highlighted in yellow what I see when I process the APC traps. I see 5 entries which then when I look I see them in the service status page of Nagios.

I'm not seeing those entries for the Liebert traps which makes me think they are not processing correctly. Does that make sense or am I way off? The log shows the same entry:
Wed Dec 27 14:51:11 2017 .1.3.6.1.4.1.476.1.42.2.3.0.0.7 Normal "Status Events" 10.15.98.234 - The agent card is alive. 5:22:33:58.87 0 1
with nothing coming up in the service status page.
You do not have the required permissions to view the files attached to this post.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: NagiosXI Adding MIB's error messages

Post by tgriep »

When the snmptt daemon gets a trap that is has to process, it looks up the OID in the snmptt.conf file or one of the included processed_mibs files and it is finds a matching entry, it runs the EXEC command and logs it in the snmptt.log file.
I see the entries in your screen shot under the yellow highlighted section and from the Liebert Device.

When the EXEC command runs, it runs the snmptraphandling.py script which sends the data to the Nagios process and from your nagios.log file, I see those entries.

So, after turning off the Active Checks, they should be displaying in the XI GUI.

I searched through the full nagios.log file and found these additional entries.
Line 2344: [1514350800] CURRENT SERVICE STATE: NR3-PDU2;SNMP Traps;OK;HARD;1;The agent card is alive. 10:0:01:01.01 0 1 / sysUpTime (TICKS):10:0:01:01.01 enterprises.476.1.42.3.2.2 ():0 enterprises.476.1.42.2.1.6 ():1
Line 2346: [1514350800] CURRENT SERVICE STATE: NR3-PDU3;SNMP Traps;OK;HARD;1;The agent card is alive. 10:0:01:02.24 0 1 / sysUpTime (TICKS):10:0:01:02.24 enterprises.476.1.42.3.2.2 ():0 enterprises.476.1.42.2.1.6 ():1
Line 2348: [1514350800] CURRENT SERVICE STATE: NR3-PDU4;SNMP Traps;OK;HARD;1;The agent card is alive. 10:0:01:01.72 0 1 / sysUpTime (TICKS):10:0:01:01.72 enterprises.476.1.42.3.2.2 ():0 enterprises.476.1.42.2.1.6 ():1
Line 21471: [1514399779] SERVICE ALERT: NR3-PDU4;SNMP Traps;OK;HARD;1;The agent card is alive. 11:0:01:01.72 0 1 / sysUpTime (TICKS):11:0:01:01.72 enterprises.476.1.42.3.2.2 ():0 enterprises.476.1.42.2.1.6 ():1
Check those hosts as well to see if the trap was received.

Go to the Home > Service Details screen and search for Traps and see if they are displayed in the GUI.

Another thing to do it to delete one of the SNMP Trap services and the host entry from XI. If there are 2 entries, one with a hostname and one with an IP address, delete them both.
Since Active Checks were enabled, maybe there is another setting causing the traps to not be displayed.
Then send a trap from that device and see if it shows up in the Unconfigured Objects menu. If it does, configure it and then it should be automatically received in XI and the service should show the status information.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
rfaraci
Posts: 20
Joined: Wed Dec 06, 2017 10:30 am

Re: NagiosXI Adding MIB's error messages

Post by rfaraci »

Hello;

I decided after reading your post to start at the Liebert Web Cards and work my way back toward the NMS. I have researched this in depth on my end and have found the following;

1. All of my APC UPS's are working fine and I am receiving their traps on the Nagios XI Service Status page.
2. I have four Liebert UPS's and I am receiving traps from one of them but not from the other three.
3. I downloaded iReasoning Mib Tester and did a WALK test with the UPS Web Cards that I am having an issue with and found that the three UPS Web Cards that are not responding in Nagios are indeed working and returning the proper OID responses. So I know the cards are OK. I even updated them to the latest firmware as well.
4. I downloaded Wireshark and tested with positive results from Wireshark. (I am receiving the trap responses).
5. I have concluded that the issue is NOT on the Liebert side.
6. I also verified that the switches that the UPS’s are networked to have SNMP enabled on them.

I did as you asked and deleted one of the hosts (Liebert-ServerRoom-UPS3) in Nagios and sent a trap. I have nothing in the
Unconfigured Objects. I have nothing in the Nogios.log either.

Looking closer at the Nagios.log you supplied in your last post, I’m noticing that those traps are not being returned my UPS’s but by PDU’s. I am ONLY trying to fix the lack of logging from the Liebert UPS’s at this time. Liebert-ServerRoom-UPS1, 3 and 4 are not logging. Liebert-ServerRoom-UPS2 IS logging. I haven’t even configured the PDU’s with Nagios yet.

I have done an extended amount of work before posting back to you because I wanted to be sure I didn’t miss anything on the Liebert web card or network switch side that was causing the issue.

I appreciate all of your help, believe me, but at this point I have to unfortunately disagree, the three UPS’s are not logging in Nagios.log. I have attached the latest log here-in for you. Please let me know your thoughts.
You do not have the required permissions to view the files attached to this post.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: NagiosXI Adding MIB's error messages

Post by tgriep »

From the log file I see 2 servers called Liebert-ServerRoom-UPS1 and Liebert-ServerRoom-UPS-1. Same IP address, different name.

Code: Select all

[1514469897] HOST ALERT: Liebert-ServerRoom-UPS-1;UP;SOFT;2;OK - 10.15.98.234: rta 0.617ms, lost 0%
[1514469897] HOST ALERT: Liebert-ServerRoom-UPS1;UP;SOFT;2;OK - 10.15.98.234: rta 0.620ms, lost 0%
I found a couple more errors in the log file.

Code: Select all

[1514468086] SERVICE ALERT: Liebert-ServerRoom-UPS-1;SNMP Traps;OK;HARD;1;Waiting for trap...
[1514468086] External command error: Malformed command
[1514474580] SERVICE ALERT: Liebert-ServerRoom-UPS3;SNMP Traps;OK;HARD;1;Waiting for trap...
[1514474580] External command error: Malformed command
Something is wrong with UPS1, UPS-1 and UPS3 and it may be a configuration issue.
Can you post your System Profile?
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it to this post.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
rfaraci
Posts: 20
Joined: Wed Dec 06, 2017 10:30 am

Re: NagiosXI Adding MIB's error messages

Post by rfaraci »

Bingo! You found the problem, and for that I thank you!

Once I saw your last post, and the log entries you sent, I went back and searched for host names and IP's that related to those UPS's. I removed all references and then set up the host snmp traps again and all seems to be testing OK.I am much farther ahead than yesterday.

I am however having an issue with two particular units and receiving those traps which I'm going to look into further and do more trouble shooting.
Can we keep this open until next week when I have a chance to delve into those?

Thanks for hanging in there with me through this, I greatly appreciate it! I hope you have a great New Years.
Top
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:
Contact dwhitfield

Re: NagiosXI Adding MIB's error messages

Post by dwhitfield »

rfaraci wrote: Can we keep this open until next week when I have a chance to delve into those?
Absolutely! Please note that our office will be closed on Monday and Tuesday next week.
Top
rfaraci
Posts: 20
Joined: Wed Dec 06, 2017 10:30 am

Re: NagiosXI Adding MIB's error messages

Post by rfaraci »

Hello and Happy New Year.

I had the opportunity to do a hard test on the Liebert UPS alerts, and of course, they failed......again.
I put the UPS into bypass mode and didn't receive any alarms and the GUI didn't show any abnormalities either. Everything showed green and normal.

snmptt.log shows:


Wed Jan 3 10:25:28 2018 .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 1 SNMPv2-SMI::enterprises.476.1.42.3.2.1.192 5:14:13:56.36
Wed Jan 3 10:25:29 2018 .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 2 SNMPv2-SMI::enterprises.476.1.42.3.2.1.193 5:14:13:56.39
Wed Jan 3 10:25:29 2018 .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 8 SNMPv2-SMI::enterprises.476.1.42.3.2.1.52.5 5:14:13:56.44
Wed Jan 3 10:25:29 2018 .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 4 SNMPv2-SMI::enterprises.476.1.42.3.2.1.72 5:14:13:56.49
Wed Jan 3 10:29:11 2018 .1.3.6.1.4.1.476.1.42.3.3.0.2 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is removed from the 8 SNMPv2-SMI::enterprises.476.1.42.3.2.1.52.5 5:14:13:56.44
Wed Jan 3 10:29:12 2018 .1.3.6.1.4.1.476.1.42.3.3.0.2 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is removed from the 2 SNMPv2-SMI::enterprises.476.1.42.3.2.1.193 5:14:13:56.39
Wed Jan 3 10:29:16 2018 .1.3.6.1.4.1.476.1.42.3.3.0.2 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is removed from the 1 SNMPv2-SMI::enterprises.476.1.42.3.2.1.192 5:14:13:56.36
Wed Jan 3 10:29:16 2018 .1.3.6.1.4.1.476.1.42.3.3.0.2 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is removed from the 4 SNMPv2-SMI::enterprises.476.1.42.3.2.1.72 5:14:13:56.49

/var/log/messages shows:


3 10:25:21 nagiosxi nagios: SERVICE ALERT: sap-sw12.mybobs.com;Physical Memory Usage;OK;SOFT;5;Physical Memory: 94%used(24076MB/25600MB) (<95%) : OK
Jan 3 10:25:28 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:28 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::mib-2.33.2 Enterprise Specific Trap (3) Uptime: 5 days, 14:13:56.06#012#011SNMPv2-SMI::mib-2.33.1.6.2.1.1.6 = INTEGER: 6#011SNMPv2-SMI::mib-2.33.1.6.2.1.2.6 = OID: SNMPv2-SMI::mib-2.33.1.6.3.6#011SNMPv2-SMI::mib-2.33.1.6.2.1.3.6 = Timeticks: (48323606) 5 days, 14:13:56.06
Jan 3 10:25:28 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:28 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.3 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.36#012#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.1 = Gauge32: 1#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.2 = OID: SNMPv2-SMI::enterprises.476.1.42.3.2.1.192#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.3 = Timeticks: (48323636) 5 days, 14:13:56.36
Jan 3 10:25:28 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:28 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.7.8 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.38#012#011SNMPv2-MIB::sysUpTime = Timeticks: (48323638) 5 days, 14:13:56.38#011SNMPv2-SMI::enterprises.476.1.42.3.7.7 = STRING: "Active:Warning:System Input Power Problem"
Jan 3 10:25:28 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:28 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.3 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.39#012#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.1 = Gauge32: 2#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.2 = OID: SNMPv2-SMI::enterprises.476.1.42.3.2.1.193#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.3 = Timeticks: (48323639) 5 days, 14:13:56.39
Jan 3 10:25:28 nagiosxi snmptt[18702]: .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 1 SNMPv2-SMI::enterprises.476.1.42.3.2.1.192 5:14:13:56.36
Jan 3 10:25:29 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:29 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.7.8 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.40#012#011SNMPv2-MIB::sysUpTime = Timeticks: (48323640) 5 days, 14:13:56.40#011SNMPv2-SMI::enterprises.476.1.42.3.7.7 = STRING: "Active:Warning:Bypass Not Available"
Jan 3 10:25:29 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:29 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.3 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.44#012#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.1 = Gauge32: 8#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.2 = OID: SNMPv2-SMI::enterprises.476.1.42.3.2.1.52.5#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.3 = Timeticks: (48323644) 5 days, 14:13:56.44
Jan 3 10:25:29 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:29 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.7.8 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.48#012#011SNMPv2-MIB::sysUpTime = Timeticks: (48323648) 5 days, 14:13:56.48#011SNMPv2-SMI::enterprises.476.1.42.3.7.7 = STRING: "Active:Warning:Bypass Frequency Error"
Jan 3 10:25:29 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:29 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.3 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.49#012#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.1 = Gauge32: 4#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.2 = OID: SNMPv2-SMI::enterprises.476.1.42.3.2.1.72#011SNMPv2-SMI::enterprises.476.1.42.3.2.3.1.3 = Timeticks: (48323649) 5 days, 14:13:56.49
Jan 3 10:25:29 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:29 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::enterprises.476.1.42.3.7.8 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:56.51#012#011SNMPv2-MIB::sysUpTime = Timeticks: (48323651) 5 days, 14:13:56.51#011SNMPv2-SMI::enterprises.476.1.42.3.7.7 = STRING: "Active:Warning:Battery Discharging"
Jan 3 10:25:29 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:29 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::mib-2.33.2 Enterprise Specific Trap (1) Uptime: 5 days, 14:13:57.07#012#011SNMPv2-SMI::mib-2.33.1.2.3.0 = INTEGER: 1170#011SNMPv2-SMI::mib-2.33.1.2.2.0 = INTEGER: 0#011SNMPv2-SMI::mib-2.33.1.9.7.0 = INTEGER: 0
Jan 3 10:25:29 nagiosxi snmptrapd[1404]: 2018-01-03 10:25:29 10.15.98.237(via UDP: [10.15.98.237]:32770->[10.0.1.203]) TRAP, SNMP v1, community N4g10sMonit0r#012#011SNMPv2-SMI::mib-2.33.2 Enterprise Specific Trap (3) Uptime: 5 days, 14:13:57.08#012#011SNMPv2-SMI::mib-2.33.1.6.2.1.1.2 = INTEGER: 2#011SNMPv2-SMI::mib-2.33.1.6.2.1.2.2 = OID: SNMPv2-SMI::mib-2.33.1.6.3.2#011SNMPv2-SMI::mib-2.33.1.6.2.1.3.2 = Timeticks: (48323707) 5 days, 14:13:57.07
Jan 3 10:25:34 nagiosxi snmptt[18702]: .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 2 SNMPv2-SMI::enterprises.476.1.42.3.2.1.193 5:14:13:56.39
Jan 3 10:25:34 nagiosxi snmptt[18702]: .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 8 SNMPv2-SMI::enterprises.476.1.42.3.2.1.52.5 5:14:13:56.44
Jan 3 10:25:34 nagiosxi snmptt[18702]: .1.3.6.1.4.1.476.1.42.3.3.0.1 Normal "Status Events" 10.15.98.237 - This notification is sent each time a condition is inserted into the 4 SNMPv2-SMI::enterprises.476.1.42.3.2.1.72 5:14:13:56.49

So it would seem that Nagios is getting trap alerts but doesn't know how to process them?
Can you please tell me what seems to be the issue now?? :x
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: NagiosXI Adding MIB's error messages

Post by tgriep »

Just an update to the post.
We found that the system was receiving the Traps but those hosts were not setup in DNS so the traps were coming in as the IP address and not the Hostname.
If the Hosts are added to DNS, that would allow the traps to come in and the status will update to the hostname and not the IP address entries in XI.

@rfaraci, if you still have any questions, post them here if you like.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
Locked

Return to “Nagios XI”