NAGIOSXI Not receiving SNMP Trap Alert Emails

[rfaraci]

Hello;
I have been trying to set up SNMP traps and email alerts on our NAGIOSXI server. I believe I am 99.9 percent there. I have followed this article:
https://assets.nagios.com/downloads/nag ... ios_XI.pdf and I an receiving the traps on the Service Status Page. I then followed this article: https://assets.nagios.com/downloads/nag ... Mailer.pdf
If I send a custom notification from the Host Status Email page\advanced I do receive it. However, When I send a trap from the UPS itself, it populates the device status page but I don't receive the email.
I do see a log in the /var/log/snmptt.log. I also see the "Service Alert" log for the device trap I initiated in the /var/log/messages and everything looks good there.
I do see updates every 5 minutes in the /var/log/maillog, however I can't seem to find a reference to my specific trap email. It just seems to be general longs from all emails.
I was seeing a lot of "Error writing Message, File to large" messages but I increased the mailbox size and I am no longer seeing those error.
Is there a bad spacing issue in my notify_host_xi_contact or notify_sevice_xi_contact commands? I have posted them herein.
All other email notifications, both Host and Service, are working just fine.

[npolovenko]

Hello, @rfaraci. Well, first and easy one - please check the spam folder Have you changed the contact command in CCM/Alerting/Contacts/your_contact/Alert Settings? You'll need to make sure that Manage Host Notification Commands and Manage Service Notification commands were changed to notify_host_xi_contact and notify_service_xi_contact.
Also, SNMP traps always return OK state by default, even if it says Critical. ((Unless you additionally configured it)). By that I mean there will still be a green checkbox next to the state no matter what. So can yo upload a screenshot of your service check in a critical state?
Also, do the other services receive email notifications? Do the other contacts receive email notifications?

[rfaraci]

Hello npolovenko;
Thank you for getting back to me. I am new to Nagios and I believe I may have been on the wrong path in my previous email. But I believe that you may still be of assistance to me if you don't mind. I went back through the message logs more closely and I think I may be confused on "Service Alerts" verses "Service Notifications". And where to look for the settings within Nagios to see what actually triggers an alert verses a notification. I AM receiving notification emails.
Also, I'd like to somehow send an SNMP trap to Nagios that actually would trigger an email to be sure they are working, so I unplugged the network cable to one of my UPS's and I did receive a notification. However, I don't believe it was a trap, but possibly NagiosXI itself sending me the notification as it couldn't communicate with the host. The reason why I say that is because the host name itself turned red but the SNMP trap said "OK" and was green. Then again, you mentioned that trap notifications will always say OK, even in critical so I am somewhat confused by this. I have attached a screenshot of the alarm state.

NagiosXI_TEST.jpg

I really don't have any other way of testing a critical notification on the UPS's as they are all production units. The only two test traps the management card will allow me to send are Recalibrations messages. Self tests don't send traps.
Also, If a battery is in a warning state, I would want an email stating that rather than waiting to get notified at a critical state.
Hope I haven't completely confused you and of course, I really appreciate your assistance.

[npolovenko]

Take a look at this manual: https://support.nagios.com/kb/article/n ... al-77.html Look for the section called "Create Custom Events In SNMPTT", and read from there to the bottom of the page.

On the screenshot, the host(UPS) is red=in critical state probably because Nagios can't ping it-->because you turned off the ups. SNMP traps are not used in this process. When the ping check failed and Nagios sent you an email notification. The reason why SNMP trap itself hasn't sent you any notifications is because it was green/OK all long. So if you want to receive alerts from SNMP traps you'll need to create custom events in smptt.conf file.For example, when SNMP returns a certain message Nagios will turn the check into critical, or warning. Take a look at the link in the first paragraph.

Now, as far as for an alert versus notification. On the screenshot that you submitted you can see the column that says Attempt 1/1. That means notification will be send immediately after the state changes, from green OK to Warning (yellow) or red Critical. If you look at the other services or hosts, some of them have Attempt 1/5. That means if something goes wrong Nagios will check the service 4 more times before sending you an email. You can configure how many times you want Nagios to recheck before alerting you, and you can also configure the time interval for those rechecks.

[rfaraci]

Thank you, I will read through this and try to work with the snmptt.conf file. I'll also get a backup of it prior to working with it.
I wanted to be sure you didn't think I was ignoring your response. Am I going to have to set a custom event for each trap or just like three events (critical, warning and informational), and apply those to the incoming traps?

[tgriep]

That is correct, you will have to setup separate configs in the snmptt.conf file for each state types.

[rfaraci]

Hey there, I have been reading through the instruction set and have come up with the following.

I don’t have a “centOS” host set up so I was not able to do the exercise practice.
Also, I don’t have a EVENT nSvcEvent .1.3.6.1.4.1.20006.1.7 "Status Events" Normal event. I believe these were all set up earlier in the exercise.

However, from reading through this article several times, I have come up with what I believe to be the steps that I need to perform. Would you mind confirming my thoughts on this before I proceed? Also, I have enclosed the snmptt.conf file for your reference. As always, I really appreciate your assistance.

1. I believe I can leave the “FORMAT” line alone. (I think it states logically the trap meaning) Thoughts?
2. The “EXEC” lines are all remed out, so I would need to remove the rem?
3. Change EACH “EXEC” line to EXEC /usr/local/bin/snmptraphandling.py "$1" "SNMP Trap - $2" "$s" "$@" "" "$4"
EXAMPLE:
EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Normal
FORMAT Device reinitialized (coldStart)
EXEC /usr/local/bin/snmptraphandling.py "$1" "SNMP Trap - $2" "$s" "$@" "" "$4"
SDESC
A coldStart trap signifies that the SNMPv2 entity, acting
in an agent role, is reinitializing itself and that its
configuration may have been altered.
EDESC
4. Change all “Severe” statuses to “CRITICAL”
5. Change all “Normal” statuses to “OK”
6. Service snmptt restart
7. A. Disconnect each host (or in this case the UPS from the network) so that all the services change their state.
B. under the host/quick actions, click Schedule a forced immediate check
C. Under Advanced/commands, click Schedule immediate check for all services on this host.
D. Commit/Done
9. Confirm a trap for each service has been received.
10. Admin/monitoring config/Unconfigured Objects, click the host, hit the play button, next, finish
Click view and I should see all of the trap services created.
Plug that UPS back in force an immediate check for all services and all the services should be updated.

[tgriep]

1. Yes, you can leave the Format line alone but you can edit it is you like.
2. I only saw the default traps remed out but you can change them if you like.
3. For most of the Exec lines, you can leave alone as when the tool that adds the entries to the file, the settings would work for most traps.
4. You would not have to change the Severe settings as the script does that for you.
5. Same as 4.
6. Yes, every time you edit the snmptt.conf file, you need to restart the daemmon.
7. I don't understand why you would do this. Traps are passive and do not come in to the XI server on a regular schedule so this would not have to be done.
8. Missing
9. That would be a good practice.
10. Monitoring the Unconfigured Objects to configure the new traps is OK but you should not need to disconnect the UPS.

[rfaraci]

OK, so basically, after setting up the notify_host_xi_contact and notify_service_xi_contact, there was nothing else I needed to do from there on. Everything seems to be working and alerting just fine. https://support.nagios.com/kb/article/n ... al-77.html was very informational and a good read to understanding NagiosXI snmp traps and alerting fully, but in all actuality, there was nothing there that I needed to do.

Thank you all for your assistance, it is greatly appreciated and my issue is resolved. Have a great Holiday season everyone.

[tgriep]

That's pretty much it. If the Traps are coming in to the server and changing states, OK to Warning, OK to Critical, Critical to OK, etc.. that will generate a hard alert and send the emails.
Have a good Holiday Season too.