'UP' alert with Unknown, warning or critical alert

[pnewlon]

I am monitoring Avnet ROOMALERT4E devices with my NagiosXI installation - ping and pulling two SNMP OIDs. Every time I get an unknown, warning, or critical alert email, I also get a second email indicating the device is 'UP'. There are no 'DOWN' alerts. I have checked my host and service configs, I ONLY have 'critical' checked in the service alert configuration and 'down' in the host configuration. How do I stop the 'UP' notices? Thanks! Phil



define service {
service_description Ping
use xiwizard_genericnetdevice_ping_service
hostgroup_name DMB_Environment
max_check_attempts 5
check_interval 5
retry_interval 1
check_period 24x7
notification_interval 180
first_notification_delay 15
notification_period 24x7
notification_options w,u,c
notifications_enabled 1
contacts nagiosadmin
contact_groups DMB Group
register 1
}

define host {
host_name 00923_ENV01
use xiwizard_genericnetdevice_host
address 10.9.23.207
max_check_attempts 5
check_interval 5
retry_interval 1
check_period 24x7
contact_groups DMB Group
notification_interval 180
notification_period 24x7
first_notification_delay 15
notification_options d
notifications_enabled 1
icon_image network_node.png
statusmap_image network_node.png
_xiwizard genericnetdevice
register 1
}

[pnewlon]

- Linux Distribution and version?

Linux lpnagv01.global.wiie.com 2.6.18-164.9.1.el5 #1 SMP Tue Dec 15 21:04:57 EST 2009 i686 i686 i386 GNU/Linux

- VMware Image or Manual Install of XI?

VMWare image

- Are there specials configurations on your system, ie; is Gnome installed? Are you using a proxy? Are you using SSL?

Nothing special, used Nagios VMW image. No proxy, no SSL

- Nagios XI version

Nagios XI 2011R1.3

[mguthrie]

What you have "should" work from what I can see. What happens if you turn off "Up" notifications for that particular contact config?

[pnewlon]

I turned off alerts to that user, still getting the 'UP' message. In fact, this is really whacked because I literally unplugged a device (MKTG_ENV01) and correctly got a 'critical' alert to the failed ping response. However, I still got the 'UP' message!



Body of 'UP' message:

***** Nagios Monitor XI Alert *****

Notification Type: PROBLEM
Host: MKTG_ENV01
State: UP
Address: 10.255.216.126
Info:

Date/Time: Thu Jun 16 15:13:31 EDT 2011

[mguthrie]

Lets try a few things. Run the following from the command-line

Code: Select all

#making sure the time is correctly synced on the server
yum install ntp
ntpdate pool.ntp.org

#kill off all nagios processes and start it fresh
killall -9 nagios
service nagios start

Also, can you look through your configs and make sure you're not getting bit by the "Implied Inheritance" rule for contacts.

http://nagios.sourceforge.net/docs/3_0/ ... tance.html

Implied Inheritance

Normally you have to either explicitly specify the value of a required variable in an object definition or inherit it from a template. There are a few exceptions to this rule, where Nagios will assume that you want to use a value that instead comes from a related object. For example, the values of some service variables will be copied from the host the service is associated with if you don't otherwise specify them.

The following table lists the object variables that will be implicitly inherited from related objects if you don't explicitly specify their value in your object definition or inherit them from a template.
Object Type Object Variable Implied Source
Services contact_groups contact_groups in the associated host definition
notification_interval notification_interval in the associated host definition
notification_period notification_period in the associated host definition
Host Escalations contact_groups contact_groups in the associated host definition
notification_interval notification_interval in the associated host definition
escalation_period notification_period in the associated host definition
Service Escalations contact_groups contact_groups in the associated service definition
notification_interval notification_interval in the associated service definition
escalation_period notification_period in the associated service definition

[pnewlon]

I can't run ntpdate as the system is running ntpd. It syncs with our internal NTP time server which syncs with our internal Stratum 0 (GPS) time source.

[root@lpnagv01 pnewl01]# /usr/sbin/ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.255.213.233 216.87.244.23 2 u 109 1024 377 17.866 0.031 0.417
LOCAL(0) .LOCL. 10 l 58 64 377 0.000 0.000 0.001

I have killed and restarte the nagios server per your instructions. I will watch for the results!

[mguthrie]

No worries on the ntpdate, I just wanted to make sure the date/time on your system was correct.

[pnewlon]

core config snapshot sent via email, let me know if you don't receive it.

[mguthrie]

Taking a look at this now...

[mguthrie]

There's a little bit to have to sort through for all of this, can I get the following info from you?

Can you post the email notification messages for the XI users 'nagiosadmin' and another user in the DMB Group? These messages are customized per-user, so you can access them from Configure->My Account Settings->Notification Messages.


Also see the following document on XI Users and Contacts. Your host and service notification period for each XI users needs to be <username>_notification_times in order to work correctly.
http://assets.nagios.com/downloads/nagi ... ntacts.pdf

Do the timestamps on the contradicting emails appear to be around the same time, or could there be some issue with the email server getting backed up?