Random emails

[Box293]

Interesting observations.

One thing to try, instead of stopping NSClient on those machines, disconnect the network cable (or if it is a VM do it in the VM settings).
Give it long enough for the host to go into a hard down state in Nagios.
Then reconnect the cable.
What appears in the debug log after doing this?

Also, run this command and post the output:

Code: Select all

ps -aef | grep nagios.cfg

[JohnFLi]

Another thing I have noticed as well........ it seems to send out these emails mostly about 3 or 4 hosts. Sometimes it will send a email a different one, but mostly it is just about the 3 or 4 same hosts.

[root@g1vpnag01 me.adm]# ps -aef | grep nagios.cfg
nagios 6463 1 0 Aug12 ? 00:02:19 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 6469 6463 0 Aug12 ? 00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 29892 6380 0 08:03 pts/0 00:00:00 grep nagios.cfg


I disabled the Nic on one of my vm's

Code: Select all

[1407943670.293067] [032.0] [pid=6463] ** Host Notification Attempt ** Host: 'G1VTOSS01', Type: NORMAL, Options: 0, Current State: 1, Last Notification: Wed Aug 13 08:17:50 2014
[1407943670.293084] [032.0] [pid=6463] Notification viability test passed.
[1407943670.293089] [032.1] [pid=6463] Current notification number: 2 (incremented)
[1407943670.293105] [032.1] [pid=6463] Host notification will NOT be escalated.
[1407943670.293112] [032.1] [pid=6463] Adding normal contacts for host to notification list.
[1407943670.293217] [032.0] [pid=6463] 2 contacts were notified.  Next possible notification time: Wed Aug 13 08:32:50 2014
[1407943670.293222] [032.0] [pid=6463] 2 contacts were notified.

[JohnFLi]

let the fun continue....
so today, I restarted two of my hosts.
anyway, almost right away, nagios started sending alerts to the $@ address (the email address that has started this entire thread) it had never sent these bogus emails out for these 2 hosts before.

nagios.debug (with logging set to 32) does not show that anything was sent out. nor should it becasue for these 2 hosts are sent as being on downtime.

So it seems that for some odd reason.....nagios wants to send an alert out everytime there is a service or host state change, but if it isn't time to send a real alert out, it just tries to send it to $@mydomain.com anyway. I swear, I have not neglected nagios, so it cant possible be feeling lonely.

[Box293]

This is a tricky one, I'm running out of ideas.

Does anyone else have any ideas as to what could be happening?

[JohnFLi]

well, i could just remove that mail account from my outlook....that way I wont see it again.
I only found out this was even going on becasue I needed to create a valid mailbox for nagios to that way I could acknowledge alerts via email (which I got working btw)
so anything in that mailbox gets deleted every 5 min......

Thank you for all your help on this.

[slansing]

Any chance I could get a copy of that config zip as well? Just for those of us in the office here, I'm doing some looking around to see if this has been reported recently on core related forums.

[JohnFLi]

Sent you a PM

[slansing]

Alrighty, I have a couple options here, firstly, can you run the following and show the output?:

Can you find the entire definition for "G1PCWEBSERV03'" in the following file and PM myself and Box it's contents please? And a service that was showing the problems as well if you can find one?

Code: Select all

/usr/local/nagios/var/retention.dat

Also, try removing:

Code: Select all

initial_state                  	o

From your "WEB-windows-server" host template, as that will auto-set any host with that template to OK upon a nagios restart, which could notify you if they were in a down state previously. Thanks!

[JohnFLi]

Here is retention.dat

[tmcdonald]

Could you also post/PM your /usr/local/nagios/var/objects.cache file? This is a pretty weird issue, to be sure.

Received and placed in shared directory