Page 5 of 9
Re: Random emails
Posted: Tue Aug 12, 2014 9:37 pm
by Box293
Interesting observations.
One thing to try, instead of stopping NSClient on those machines, disconnect the network cable (or if it is a VM do it in the VM settings).
Give it long enough for the host to go into a hard down state in Nagios.
Then reconnect the cable.
What appears in the debug log after doing this?
Also, run this command and post the output:
Re: Random emails
Posted: Wed Aug 13, 2014 10:29 am
by JohnFLi
Another thing I have noticed as well........ it seems to send out these emails mostly about 3 or 4 hosts. Sometimes it will send a email a different one, but mostly it is just about the 3 or 4 same hosts.
[root@g1vpnag01 me.adm]# ps -aef | grep nagios.cfg
nagios 6463 1 0 Aug12 ? 00:02:19 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 6469 6463 0 Aug12 ? 00:00:01 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 29892 6380 0 08:03 pts/0 00:00:00 grep nagios.cfg
I disabled the Nic on one of my vm's
Code: Select all
[1407943670.293067] [032.0] [pid=6463] ** Host Notification Attempt ** Host: 'G1VTOSS01', Type: NORMAL, Options: 0, Current State: 1, Last Notification: Wed Aug 13 08:17:50 2014
[1407943670.293084] [032.0] [pid=6463] Notification viability test passed.
[1407943670.293089] [032.1] [pid=6463] Current notification number: 2 (incremented)
[1407943670.293105] [032.1] [pid=6463] Host notification will NOT be escalated.
[1407943670.293112] [032.1] [pid=6463] Adding normal contacts for host to notification list.
[1407943670.293217] [032.0] [pid=6463] 2 contacts were notified. Next possible notification time: Wed Aug 13 08:32:50 2014
[1407943670.293222] [032.0] [pid=6463] 2 contacts were notified.
Re: Random emails
Posted: Thu Aug 14, 2014 5:40 pm
by JohnFLi
let the fun continue....
so today, I restarted two of my hosts.
anyway, almost right away, nagios started sending alerts to the $@ address (the email address that has started this entire thread) it had never sent these bogus emails out for these 2 hosts before.
nagios.debug (with logging set to 32) does not show that anything was sent out. nor should it becasue for these 2 hosts are sent as being on downtime.
So it seems that for some odd reason.....nagios wants to send an alert out everytime there is a service or host state change, but if it isn't time to send a real alert out, it just tries to send it to
[email protected] anyway. I swear, I have not neglected nagios, so it cant possible be feeling lonely.
Re: Random emails
Posted: Fri Aug 15, 2014 1:15 am
by Box293
This is a tricky one, I'm running out of ideas.
Does anyone else have any ideas as to what could be happening?
Re: Random emails
Posted: Fri Aug 15, 2014 10:12 am
by JohnFLi
well, i could just remove that mail account from my outlook....that way I wont see it again.
I only found out this was even going on becasue I needed to create a valid mailbox for nagios to that way I could acknowledge alerts via email (which I got working btw)
so anything in that mailbox gets deleted every 5 min......
Thank you for all your help on this.
Re: Random emails
Posted: Mon Aug 18, 2014 4:40 pm
by slansing
Any chance I could get a copy of that config zip as well? Just for those of us in the office here, I'm doing some looking around to see if this has been reported recently on core related forums.
Re: Random emails
Posted: Tue Aug 19, 2014 10:06 am
by JohnFLi
Sent you a PM
Re: Random emails
Posted: Tue Aug 19, 2014 5:10 pm
by slansing
Alrighty, I have a couple options here, firstly, can you run the following and show the output?:
Can you find the entire definition for "G1PCWEBSERV03'" in the following file and PM myself and Box it's contents please? And a service that was showing the problems as well if you can find one?
Code: Select all
/usr/local/nagios/var/retention.dat
Also, try removing:
From your "WEB-windows-server" host template, as that will auto-set any host with that template to OK upon a nagios restart, which could notify you if they were in a down state previously. Thanks!
Re: Random emails
Posted: Mon Aug 25, 2014 11:27 am
by JohnFLi
Here is retention.dat
Re: Random emails
Posted: Tue Aug 26, 2014 5:14 pm
by tmcdonald
Could you also post/PM your /usr/local/nagios/var/objects.cache file? This is a pretty weird issue, to be sure.
Received and placed in shared directory