Page 3 of 4
Re: Multiple email notifications
Posted: Wed Oct 10, 2012 10:22 am
by scottwilkerson
These both look normal.
I am sure you have tried this, but if you send a regular email to this list is it duplicated?
The reason I ask, is that you have already shown that nagios is only sending one notification in the log... Can you verify on the listserv the received emails?
Re: Multiple email notifications
Posted: Thu Oct 11, 2012 3:05 am
by aap
Thanks.
Regularv emails are not dupliacted - they are only sent once. This behaviour is very odd. I am wondering if I will need to do a database repair.
The reason I say this is that the issue only happens to a few hosts. There are hosts that have multiple notification groups and still report in the correct manner. I really don't know where else to look.
Re: Multiple email notifications
Posted: Thu Oct 11, 2012 9:41 am
by scottwilkerson
It certainly wouldn't hurt...
you can run
Code: Select all
/usr/local/nagiosxi/scripts/repairmysql.sh nagios
echo "vacuum;vacuum analyse;vacuum full;"|psql nagiosxi postgres
Re: Multiple email notifications
Posted: Fri Oct 19, 2012 4:05 am
by aap
Hi,
I have ran the database repair and the additional command that you specified with the results below:
VACUUM
VACUUM
VACUUM
Unfortunately, it has not made a difference. We are still receiving multiple alerts for "some" services.
Re: Multiple email notifications
Posted: Mon Oct 22, 2012 3:07 pm
by scottwilkerson
Is it the same few hosts? Are the notifications absolutely identical?
I'm wondering if we don't somehow have these hosts duplicated.
Re: Multiple email notifications
Posted: Wed Oct 24, 2012 2:48 am
by aap
Thanks.
I am starting to see more hosts exhibiting the same behaviuor. We monitor over 8300 services so it is getting a bit frustrating.
I have manually checked the files and there is no duplication anywhere.
The issue also can be very random. In the attached, you can see an alert at 4:27 sent twice whilst the alert at 4:47 for the same service(in purple) is sent only once. Not sure how to explain this.
Any ideas?
Moderator Note: File deleted at user request
Re: Multiple email notifications
Posted: Wed Oct 24, 2012 7:07 am
by scottwilkerson
I have a feeling this could be caused by duplicate versions of a cronjob running in parrallel
Can you run the following
Not sure if we can catch it happening, but worth a try...
Re: Multiple email notifications
Posted: Wed Oct 24, 2012 8:20 am
by aap
Thanks.
Please see results below:
nagios 30894 30883 0 14:19 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 30896 30894 1 14:19 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 30904 30875 0 14:19 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 30909 30904 1 14:19 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
root 31743 29808 0 14:19 pts/0 00:00:00 grep eventman
Re: Multiple email notifications
Posted: Wed Oct 24, 2012 11:25 am
by lmiltchev
I believe we are moving to the right direction... Please, run the following commands, and post the output:
Code: Select all
cat /etc/cron.d/nagiosxi
crontab -u nagios -l
Re: Multiple email notifications
Posted: Thu Oct 25, 2012 2:11 am
by aap
See both outputs below:
cat /etc/cron.d/nagiosxi
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi
# Backup MySQL & PostgreSQL Databases
0 7 * * * root /root/scripts/automysqlbackup
0 8 * * * root /root/scripts/autopostgresqlbackup
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01 * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
crontab -u nagios -l
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01 * * * * /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1