Multiple email notifications

[scottwilkerson]

These both look normal.

I am sure you have tried this, but if you send a regular email to this list is it duplicated?

The reason I ask, is that you have already shown that nagios is only sending one notification in the log... Can you verify on the listserv the received emails?

[aap]

Thanks.

Regularv emails are not dupliacted - they are only sent once. This behaviour is very odd. I am wondering if I will need to do a database repair.

The reason I say this is that the issue only happens to a few hosts. There are hosts that have multiple notification groups and still report in the correct manner. I really don't know where else to look.

[scottwilkerson]

It certainly wouldn't hurt...

you can run

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagios
echo "vacuum;vacuum analyse;vacuum full;"|psql nagiosxi postgres

[aap]

Hi,

I have ran the database repair and the additional command that you specified with the results below:

VACUUM
VACUUM
VACUUM

Unfortunately, it has not made a difference. We are still receiving multiple alerts for "some" services.

[scottwilkerson]

Is it the same few hosts? Are the notifications absolutely identical?

I'm wondering if we don't somehow have these hosts duplicated.

[aap]

Thanks.

I am starting to see more hosts exhibiting the same behaviuor. We monitor over 8300 services so it is getting a bit frustrating.

I have manually checked the files and there is no duplication anywhere.

The issue also can be very random. In the attached, you can see an alert at 4:27 sent twice whilst the alert at 4:47 for the same service(in purple) is sent only once. Not sure how to explain this.

Any ideas?

Moderator Note: File deleted at user request

[scottwilkerson]

I have a feeling this could be caused by duplicate versions of a cronjob running in parrallel

Can you run the following

Code: Select all

ps -ef |grep eventman

Not sure if we can catch it happening, but worth a try...

[aap]

Thanks.

Please see results below:

nagios 30894 30883 0 14:19 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 30896 30894 1 14:19 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 30904 30875 0 14:19 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 30909 30904 1 14:19 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
root 31743 29808 0 14:19 pts/0 00:00:00 grep eventman

[lmiltchev]

I believe we are moving to the right direction... Please, run the following commands, and post the output:

Code: Select all

cat /etc/cron.d/nagiosxi
crontab -u nagios -l

[aap]

See both outputs below:

cat /etc/cron.d/nagiosxi
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0 7 * * * root /root/scripts/automysqlbackup
0 8 * * * root /root/scripts/autopostgresqlbackup

* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01 * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1


crontab -u nagios -l
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
* * * * * /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01 * * * * /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1