Nagios service restarts and send duplicate alerts

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Johnsmit
Posts: 95
Joined: Thu Apr 19, 2018 2:03 pm

Nagios service restarts and send duplicate alerts

Post by Johnsmit »

Hi,

we have a hot standby failover nagiosXI with 5.5.6 runing. we stopped nagios service to stop duplicate monitoring but today the nagios service has been started by audit manager and sending duplicate alerts. for every alert it keeps sending 4 alerts at the same time?


here is what i found on logs, who is this auditmanager, why he wakes up nagios? and why am getting 4 alerts instead one alert from filover server.
Dec 4 13:18:16 Server_Server_Hostname auditmanager: Received wakeup signal before sleep finished
Dec 4 13:18:16 Server_Server_Hostname auditmanager: Received wakeup signal before sleep finished
Dec 4 13:18:16 Server_Server_Hostname auditmanager: Received wakeup signal before sleep finished
Dec 4 13:20:06 Server_Server_Hostname systemd: Starting Nagios Core 4.4.2...
Dec 4 13:20:06 Server_Server_Hostname nagios: Nagios Core 4.4.2
Dec 4 13:20:06 Server_Server_Hostname nagios: Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Dec 4 13:20:06 Server_Server_Hostname nagios: Copyright (c) 1999-2009 Ethan Galstad
Dec 4 13:20:06 Server_Server_Hostname nagios: Last Modified: 2018-08-16
Dec 4 13:20:06 Server_Server_Hostname nagios: License: GPL
Dec 4 13:20:06 Server_Server_Hostname nagios: Website: https://www.nagios.org
Dec 4 13:20:06 Server_Server_Hostname nagios: Reading configuration data...
Dec 4 13:20:06 Server_Server_Hostname nagios: Read main config file okay...
Dec 4 13:20:06 Server_Server_Hostname nagios: Warning: Duplicate definition found for service 'Total Processes' on host 'client_server_name' (config file '/usr/local/nagios/etc/services/client_server_name.cfg', starting on line 175)
Dec 4 13:20:06 Server_Server_Hostname nagios: Read object config files okay...
Dec 4 13:20:06 Server_Server_Hostname nagios: Running pre-flight check on configuration data...
Dec 4 13:20:06 Server_Server_Hostname nagios: Checking objects...
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 96 services.
Dec 4 13:20:06 Server_Server_Hostname nagios: Warning: Host 'client_server_name' has no default contacts or contactgroups defined!
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 7 hosts.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 16 host groups.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 0 service groups.
Dec 4 13:20:06 Server_Server_Hostname nagios: Warning: Contact 'datasbase testing' has no service notification time period defined!
Dec 4 13:20:06 Server_Server_Hostname nagios: Warning: Contact 'datasbase testing' has no host notification time period defined!
Dec 4 13:20:06 Server_Server_Hostname nagios: Warning: Host recovery notification option for contact 'datasbase testing' doesn't make any sense - specify down and/or unreachable options as well
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 18 contacts.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 5 contact groups.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 127 commands.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 21 time periods.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 0 host escalations.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 0 service escalations.
Dec 4 13:20:06 Server_Server_Hostname nagios: Checking for circular paths...
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 7 hosts
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 0 service dependencies
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 0 host dependencies
Dec 4 13:20:06 Server_Server_Hostname nagios: Checked 21 timeperiods
Dec 4 13:20:06 Server_Server_Hostname nagios: Checking global event handlers...
Dec 4 13:20:06 Server_Server_Hostname nagios: Checking obsessive compulsive processor commands...
Dec 4 13:20:06 Server_Server_Hostname nagios: Checking misc settings...
Dec 4 13:20:06 Server_Server_Hostname nagios: Total Warnings: 4
Dec 4 13:20:06 Server_Server_Hostname nagios: Total Errors: 0


This i what i have in Nagios Host Notifications.4 correction alerts for 1 hard alert at the same time.

2018-12-04 11:19:39 client_server_name MySQL Server Service Recovery No OK ismail Nagios XI OK: Service mariadb is running!
2018-12-04 11:19:39 client_server_name MySQL Server Service Recovery No OK joseph Nagios XI OK: Service mariadb is running!
2018-12-04 11:19:39 client_server_name MySQL Server Service Recovery No OK mohan Nagios XI OK: Service mariadb is running!
2018-12-04 11:19:39 client_server_name MySQL Server Service Recovery No OK robyn Nagios XI OK: Service mariadb is running!
2018-12-04 10:49:35 client_server_name MySQL Server Service Problem No CRITICAL ismail Nagios XI CRITICAL: Service mariadb is not running!


thanks,
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Nagios service restarts and send duplicate alerts

Post by npolovenko »

Hello, @Johnsmit. Are you using some kind of software to transfer backups from the primary XI server to the backup XI server? The restore_xi script contains the command that starts the Nagios process.
Can you also run ps -ef on the backup XI server and attach the output in a text file?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios service restarts and send duplicate alerts

Post by cdienger »

It looks like the restart is triggered by a McAfee program:

https://kc.mcafee.com/corporate/index?p ... cale=en_US

The 4 messages are being sent to 4 different contacts - ismail,joseph,mohan,robyn. This logging is expected. You can modify the service check and remove contacts if you wish to not send to all 4 people.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Johnsmit
Posts: 95
Joined: Thu Apr 19, 2018 2:03 pm

Re: Nagios service restarts and send duplicate alerts

Post by Johnsmit »

npolovenko wrote:Hello, @Johnsmit. Are you using some kind of software to transfer backups from the primary XI server to the backup XI server? The restore_xi script contains the command that starts the Nagios process.
Can you also run ps -ef on the backup XI server and attach the output in a text file?
Hi, here is the output of ps -ef on backup server. I don't use any software to transfer backups from primary to secondary. wqe have master-master replication between databases and cron job to sync files.

Thanks,
You do not have the required permissions to view the files attached to this post.
Johnsmit
Posts: 95
Joined: Thu Apr 19, 2018 2:03 pm

Re: Nagios service restarts and send duplicate alerts

Post by Johnsmit »

cdienger wrote:It looks like the restart is triggered by a McAfee program:

https://kc.mcafee.com/corporate/index?p ... cale=en_US

The 4 messages are being sent to 4 different contacts - ismail,joseph,mohan,robyn. This logging is expected. You can modify the service check and remove contacts if you wish to not send to all 4 people.
The logging is fine to send 4 alerts to 4 persons, but each person is getting 4 alerts for a single service at a time.
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios service restarts and send duplicate alerts

Post by cdienger »

The ps output does show McAfee software on the machine it looks like the question of where those messages are are coming from is confirmed.

I can't think of a reason why it would send 4 emails - do you see 4 notification messages for each account if you look in /usr/local/nagios/var/nagios.log? What do your mail settings look like under Admin > System Config > Manage Email Setting ? Do you get 4 emails if you use the "send a test email" button on that screen? Does the email server show 4 emails coming from the XI machine and can we confirm that the duplication isn't happening elsewhere?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Johnsmit
Posts: 95
Joined: Thu Apr 19, 2018 2:03 pm

Re: Nagios service restarts and send duplicate alerts

Post by Johnsmit »

cdienger wrote:The ps output does show McAfee software on the machine it looks like the question of where those messages are are coming from is confirmed.

I can't think of a reason why it would send 4 emails - do you see 4 notification messages for each account if you look in /usr/local/nagios/var/nagios.log? What do your mail settings look like under Admin > System Config > Manage Email Setting ? Do you get 4 emails if you use the "send a test email" button on that screen? Does the email server show 4 emails coming from the XI machine and can we confirm that the duplication isn't happening elsewhere?
Good Morning,

i tested email settings am getting only one email whereas am getting double notifications. when i checked logs, i see one alert and 2 correction email for that alert.

Notification alert:
Dec 4 10:48:53 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;1;CRITICAL: Service mariadb is not running!
Dec 4 10:48:53 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;1;CRITICAL: Service mariadb is not running!
Dec 4 10:48:57 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;2;CRITICAL: Service mariadb is not running!
Dec 4 10:48:57 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;2;CRITICAL: Service mariadb is not running!
Dec 4 10:49:11 Nagios_master xinetd[26512]: START: nrpe pid=14809 from=10.24.158.40
Dec 4 10:49:11 Nagios_master xinetd[26512]: EXIT: nrpe status=0 pid=14809 duration=0(sec)
Dec 4 10:49:32 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;3;CRITICAL: Service mariadb is not running!
Dec 4 10:49:32 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;3;CRITICAL: Service mariadb is not running!
Dec 4 10:49:34 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;4;CRITICAL: Service mariadb is not running!
Dec 4 10:49:34 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;SOFT;4;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: ismail;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: joseph;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: robyn;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;HARD;5;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: ismail;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: joseph;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE NOTIFICATION: robyn;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 4 10:49:35 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;HARD;5;CRITICAL: Service mariadb is not running!


Correction alert:

Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: ismail;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: joseph;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: robyn;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE ALERT: client server;MySQL Server;OK;HARD;1;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: ismail;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: joseph;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 4 11:19:39 Nagios_master nagios: SERVICE NOTIFICATION: robyn;client server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!

I checked today
Notification:
Dec 6 12:04:41 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 6 12:04:41 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client_server;MySQL Server;CRITICAL;xi_service_notification_handler;CRITICAL: Service mariadb is not running!
Dec 6 12:04:41 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;HARD;5;CRITICAL: Service mariadb is not running!
Dec 6 12:04:41 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;CRITICAL;HARD;5;CRITICAL: Service mariadb is not running!
Correction Alert:
Dec 6 12:19:41 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client_server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 6 12:19:41 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;OK;HARD;1;OK: Service mariadb is running!
Dec 6 12:19:41 Nagios_master nagios: SERVICE NOTIFICATION: mohan;client_server;MySQL Server;OK;xi_service_notification_handler;OK: Service mariadb is running!
Dec 6 12:19:41 Nagios_master nagios: SERVICE ALERT: client_server;MySQL Server;OK;HARD;1;OK: Service mariadb is running!


why am i getting 2 correction alerts for one notification. Correction out of 4 alerts 2 are from primary and 2 are from secondary.
Do i need to correct any settings in CCM so as to get one correction alert for 1 hard state notification.


Thanks,
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios service restarts and send duplicate alerts

Post by cdienger »

There may be duplicate notification methods selected. Edit the user(s) under Configure > Core Config Manager > Alerts > Contacts > *select contact* > Alert Settings > Manage Host Notification Commands & Manage Service Notification Commands. Usually there should only be one command for each - xi_host_notification_handler & xi_service_notification_handler respectively.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Johnsmit
Posts: 95
Joined: Thu Apr 19, 2018 2:03 pm

Re: Nagios service restarts and send duplicate alerts

Post by Johnsmit »

cdienger wrote:There may be duplicate notification methods selected. Edit the user(s) under Configure > Core Config Manager > Alerts > Contacts > *select contact* > Alert Settings > Manage Host Notification Commands & Manage Service Notification Commands. Usually there should only be one command for each - xi_host_notification_handler & xi_service_notification_handler respectively.
Still, am getting 2 correction alerts for a single notification, even after the changes made as you suggested. Is there any another setting to setup.

Thanks,
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios service restarts and send duplicate alerts

Post by cdienger »

Please PM me a profile which can be generated under Admin > System Config > System Profile > Download Profile.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Locked