Page 1 of 1
NagiosXI email alerts are sent with huge delay
Posted: Tue Mar 12, 2019 3:31 pm
by Sampath.Basireddy
Hello There,
Since last few days, am noticing huge delay in the email alerts from NagiosXI. The delay is like more than a day or 2.
When I send a test email, that comes immediately.
Recently upgraded to NagiosXI 5.5.10 and I suspect we may have started noticing this issue after upgrade.
Here is a test scenario I tried few mins back. Changed the threshold for a disk usage check to trigger a critical alert.
Few mins after applying the config, I can see the "/usr/local/nagios/var/nagios.log" file updated with the Service Alert & Service Notification. But neither I see the service check turning critical on XI Console, nor I see an email in my Inbox.
Been more than 15mins applied the config, I still don't see the service check turning critical.
Please suggest.
Thanks.
Re: NagiosXI email alerts are sent with huge delay
Posted: Tue Mar 12, 2019 4:32 pm
by tgriep
Is the Database Backend (ndo2db) running on the system or a duplicated nagios process?
If it is not running, that could be one cause of the issue you are describing as well as a duplicated process.
Run this as root to stop and restart the processes.
Code: Select all
service nagios stop
service ndo2db stop
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
service ndo2db start
service nagios start
Give the system 10 minutes to run and check to see if the Notifications are running.
If it does not work, can you upload a System Profile to the post?
Re: NagiosXI email alerts are sent with huge delay
Posted: Tue Mar 12, 2019 10:06 pm
by Sampath.Basireddy
No luck, sent you the profile via PM.
The Database Backend (ndo2db) is running on the system.
Code: Select all
root@nagios1:[~]: systemctl status ndo2db.service
● ndo2db.service - Nagios Data Out Daemon
Loaded: loaded (/usr/lib/systemd/system/ndo2db.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2019-03-12 18:11:46 EDT; 5h 3min ago
Docs: http://www.nagios.org/documentation
Process: 13177 ExecStopPost=/bin/rm -f /usr/local/nagios/var/ndo2db.lock (code=exited, status=0/SUCCESS)
Process: 13928 ExecStartPre=/bin/rm -f /usr/local/nagios/var/ndo.sock (code=exited, status=0/SUCCESS)
Process: 13926 ExecStartPre=/bin/rm -f /usr/local/nagios/var/ndo2db.lock (code=exited, status=0/SUCCESS)
Main PID: 13933 (ndo2db)
Tasks: 3
Memory: 7.7M
CGroup: /system.slice/ndo2db.service
├─13933 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
├─14390 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
└─14391 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming timedevents.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming systemcommands.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming servicechecks.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming hostchecks.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming eventhandlers.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming timedevents.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming systemcommands.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming servicechecks.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming hostchecks.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming eventhandlers.
root@nagios1:[~]:
Code: Select all
root@nagios1:[~]: ps -ef | grep -i ndo2db
root 2375 18750 0 23:15 pts/0 00:00:00 grep --color=auto -i ndo2db
nagios 13933 1 0 18:11 ? 00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
nagios 14390 13933 0 18:12 ? 00:00:30 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
nagios 14391 14390 3 18:12 ? 00:10:33 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
root@nagios1:[~]:
Re: NagiosXI email alerts are sent with huge delay
Posted: Wed Mar 13, 2019 9:14 am
by tgriep
In the MYSQL database log, it shows that the partition that is holding the MYSQL database ran out of space and that could of corrupted the database and cause the delay.
Lets stop the processes, truncate the temporary data from the database, repair it and start the processes.
Run the following as root.
Code: Select all
service crond stop
service nagios stop
service ndo2db stop
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
mysqlcheck -f -r -u root -pnagiosxi --all-databases
service mysqld restart
service httpd restart
service ndo2db start
service nagios start
service crond start
Then, wait for the system to generate an Notification and see if it gets sent and received is a timely fashion as well as the GUI updating with the correct data.
Re: NagiosXI email alerts are sent with huge delay
Posted: Wed Mar 13, 2019 6:19 pm
by Sampath.Basireddy
After following the steps, I can see the status is changing quickly and alert too is coming on time.
But when I ran the check commands, I saw this message "The storage engine for the table doesn't support repair" for lot of tables
Here is the complete list:
Code: Select all
mysql.columns_priv OK
mysql.db OK
mysql.event OK
mysql.func OK
mysql.help_category OK
mysql.help_keyword OK
mysql.help_relation OK
mysql.help_topic OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.proxies_priv OK
mysql.servers OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user OK
nagios.nagios_acknowledgements OK
nagios.nagios_commands OK
nagios.nagios_commenthistory OK
nagios.nagios_comments OK
nagios.nagios_configfiles OK
nagios.nagios_configfilevariables OK
nagios.nagios_conninfo OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands OK
nagios.nagios_contactgroup_members OK
nagios.nagios_contactgroups OK
nagios.nagios_contactnotificationmethods OK
nagios.nagios_contactnotifications OK
nagios.nagios_contacts OK
nagios.nagios_contactstatus OK
nagios.nagios_customvariables OK
nagios.nagios_customvariablestatus OK
nagios.nagios_dbversion OK
nagios.nagios_downtimehistory OK
nagios.nagios_eventhandlers OK
nagios.nagios_externalcommands OK
nagios.nagios_flappinghistory OK
nagios.nagios_host_contactgroups OK
nagios.nagios_host_contacts OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members OK
nagios.nagios_hostgroups OK
nagios.nagios_hosts OK
nagios.nagios_hoststatus OK
nagios.nagios_instances OK
nagios.nagios_logentries OK
nagios.nagios_notifications OK
nagios.nagios_objects OK
nagios.nagios_processevents OK
nagios.nagios_programstatus OK
nagios.nagios_runtimevariables OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups OK
nagios.nagios_service_contacts OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members OK
nagios.nagios_servicegroups OK
nagios.nagios_services OK
nagios.nagios_servicestatus OK
nagios.nagios_statehistory OK
nagios.nagios_systemcommands OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges OK
nagios.nagios_timeperiods OK
nagiosql.tbl_command OK
nagiosql.tbl_contact OK
nagiosql.tbl_contactgroup OK
nagiosql.tbl_contacttemplate OK
nagiosql.tbl_domain OK
nagiosql.tbl_host OK
nagiosql.tbl_hostdependency OK
nagiosql.tbl_hostescalation OK
nagiosql.tbl_hostextinfo OK
nagiosql.tbl_hostgroup OK
nagiosql.tbl_hosttemplate OK
nagiosql.tbl_info OK
nagiosql.tbl_lnkContactToCommandHost OK
nagiosql.tbl_lnkContactToCommandService OK
nagiosql.tbl_lnkContactToContactgroup OK
nagiosql.tbl_lnkContactToContacttemplate OK
nagiosql.tbl_lnkContactToVariabledefinition OK
nagiosql.tbl_lnkContactgroupToContact OK
nagiosql.tbl_lnkContactgroupToContactgroup OK
nagiosql.tbl_lnkContacttemplateToCommandHost OK
nagiosql.tbl_lnkContacttemplateToCommandService OK
nagiosql.tbl_lnkContacttemplateToContactgroup OK
nagiosql.tbl_lnkContacttemplateToContacttemplate OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact OK
nagiosql.tbl_lnkHostToContactgroup OK
nagiosql.tbl_lnkHostToHost OK
nagiosql.tbl_lnkHostToHostgroup OK
nagiosql.tbl_lnkHostToHosttemplate OK
nagiosql.tbl_lnkHostToVariabledefinition OK
nagiosql.tbl_lnkHostdependencyToHost_DH OK
nagiosql.tbl_lnkHostdependencyToHost_H OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H OK
nagiosql.tbl_lnkHostescalationToContact OK
nagiosql.tbl_lnkHostescalationToContactgroup OK
nagiosql.tbl_lnkHostescalationToHost OK
nagiosql.tbl_lnkHostescalationToHostgroup OK
nagiosql.tbl_lnkHostgroupToHost OK
nagiosql.tbl_lnkHostgroupToHostgroup OK
nagiosql.tbl_lnkHosttemplateToContact OK
nagiosql.tbl_lnkHosttemplateToContactgroup OK
nagiosql.tbl_lnkHosttemplateToHost OK
nagiosql.tbl_lnkHosttemplateToHostgroup OK
nagiosql.tbl_lnkHosttemplateToHosttemplate OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition OK
nagiosql.tbl_lnkServiceToContact OK
nagiosql.tbl_lnkServiceToContactgroup OK
nagiosql.tbl_lnkServiceToHost OK
nagiosql.tbl_lnkServiceToHostgroup OK
nagiosql.tbl_lnkServiceToServicegroup OK
nagiosql.tbl_lnkServiceToServicetemplate OK
nagiosql.tbl_lnkServiceToVariabledefinition OK
nagiosql.tbl_lnkServicedependencyToHost_DH OK
nagiosql.tbl_lnkServicedependencyToHost_H OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H OK
nagiosql.tbl_lnkServicedependencyToService_DS OK
nagiosql.tbl_lnkServicedependencyToService_S OK
nagiosql.tbl_lnkServiceescalationToContact OK
nagiosql.tbl_lnkServiceescalationToContactgroup OK
nagiosql.tbl_lnkServiceescalationToHost OK
nagiosql.tbl_lnkServiceescalationToHostgroup OK
nagiosql.tbl_lnkServiceescalationToService OK
nagiosql.tbl_lnkServicegroupToService OK
nagiosql.tbl_lnkServicegroupToServicegroup OK
nagiosql.tbl_lnkServicetemplateToContact OK
nagiosql.tbl_lnkServicetemplateToContactgroup OK
nagiosql.tbl_lnkServicetemplateToHost OK
nagiosql.tbl_lnkServicetemplateToHostgroup OK
nagiosql.tbl_lnkServicetemplateToServicegroup OK
nagiosql.tbl_lnkServicetemplateToServicetemplate OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod OK
nagiosql.tbl_logbook OK
nagiosql.tbl_mainmenu OK
nagiosql.tbl_permission
note : The storage engine for the table doesn't support repair
nagiosql.tbl_permission_inactive
note : The storage engine for the table doesn't support repair
nagiosql.tbl_service OK
nagiosql.tbl_servicedependency OK
nagiosql.tbl_serviceescalation OK
nagiosql.tbl_serviceextinfo OK
nagiosql.tbl_servicegroup OK
nagiosql.tbl_servicetemplate OK
nagiosql.tbl_session OK
nagiosql.tbl_session_locks OK
nagiosql.tbl_settings OK
nagiosql.tbl_submenu OK
nagiosql.tbl_timedefinition OK
nagiosql.tbl_timeperiod OK
nagiosql.tbl_user OK
nagiosql.tbl_variabledefinition OK
nagiosxi.xi_auditlog
note : The storage engine for the table doesn't support repair
nagiosxi.xi_auth_tokens
note : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata
note : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata_log
note : The storage engine for the table doesn't support repair
nagiosxi.xi_commands
note : The storage engine for the table doesn't support repair
nagiosxi.xi_eventqueue OK
nagiosxi.xi_events
note : The storage engine for the table doesn't support repair
nagiosxi.xi_incidents
note : The storage engine for the table doesn't support repair
nagiosxi.xi_meta
note : The storage engine for the table doesn't support repair
nagiosxi.xi_options
note : The storage engine for the table doesn't support repair
nagiosxi.xi_sessions
note : The storage engine for the table doesn't support repair
nagiosxi.xi_sysstat
note : The storage engine for the table doesn't support repair
nagiosxi.xi_usermeta
note : The storage engine for the table doesn't support repair
nagiosxi.xi_users
note : The storage engine for the table doesn't support repair
Re: NagiosXI email alerts are sent with huge delay
Posted: Thu Mar 14, 2019 7:06 am
by scottwilkerson
That is normal on the nagiosxi database and not an error
Re: NagiosXI email alerts are sent with huge delay
Posted: Thu Mar 14, 2019 1:21 pm
by Sampath.Basireddy
Great, Thank You
@tgriep &
@scottwilkerson.
Please close this topic.
Re: NagiosXI email alerts are sent with huge delay
Posted: Thu Mar 14, 2019 1:27 pm
by scottwilkerson