NagiosXI email alerts are sent with huge delay

[Sampath.Basireddy]

Hello There,

Since last few days, am noticing huge delay in the email alerts from NagiosXI. The delay is like more than a day or 2.

When I send a test email, that comes immediately.

Recently upgraded to NagiosXI 5.5.10 and I suspect we may have started noticing this issue after upgrade.

Here is a test scenario I tried few mins back. Changed the threshold for a disk usage check to trigger a critical alert.

Few mins after applying the config, I can see the "/usr/local/nagios/var/nagios.log" file updated with the Service Alert & Service Notification. But neither I see the service check turning critical on XI Console, nor I see an email in my Inbox.

Been more than 15mins applied the config, I still don't see the service check turning critical.

Please suggest.

Thanks.

[tgriep]

Is the Database Backend (ndo2db) running on the system or a duplicated nagios process?
If it is not running, that could be one cause of the issue you are describing as well as a duplicated process.
Run this as root to stop and restart the processes.

Code: Select all

service nagios stop
service ndo2db stop
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
service ndo2db start
service nagios start

Give the system 10 minutes to run and check to see if the Notifications are running.

If it does not work, can you upload a System Profile to the post?

[Sampath.Basireddy]

No luck, sent you the profile via PM.

The Database Backend (ndo2db) is running on the system.

Code: Select all

root@nagios1:[~]: systemctl status ndo2db.service
● ndo2db.service - Nagios Data Out Daemon
   Loaded: loaded (/usr/lib/systemd/system/ndo2db.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-03-12 18:11:46 EDT; 5h 3min ago
     Docs: http://www.nagios.org/documentation
  Process: 13177 ExecStopPost=/bin/rm -f /usr/local/nagios/var/ndo2db.lock (code=exited, status=0/SUCCESS)
  Process: 13928 ExecStartPre=/bin/rm -f /usr/local/nagios/var/ndo.sock (code=exited, status=0/SUCCESS)
  Process: 13926 ExecStartPre=/bin/rm -f /usr/local/nagios/var/ndo2db.lock (code=exited, status=0/SUCCESS)
 Main PID: 13933 (ndo2db)
    Tasks: 3
   Memory: 7.7M
   CGroup: /system.slice/ndo2db.service
           ├─13933 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
           ├─14390 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
           └─14391 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f

Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming timedevents.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming systemcommands.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming servicechecks.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming hostchecks.
Mar 12 23:13:26 nagios1.ux.corp.local ndo2db[14391]: Trimming eventhandlers.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming timedevents.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming systemcommands.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming servicechecks.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming hostchecks.
Mar 12 23:14:28 nagios1.ux.corp.local ndo2db[14391]: Trimming eventhandlers.
root@nagios1:[~]:

Code: Select all

root@nagios1:[~]: ps -ef | grep -i ndo2db
root      2375 18750  0 23:15 pts/0    00:00:00 grep --color=auto -i ndo2db
nagios   13933     1  0 18:11 ?        00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
nagios   14390 13933  0 18:12 ?        00:00:30 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
nagios   14391 14390  3 18:12 ?        00:10:33 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg -f
root@nagios1:[~]:

[tgriep]

In the MYSQL database log, it shows that the partition that is holding the MYSQL database ran out of space and that could of corrupted the database and cause the delay.
Lets stop the processes, truncate the temporary data from the database, repair it and start the processes.
Run the following as root.

Code: Select all

service crond stop
service nagios stop
service ndo2db stop
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
mysqlcheck -f -r -u root -pnagiosxi --all-databases
service mysqld restart
service httpd restart
service ndo2db start
service nagios start
service crond start

Then, wait for the system to generate an Notification and see if it gets sent and received is a timely fashion as well as the GUI updating with the correct data.

[Sampath.Basireddy]

After following the steps, I can see the status is changing quickly and alert too is coming on time.

But when I ran the check commands, I saw this message "The storage engine for the table doesn't support repair" for lot of tables

Here is the complete list:

Code: Select all

mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.servers                                      OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
nagios.nagios_acknowledgements                     OK
nagios.nagios_commands                             OK
nagios.nagios_commenthistory                       OK
nagios.nagios_comments                             OK
nagios.nagios_configfiles                          OK
nagios.nagios_configfilevariables                  OK
nagios.nagios_conninfo                             OK
nagios.nagios_contact_addresses                    OK
nagios.nagios_contact_notificationcommands         OK
nagios.nagios_contactgroup_members                 OK
nagios.nagios_contactgroups                        OK
nagios.nagios_contactnotificationmethods           OK
nagios.nagios_contactnotifications                 OK
nagios.nagios_contacts                             OK
nagios.nagios_contactstatus                        OK
nagios.nagios_customvariables                      OK
nagios.nagios_customvariablestatus                 OK
nagios.nagios_dbversion                            OK
nagios.nagios_downtimehistory                      OK
nagios.nagios_eventhandlers                        OK
nagios.nagios_externalcommands                     OK
nagios.nagios_flappinghistory                      OK
nagios.nagios_host_contactgroups                   OK
nagios.nagios_host_contacts                        OK
nagios.nagios_host_parenthosts                     OK
nagios.nagios_hostchecks                           OK
nagios.nagios_hostdependencies                     OK
nagios.nagios_hostescalation_contactgroups         OK
nagios.nagios_hostescalation_contacts              OK
nagios.nagios_hostescalations                      OK
nagios.nagios_hostgroup_members                    OK
nagios.nagios_hostgroups                           OK
nagios.nagios_hosts                                OK
nagios.nagios_hoststatus                           OK
nagios.nagios_instances                            OK
nagios.nagios_logentries                           OK
nagios.nagios_notifications                        OK
nagios.nagios_objects                              OK
nagios.nagios_processevents                        OK
nagios.nagios_programstatus                        OK
nagios.nagios_runtimevariables                     OK
nagios.nagios_scheduleddowntime                    OK
nagios.nagios_service_contactgroups                OK
nagios.nagios_service_contacts                     OK
nagios.nagios_service_parentservices               OK
nagios.nagios_servicechecks                        OK
nagios.nagios_servicedependencies                  OK
nagios.nagios_serviceescalation_contactgroups      OK
nagios.nagios_serviceescalation_contacts           OK
nagios.nagios_serviceescalations                   OK
nagios.nagios_servicegroup_members                 OK
nagios.nagios_servicegroups                        OK
nagios.nagios_services                             OK
nagios.nagios_servicestatus                        OK
nagios.nagios_statehistory                         OK
nagios.nagios_systemcommands                       OK
nagios.nagios_timedeventqueue                      OK
nagios.nagios_timedevents                          OK
nagios.nagios_timeperiod_timeranges                OK
nagios.nagios_timeperiods                          OK
nagiosql.tbl_command                               OK
nagiosql.tbl_contact                               OK
nagiosql.tbl_contactgroup                          OK
nagiosql.tbl_contacttemplate                       OK
nagiosql.tbl_domain                                OK
nagiosql.tbl_host                                  OK
nagiosql.tbl_hostdependency                        OK
nagiosql.tbl_hostescalation                        OK
nagiosql.tbl_hostextinfo                           OK
nagiosql.tbl_hostgroup                             OK
nagiosql.tbl_hosttemplate                          OK
nagiosql.tbl_info                                  OK
nagiosql.tbl_lnkContactToCommandHost               OK
nagiosql.tbl_lnkContactToCommandService            OK
nagiosql.tbl_lnkContactToContactgroup              OK
nagiosql.tbl_lnkContactToContacttemplate           OK
nagiosql.tbl_lnkContactToVariabledefinition        OK
nagiosql.tbl_lnkContactgroupToContact              OK
nagiosql.tbl_lnkContactgroupToContactgroup         OK
nagiosql.tbl_lnkContacttemplateToCommandHost       OK
nagiosql.tbl_lnkContacttemplateToCommandService    OK
nagiosql.tbl_lnkContacttemplateToContactgroup      OK
nagiosql.tbl_lnkContacttemplateToContacttemplate   OK
nagiosql.tbl_lnkContacttemplateToVariabledefinition OK
nagiosql.tbl_lnkHostToContact                      OK
nagiosql.tbl_lnkHostToContactgroup                 OK
nagiosql.tbl_lnkHostToHost                         OK
nagiosql.tbl_lnkHostToHostgroup                    OK
nagiosql.tbl_lnkHostToHosttemplate                 OK
nagiosql.tbl_lnkHostToVariabledefinition           OK
nagiosql.tbl_lnkHostdependencyToHost_DH            OK
nagiosql.tbl_lnkHostdependencyToHost_H             OK
nagiosql.tbl_lnkHostdependencyToHostgroup_DH       OK
nagiosql.tbl_lnkHostdependencyToHostgroup_H        OK
nagiosql.tbl_lnkHostescalationToContact            OK
nagiosql.tbl_lnkHostescalationToContactgroup       OK
nagiosql.tbl_lnkHostescalationToHost               OK
nagiosql.tbl_lnkHostescalationToHostgroup          OK
nagiosql.tbl_lnkHostgroupToHost                    OK
nagiosql.tbl_lnkHostgroupToHostgroup               OK
nagiosql.tbl_lnkHosttemplateToContact              OK
nagiosql.tbl_lnkHosttemplateToContactgroup         OK
nagiosql.tbl_lnkHosttemplateToHost                 OK
nagiosql.tbl_lnkHosttemplateToHostgroup            OK
nagiosql.tbl_lnkHosttemplateToHosttemplate         OK
nagiosql.tbl_lnkHosttemplateToVariabledefinition   OK
nagiosql.tbl_lnkServiceToContact                   OK
nagiosql.tbl_lnkServiceToContactgroup              OK
nagiosql.tbl_lnkServiceToHost                      OK
nagiosql.tbl_lnkServiceToHostgroup                 OK
nagiosql.tbl_lnkServiceToServicegroup              OK
nagiosql.tbl_lnkServiceToServicetemplate           OK
nagiosql.tbl_lnkServiceToVariabledefinition        OK
nagiosql.tbl_lnkServicedependencyToHost_DH         OK
nagiosql.tbl_lnkServicedependencyToHost_H          OK
nagiosql.tbl_lnkServicedependencyToHostgroup_DH    OK
nagiosql.tbl_lnkServicedependencyToHostgroup_H     OK
nagiosql.tbl_lnkServicedependencyToService_DS      OK
nagiosql.tbl_lnkServicedependencyToService_S       OK
nagiosql.tbl_lnkServiceescalationToContact         OK
nagiosql.tbl_lnkServiceescalationToContactgroup    OK
nagiosql.tbl_lnkServiceescalationToHost            OK
nagiosql.tbl_lnkServiceescalationToHostgroup       OK
nagiosql.tbl_lnkServiceescalationToService         OK
nagiosql.tbl_lnkServicegroupToService              OK
nagiosql.tbl_lnkServicegroupToServicegroup         OK
nagiosql.tbl_lnkServicetemplateToContact           OK
nagiosql.tbl_lnkServicetemplateToContactgroup      OK
nagiosql.tbl_lnkServicetemplateToHost              OK
nagiosql.tbl_lnkServicetemplateToHostgroup         OK
nagiosql.tbl_lnkServicetemplateToServicegroup      OK
nagiosql.tbl_lnkServicetemplateToServicetemplate   OK
nagiosql.tbl_lnkServicetemplateToVariabledefinition OK
nagiosql.tbl_lnkTimeperiodToTimeperiod             OK
nagiosql.tbl_logbook                               OK
nagiosql.tbl_mainmenu                              OK
nagiosql.tbl_permission
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_permission_inactive
note     : The storage engine for the table doesn't support repair
nagiosql.tbl_service                               OK
nagiosql.tbl_servicedependency                     OK
nagiosql.tbl_serviceescalation                     OK
nagiosql.tbl_serviceextinfo                        OK
nagiosql.tbl_servicegroup                          OK
nagiosql.tbl_servicetemplate                       OK
nagiosql.tbl_session                               OK
nagiosql.tbl_session_locks                         OK
nagiosql.tbl_settings                              OK
nagiosql.tbl_submenu                               OK
nagiosql.tbl_timedefinition                        OK
nagiosql.tbl_timeperiod                            OK
nagiosql.tbl_user                                  OK
nagiosql.tbl_variabledefinition                    OK
nagiosxi.xi_auditlog
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_auth_tokens
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_cmp_trapdata_log
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_commands
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_eventqueue                             OK
nagiosxi.xi_events
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_incidents
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_meta
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_options
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_sessions
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_sysstat
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_usermeta
note     : The storage engine for the table doesn't support repair
nagiosxi.xi_users
note     : The storage engine for the table doesn't support repair

[scottwilkerson]

That is normal on the nagiosxi database and not an error

[Sampath.Basireddy]

Great, Thank You @tgriep & @scottwilkerson.

Please close this topic.

[scottwilkerson]

Great, Thank You @tgriep & @scottwilkerson.

Please close this topic.

Great!

Locking thread