Unable to start Monitoring Engine

[rks]

My NagiosXI installation was rebooted. After reboot, I find that the
monitoring service and performance grapher are no longer running (red exclamation mark).

When I attempt to start it manually (off the GUI gear icon), it yields a message that
"the command could not be processed in a timely manner").

Pl help!

rk

[rks]

Btw, I see some ominous messages in the log:

Code: Select all

Aug 21 18:41:38 ctvmanager ndo2db: mysql_error: 'Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired'
Aug 21 18:41:38 ctvmanager ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_systemcommands SET instance_id='1', start_time=FROM_UNIXTIME(1377124898), start_time_usec='126691', end_time=FROM_UNIXTIME(1377124898), end_time_usec='130434', command_line='/bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/1377124898\.perfdata\.service', timeout='5', early_timeout='0', execution_time='0.003000', return_code='0', output='', long_output='' ON DUPLICATE KEY UPDATE instance_id='1', start_time=FROM_UNIXTIME(1377124898), start_time_usec='126691', end_time=FROM_UNIXTIME(1377124898), end_time_usec='130434', command_line='/bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/1377124898\.perfdata\.service', timeout='5', early_timeout='0', execution_time='0.003000', return_code='0', output='', long_output='''
Aug 21 18:41:38 ctvmanager ndo2db: mysql_error: 'Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired'

It appears that the MySQL database is encountering errors. Could you pl suggest a root cause?

thanks,

rk

[scottwilkerson]

Run the following

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagios

[rks]

Thanks. I rant the script and it seemingly executed successfully:
However, I did not stop services (was I supposed to?).

Code: Select all

[root@ctvmanager ~]# /usr/local/nagiosxi/scripts/repairmysql.sh nagios
DATABASE: nagios
TABLE:    
/var/lib/mysql/nagios ~
Stopping mysqld:                                           [  OK  ]
- recovering (with sort) MyISAM-table 'nagios_acknowledgements.MYI'
Data records: 2
- Fixing index 1
          
---------
// .....


- recovering (with sort) MyISAM-table 'nagios_timeperiod_timeranges.MYI'
Data records: 33
- Fixing index 1
- Fixing index 2
Starting mysqld:                                           [  OK  ]
~
 
===============
REPAIR COMPLETE
===============
[root@ctvmanager ~]# 

After the script completed, I see error messages in the log about connections failures to mysql:

Code: Select all

Aug 22 12:00:16 ctvmanager ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Aug 22 12:00:16 ctvmanager ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Aug 22 12:00:16 ctvmanager ndo2db: Error: Could not connect to MySQL database: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Aug 22

And I am still not able to start either the monitoring engine or the performance grapher.

thanks!

rk

[lmiltchev]

Let's see what services do you have running. Run the following commands, and show the output:

Code: Select all

service nagios status
service ndo2db status
service mysqld status
service postgresql status
service crond status
service npcd status

Also, run the following commands, and show the output:

Code: Select all

service mysqld restart
tail /var/log/mysqld.log

[rks]

Thanks! crond had stopped. I restarted it (and mysqld as well) and everything is green.

Code: Select all

[root@ctvmanager ~]# service nagios status
nagios (pid 29192) is running...
[root@ctvmanager ~]# service ndo2db status
ndo2db (pid 1819) is running...
[root@ctvmanager ~]# service mysqld status
mysqld (pid  15539) is running...
[root@ctvmanager ~]# service postgresql status
postmaster (pid  1452) is running...
[root@ctvmanager ~]# service crond status
crond (pid  5863) is running...
[root@ctvmanager ~]# service npcd status
NPCD running (pid 1639).
[root@ctvmanager ~]# service mysqld restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]
[root@ctvmanager ~]# tail /var/log/mysqld.log
130822 15:36:28 [Note] /usr/libexec/mysqld: Shutdown complete

130822 15:36:28 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130822 15:36:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130822 15:36:29  InnoDB: Initializing buffer pool, size = 8.0M
130822 15:36:29  InnoDB: Completed initialization of buffer pool
130822 15:36:29  InnoDB: Started; log sequence number 0 44233
130822 15:36:29 [Note] Event Scheduler: Loaded 0 events
130822 15:36:29 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
[root@ctvmanager ~]# tail -f /var/log/mysqld.log
130822 15:36:28 [Note] /usr/libexec/mysqld: Shutdown complete

130822 15:36:28 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
130822 15:36:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130822 15:36:29  InnoDB: Initializing buffer pool, size = 8.0M
130822 15:36:29  InnoDB: Completed initialization of buffer pool
130822 15:36:29  InnoDB: Started; log sequence number 0 44233
130822 15:36:29 [Note] Event Scheduler: Loaded 0 events
130822 15:36:29 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.67'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

Thanks!

rk