Server restart breaks my tables

[gormank]

Sorry, I posted this before but lost the post...

A shutdown -ry 0 of my 4 servers resulted in the monitoring engine not starting on 3 servers. A check shows missing tables. Actually, the tables are missing after running repairmysql.sh. Whether they we missing before I'm not sure.

Running repairmysql.sh brought back 2 of the 3, but still, one is broken.

# echo 'check table nagios_servicestatu;' | mysql -t -pnagiosxi nagios
+----------------------------+-------+----------+--------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+----------------------------+-------+----------+--------------------------------------------------+
| nagios.nagios_servicestatu | check | Error | Table 'nagios.nagios_servicestatu' doesn't exist |
| nagios.nagios_servicestatu | check | status | Operation failed |
+----------------------------+-------+----------+--------------------------------------------------+

Normally, the following fixes things.

service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios

So most important, how should this be fixed?
Less important is why can't the system be restarted with shutdown?

[root@cocsm2mlnag001 ~]# chkconfig --list | grep -e nagios -e mysql -e postgres
mysqld 0:off 1:off 2:off 3:on 4:off 5:on 6:off
nagios 0:off 1:off 2:off 3:on 4:off 5:on 6:off
nagiosxi 0:off 1:off 2:off 3:on 4:off 5:on 6:off
postgresql 0:off 1:off 2:off 3:on 4:off 5:on 6:off

[lmiltchev]

You have a typo... Instead of this:

Code: Select all

echo 'check table nagios_servicestatu;' | mysql -t -pnagiosxi nagios

run this:

Code: Select all

echo 'check table nagios_servicestatus;' | mysql -t -pnagiosxi nagios

Normally, you need to shut down the server by running:

Code: Select all

shutdown -h now

You can repair the database by running:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases

or

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagios
/usr/local/nagiosxi/scripts/repairmysql.sh nagiosql

You can also run:

Code: Select all

mysqlcheck -r -f -u root -pnagiosxi --databases nagios nagiosql

If none of the above works, you can try:

Code: Select all

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f *
cd /var/lib/mysql/nagiosql
myisamchk -r -f *
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
/usr/bin/php /usr/local/nagiosxi/cron/dbmaint.php

Hope this helps.

[gormank]

You're right. I fixed my DB checks and without the typo and the .MYI at the end of the name the mysql DB looks better. My mistake.

shutdown -h is halt
shutdown -r is restart

Neither should break the DB.

Attached is a file w/ the output of all suggestions in the previous post.

[root@cocsm2mlnag001 nagiosql]# service nagios status
nagios is not running
[root@cocsm2mlnag001 nagiosql]# service nagios start
Starting nagios: done.
[root@cocsm2mlnag001 nagiosql]# service nagios status
nagios is not running

The nagios service isn't running, but the syslog and nagios.log don't seem to have anything suggesting why...

[1439412622] Warning: use_embedded_perl_implicitly is deprecated and will be removed.
[1439412622] Warning: sleep_time is deprecated and will be removed.
[1439412622] Warning: p1_file is deprecated and will be removed.
[1439412622] Warning: external_command_buffer_slots is deprecated and will be removed. All commands are always processed upon arrival
[1439412622] Warning: enable_embedded_perl is deprecated and will be removed.
[1439412622] Warning: command_check_interval is deprecated and will be removed. Commands are always handled on arrival
[1439412622] Nagios 4.0.8 starting... (PID=23334)
[1439412622] Local time is Wed Aug 12 20:50:22 UTC 2015
[1439412622] LOG VERSION: 2.0
[1439412622] qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
[1439412622] qh: core query handler registered
[1439412622] nerd: Channel hostchecks registered successfully
[1439412622] nerd: Channel servicechecks registered successfully
[1439412622] nerd: Channel opathchecks registered successfully
[1439412622] nerd: Fully initialized and ready to rock!
[1439412622] wproc: Successfully registered manager as @wproc with query handler
[1439412622] wproc: Registry request: name=Core Worker 23336;pid=23336
[1439412622] wproc: Registry request: name=Core Worker 23337;pid=23337
[1439412622] wproc: Registry request: name=Core Worker 23339;pid=23339
[1439412622] wproc: Registry request: name=Core Worker 23341;pid=23341
[1439412622] wproc: Registry request: name=Core Worker 23340;pid=23340
[1439412622] wproc: Registry request: name=Core Worker 23338;pid=23338
[1439412622] mod_gearman: initialized version 2.1.2 (libgearman 1.1.12)
[1439412622] Event broker module '/usr/local/lib/mod_gearman2/mod_gearman2.o' initialized successfully.
[1439412622] ndomod: NDOMOD 2.0.0 (02-28-2014) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1439412622] ndomod: Successfully connected to data sink. 0 queued items to flush.
[1439412622] ndomod registered for process data
[1439412622] ndomod registered for log data'
[1439412622] ndomod registered for system command data'
[1439412622] ndomod registered for event handler data'
[1439412622] ndomod registered for notification data'
[1439412622] ndomod registered for comment data'
[1439412622] ndomod registered for downtime data'
[1439412622] ndomod registered for flapping data'
[1439412622] ndomod registered for program status data'
[1439412622] ndomod registered for host status data'
[1439412622] ndomod registered for service status data'
[1439412622] ndomod registered for adaptive program data'
[1439412622] ndomod registered for adaptive host data'
[1439412622] ndomod registered for adaptive service data'
[1439412622] ndomod registered for external command data'
[1439412622] ndomod registered for aggregated status data'
[1439412622] ndomod registered for retention data'
[1439412622] ndomod registered for contact data'
[1439412622] ndomod registered for contact notification data'
[1439412622] ndomod registered for acknowledgement data'
[1439412622] ndomod registered for state change data'
[1439412622] ndomod registered for contact status data'
[1439412622] ndomod registered for adaptive contact data'
[1439412622] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1439412623] Successfully launched command file worker with pid 23346

Logs attached.

[ssax]

Just for good measure, verify the configs are valid:

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

What is the output of this command:

Code: Select all

ps -ef | grep /usr/local/nagios/bin/nagios

[gormank]

[root@cocsm2mlnag001 var]# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
Warning: use_embedded_perl_implicitly is deprecated and will be removed.
Warning: sleep_time is deprecated and will be removed.
Warning: p1_file is deprecated and will be removed.
Warning: external_command_buffer_slots is deprecated and will be removed. All commands are always processed upon arrival
Warning: enable_embedded_perl is deprecated and will be removed.
Warning: command_check_interval is deprecated and will be removed. Commands are always handled on arrival
Read main config file okay...
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 858 services.
Checked 77 hosts.
Checked 26 host groups.
Checked 0 service groups.
Checked 22 contacts.
Checked 4 contact groups.
Checked 134 commands.
Checked 23 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 77 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 23 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
[root@cocsm2mlnag001 var]# ps -ef | grep /usr/local/nagios/bin/nagios
root 29196 2904 0 21:09 pts/0 00:00:00 grep /usr/local/nagios/bin/nagios

[gormank]

I removed the configuration added to disable IPv6 and restarted. The system is running now but npcd wasn't. It starts manually though.
I manually stopped mysqld and then restarted the servers and everything came back normally.
You may just want to remove this thread...
Thanks!