mysql_error in /var/log/messages

[mlawrence]

Hello - we recently set up two Nagios XI installs and have begun adding many services/hosts to each, mostly smooth going.

On both installs, I am noticing errors like the ones below:

Nov 13 16:01:50 scnagiosxi ndo2db: mysql_error: 'Table './nagios/nagios_timedeventqueue' is marked as crashed and should be repaired'

They will be preceded by an error in trying to insert or delete from that table, for example:

Nov 13 16:01:50 scnagiosxi ndo2db: Error: mysql_query() failed for 'DELETE FROM nagios_timedeventqueue WHERE instance_id='1' AND event_type='1' AND scheduled_time=FROM_UNIXTIME(1352851309) AND recurring_event='1' AND object_id='0''

I was wondering what the best way to repair that table is?

Also why would both instances get the same error at the same time? Are there any reasons why this would have happened to both installs, and are there any steps we can take to help avoid it happening again?

The system's info is below. Thanks!

Edit - I had to remove the URLs from the Admin->System Profile info as the forum would not allow that many URLs to go in to this post.

------------------------------------------------

1) CentOS, version 6.3
2) 64 bit
3) VMware Image
4) Admin->System Profile info:

Nagios XI Installation Profile
Download Profile
System:
Nagios XI Version : 2012R1.2
scnagiosxi.pfsc.local 2.6.32-279.9.1.el6.x86_64 x86_64
CentOS release 6.3 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0
Server Name: scnagiosxi.practicefusion.com
Server Address: 10.11.13.22
Server Port: 80
Date/Time
PHP Timezone: America/Chicago
PHP Time: Tue, 13 Nov 2012 17:58:27 -0600
System Time: Tue, 13 Nov 2012 15:58:27 -0800
Nagios XI Data
nagios (pid 30116) is running...
NPCD running (pid 1797).
ndo2db (pid 1861) is running...
CPU Load 15: 1.74
Total Hosts: 85
Total Services: 1593
Function 'get_base_uri' returns: SNIP
Function 'get_base_url' returns: SNIP
Function 'get_backend_url(internal_call=false)' returns: SNIP
Function 'get_backend_url(internal_call=true)' returns: SNIP
Ping Test localhost
Running:

/bin/ping -c 3 localhost 2>&1

PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.030 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.033 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.029/0.030/0.033/0.006 ms
Test wget To locahost
WGET From URL: http://localhost/nagiosql/index.php
Running:

/usr/bin/wget http://localhost/nagiosql/index.php

--2012-11-13 15:58:29-- http://localhost/nagiosql/index.php
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5259 (5.1K) [text/html]
Saving to: `/tmp/nagiosql_index.tmp'

0K ..... 100% 582M=0s

2012-11-13 15:58:29 (582 MB/s) - `/tmp/nagiosql_index.tmp' saved [5259/5259]

[scottwilkerson]

To repair the database, follow this procedure
http://library.nagios.com/library/produ ... i-database

The most common cause of this type of database problem is sudden power loss.

On a VMware image, if you need to shutdown the server or VM it is always best to shutdown the VM through the console first with the following command

Code: Select all

shutdown -h now

[mlawrence]

Thanks for the response, I've applied the fix to one of our XI instances and now it's nice and quiet again.

Aside from the "shutdown -h now" advice - is there any best practice for turning off the Nagios services? Ie, should I do something like:

service nagios stop
service ndo2db stop
service mysqld stop

before executing the shutdown command? Both of our instances were recently rebooted, I didn't do the reboots myself but I believe they were done cleanly, yet it looks like these errors came after the reboots. I can't seem to find a "best practices" doc for how to stop/start all Nagios and Nagios XI services anywhere.

[slansing]

The shutdown command will allow the services to close before the whole system is stopped, so you should not have to worry about anything besides that.

[mlawrence]

OK, I was told the command used was "poweroff" so we can try shutdown in the future.

I am still curious though about the correct way to safely stop/start all of the Nagios and XI services in CentOS. Let's say I didn't need to power off or reboot, but to perform OS or security patching and that I needed the services to be shut down for the patches to be applied. Assume further the patching didn't require a reboot and that post-patching I would need to start the services back up. What commands would be the best practice method for turning on and starting up the services again manually for that kind of situation?

[scottwilkerson]

To stop all nagios related process you would need to perform these steps

Code: Select all

service nagios stop
service ndo2db stop
service mysqld stop
service npcd stop
service postgresql stop
service httpd stop
service crond stop
service xinetd stop

to start them back up

Code: Select all

service nagios start
service ndo2db start
service mysqld start
service npcd start
service postgresql start
service httpd start
service crond start
service xinetd start