Services disappeared

[DragonJTS]

All my services have disappeared from NagiosXI. They still appear in Nagios core, and the xi core config manager, and my hosts appear in XI. I've tried restarting the VM, and restarting the nagios service.

The first time I logged off and on on Nagios after the problem arouse, I got the message "SQL: SQL Error [ndoutils] : Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired". I ran the directions for repairing the server in the admin docs, it didn't do anything.

1. CentOS 5.6
2. 32 Bit
3. VMware
4. No special configurations

[lmiltchev]

Try repairing the database, following the steps described in this document:

http://assets.nagios.com/downloads/nagi ... tabase.pdf

[DragonJTS]

Repairing the database did not fix the issue (I meant to say I had repaired the database in my original post, not repair the server).

[lmiltchev]

What version of Nagios XI are you running?

Do you have any "segmentation fault" errors in your "/var/log/httpd/error_log"?

If you are running an older version of XI, you can try upgrading to R1.6. Before you proceed with an upgrade, please, review our documentation on the upgrade process:

http://assets.nagios.com/downloads/nagi ... ctions.pdf

[DragonJTS]

I am running the latest version.

No segmentation faults, but there are a huge amount of errors (several a second since around the time the issue started) mentioning an undefined index Max/min/avg_execution/latency_time in /usr/local/nagiosxi/html/includex/utils-xmlsysstat.inc.php on line 117/18/19/20, with the referer being my nagiosxi config url.

[mguthrie]

Try the following, and send us the output from the dbmaint script that is run. There's a table that's still corrupt and doesn't appear to have been repaired by the script.

Code: Select all

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<corrupted_table>

service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
/usr/local/nagiosxi/cron/dbmaint.php 

[DragonJTS]

After I ran those commands, it started working again. Do you still want the output?

[lmiltchev]

Sure. It would be nice to have some feedback. Thank you!

[DragonJTS]

Code: Select all

CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1281532528)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1281532528)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1312463728)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1305292528)
CLEANING ndoutils TABLE 'notifications'...
SQL: DELETE FROM nagios_notifications WHERE start_time < FROM_UNIXTIME(1305292528)
CLEANING ndoutils TABLE 'contactnotifications'...
SQL: DELETE FROM nagios_contactnotifications WHERE start_time < FROM_UNIXTIME(1305292528)
CLEANING ndoutils TABLE 'contactnotificationmethods'...
SQL: DELETE FROM nagios_contactnotificationmethods WHERE start_time < FROM_UNIXTIME(1305292528)
CLEANING ndoutils TABLE 'statehistory'...
SQL: DELETE FROM nagios_statehistory WHERE state_time < FROM_UNIXTIME(1249996528)
CLEANING ndoutils TABLE 'timedevents'...
SQL: DELETE FROM nagios_timedevents WHERE event_time < FROM_UNIXTIME(1313068228)
CLEANING ndoutils TABLE 'systemcommands'...
SQL: DELETE FROM nagios_systemcommands WHERE start_time < FROM_UNIXTIME(1313068228)
	SQL: 	SQL Error [ndoutils] :</b> Table './nagios/nagios_systemcommands' is marked as crashed and should be repairedCLEANING ndoutils TABLE 'servicechecks'...
SQL: DELETE FROM nagios_servicechecks WHERE start_time < FROM_UNIXTIME(1313068228)
	SQL: 	SQL Error [ndoutils] :</b> Table './nagios/nagios_servicechecks' is marked as crashed and should be repairedCLEANING ndoutils TABLE 'hostchecks'...
SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1313068228)
CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1313068228)
LASTOPT:  1313065506
INTERVAL: 60
NOW:      1313068528
OPTTIME:  1313069106
CLEANING nagiosxi TABLE 'commands'...
SQL: DELETE FROM xi_commands WHERE processing_time < 1313039728::abstime::timestamp without time zone
CLEANING nagiosxi TABLE 'events'...
SQL: DELETE FROM xi_events WHERE processing_time < 1313039728::abstime::timestamp without time zone
SQL1: SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL
SQL2: DELETE FROM xi_meta WHERE meta_id IN (SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL)
CLEANING nagiosql TABLE 'logbook'...
SQL: DELETE FROM tbl_logbook WHERE time < FROM_UNIXTIME(1313039728)
Repair Complete: Removing Lock File

[lmiltchev]

Thanks!