Everything seems to have disappeared

[stebbo]

Hi All,

our server hosting our CentOS VM running under Fusion crashed earlier today, and now everything seems to have disappeared from within our Nagios installation.

For a while we couldn't login, we were getting the "Sorry Dave" message. I wasn't able to look into the problem for a few hours due to other commitments, but when I returned that seems to have resolved itself but nothing is working.

I have a "system problem" which says that the Performance Grapher is not running.

I saw an error message regarding the SQL tables being dirty at some stage, but cannot get that to return and I was unable to write it down fully at the time. I have no idea how to fix it.

There are no hosts and no services listed on the host or service status pages, and my main dashboard lists nothing as well. When I go into the configuration manager I can see host and service information, but nothing appears to be running or displaying.

I do have a backup of the configuration, but am not sure whether I should just restore from this backup (nor how to achieve this) or if there's something else I can do to get the system back running again.

Please help.

Cheers,
Chris.

[mguthrie]

My guess is that either mysql has crashed, or the ndo2db service is not running. Try the following commands.

Code: Select all

service nagios stop
service ndo2db stop
service mysqld stop
myisamchk -r -f /var/lib/nagios/*.MYI
service mysqld start
service ndo2db start
service nagios start
service npcd restart

[stebbo]

Hi mguthrie,

thanks for the reply. I tried the above commands but to no avial.

When I tried the myisamchk command, I received the following error
error: File '/var/lib/nagios/*.MYI' doesn't exist.
I checked and there is no directory nagio in /var/lib

When I did the service npcd restart it told me that npcd wasn't running, but started the service.

I now have all green ticks at the top right of the screen, but no hosts or services listed on any of the dashboards or host/status summary pages.

When I go into the config manager, all my hosts and services are listed, and their status is Active, however I can't see anything happening on any of my screens.

Cheers,
Chris.

[mguthrie]

Can you post a screenshot of what you're seeing?

[stebbo]

Hi mguthrie,

it's all good now. Support emailed me the mysql repair instructions and that did the trick. All the services and graphs popped right up while I was watching it..

It seems as though some data was being logged all through the problem but there are a few holes in the graphs which is no big deal at all - very happy to have it back up and running.

Cheers,
Chris.

[slansing]

Have a great Friday Chris, closing this thread.