Nagios at a standstill?

[waydms]

Virtual PC, 32-bit.
Standard Edition, Trial.

Hey everyone!

Me again with another potentially simple issue and fix I've overlooked while searching around.
Went on vacation last Thursday and came back yesterday. Attempted to access my Nagios site and it just kept spinning...odd. Checked the virtual pc on one of our test machines and it wasn't even online. I guess someone restarted the machine that wasn't supposed to be

ANYWAYS...

Ever since then, whenever I start up the virtual pc, I am able to login just fine, after that I will run...let's say...the shutdown command and after pressing enter, nothing will happen. Just a standstill. Even attempting to restart some services, nada.

I've attached an image to somehow show the issue.
If anyone can point me in the right direction, much appreciation.
Hopefully all of these issues I've been posting on are related to just this being a virtual pc and not anything major with Nagios itself. Attempting to schedule a demo with the sales reps and more questions are coming!

Keep in mind, novice at Linux. So I'm assuming this is in general linux issue, not ultimately Nagios.

[abrist]

Lets check the logs for any hints:

Code: Select all

tail -25 /var/log/messages
grep seg /var/log/messages
df -h
df -i

Have you received any errors concerning the disk or filesystem?

[waydms]

Was able to run the tail command but attempted grep and received standstill. The end portion of the command showed...

Sep 4 15:50:35 nagios ndo2b: mysql_error: 'Table ' ./nagios/nagios_servicestatus' is marked as crashed and should be repaired'

It seems I've repaired the DB to this date about four times. I will do so again and attempt a normal restart/shutdown without forcing it.
Doubtful that is the main cause of this issue though.

Keep the ideas/comments/thoughts flowing!


ADDED!
Also for the errors question,
I do receive an invalid checksum 0x89 at boot.

[slansing]

Let us know what comes of it, if you are forcing shutdowns that WILL cause DB corruption, just make sure to properly restart the server. Though, this does sound like an issue with the VM itself, or a networking related issue.

[waydms]

It seems the repair brought it back to life along with some proper reboots.

Although missing a majority of my host data/info now.
For example, my hosts and services are configured in CCM although nothing is in the Service and Host details. Nor on the maps, graphs, etc.

Anyone have an idea on this? Believe I have a backup somewhere so not worried or I can just autodiscover everything. All component statuses show good.

[abrist]

Lets tail the http error logs for more information:

Code: Select all

tail -25 /var/log/httpd/error_log

And some others:

Code: Select all

tail -25 /usr/local/nagios/var/nagios.log
tail -25 /var/log/mysqld.log

[waydms]

Hey!

Just got the log files for error_log and nagios.log and those are attached. For whatever reason couldn't get my hands onto the mysqld log file on my main machine so searching through that. Tons of " is marked as crashed and should be repaired." Only at 7%, hah.

Not certain why error_log is stating issues with client x.x.2.72, the IP for Nagios is .210 and the host machine is .220.
Weird. No hosts I've configured have that address either.

Not certain what to make out of the nagios log. Will post the mysqld file when available.

UPDATE
SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_programstatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_objects' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_programstatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_objects' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_programstatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_objects' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_programstatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_objects' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_programstatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_objects' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_programstatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_objects' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_programstatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_objects'

Wonderful error when attempting to view service detail for those interested. O___O

[abrist]

Lets resolve the database errors first (if possible). Run another repair:

Code: Select all

service mysqld stop
/usr/local/nagiosxi/scripts/repairmysql.sh nagios

The database may experienced some serious damage from the unintended shutdown.

[waydms]

Annnnnnd attempt the service stop...linux is moving nowhere. Ctrl+c'd and tried just the reboot command...posted the restart message and going nowhere. Something has to be off with the VM.

[slansing]

Would it be a possibility for you to backup and restore your nagios configs on a new VM?