Problems post upgrade (2012R1.5b to 2014R2.7)

[jcmanous]

Greetings,

I did a manual upgrade from 2012R1.5b to 2014R2.7 (which by all appearances worked fine) but when restarting Nagios XI (and the whole host), I have 3 services that come up disabled:

Active Host Checks
Active Service Checks
Notifications

For reference, this is running on a fully patched Redhat 6.6 host.

Thanks!

Jason

[tgriep]

In the /usr/local/nagios/etc/nagios.cfg file, what are the following variables set to?

Code: Select all

enable_notifications
execute_service_checks
execute_host_checks

If they are set to 1, that will enable them at startup.

[lmiltchev]

Run the following commands and show us the output:

Code: Select all

/usr/local/nagios/bin/nagios | head -2
/usr/local/nagios/bin/ndo2db | head -2
service mysqld restart
service nagios stop
service ndo2db stop
service ndo2db start
service nagios start
service mysqld status
service nagios status
service ndo2db status

Were you using mod gearman or mk livestatus?

[jcmanous]

The output of the commands requested is listed below. Additionally, all values for the following settings were set to =1:

Code: Select all

enable_notifications
execute_service_checks
execute_host_checks

Code: Select all

[root]# /usr/local/nagios/bin/nagios | head -2

Nagios Core 4.0.8
[root]# /usr/local/nagios/bin/ndo2db | head -2

NDO2DB 2.0.0
[root]# service nagios stop
Stopping nagios: .done.
[root]# service ndo2db stop
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
[root]# service ndo2db start
Starting ndo2db: done.
[root]# service nagios start
Starting nagios: done.
[root]# service mysqld status
mysqld (pid  1509) is running...
[root]# service nagios status
nagios (pid 6719) is running...
[root]# service ndo2db status
ndo2db (pid 6702) is running...

[lmiltchev]

Were you using mod gearman or mk livestatus prior to the upgrade? Can you run the following commands and post the output in code wraps?

Code: Select all

tail -100 /var/log/messages
tail -100 /usr/local/nagios/var/nagios.log

Also, post the "/tmp/nagiosxi/upgrade.log".

[jcmanous]

There have been a few days since I've last updated and another admin did some manual poking around on the system hence, those commands probably don't net what you want.

What's the easiet way to check on the status of those?

[jcmanous]

I went back through the logs on the day I started this thread and this keeps coming up:

From /var/log/messages

Code: Select all

Jun 23 01:45:43 smart ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_systemcommands SET instance_id='1', start_time=FROM_UNIXTIME(1435038343), start_time_usec='163695', end_time=FROM_UNIXTIME
(0), end_time_usec='0', command_line='/bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/1435038343\.perfdata\.service', timeout='5', early_timeout='0', execution_time='0.000
000', return_code='0', output='', long_output='' ON DUPLICATE KEY UPDATE instance_id='1', start_time=FROM_UNIXTIME(1435038343), start_time_usec='163695', end_time=FROM_UNIXTIME(0), end_time_usec='0', comm
and_line='/bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/1435038343\.perfdata\.service', timeout='5', early_timeout='0', execution_time='0.000000', return_code='0', outpu
t='', long_output='''

[tgriep]

It looks like the mysql database is corrupt. Run the following to repair it.

Code: Select all

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f *
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

[jcmanous]

Ran the code given and still have 3 of 6 services showing a blue "!"

Active Host Checks Disabled
Active Service Checks Disabled
Notifications are Disabled

[jcmanous]

At this point I think the DB must be in such sad shape that these things aren't taking.

We only have one plugin that's custom (external) and that's the AD Authentication Plugin. That said, is there a quick and dirty method to back up the hosts and their services files to migrate to a fresh install?