Page 1 of 2
Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Tue Jun 23, 2015 2:10 pm
by jcmanous
Greetings,
I did a manual upgrade from 2012R1.5b to 2014R2.7 (which by all appearances worked fine) but when restarting Nagios XI (and the whole host), I have 3 services that come up disabled:
Active Host Checks
Active Service Checks
Notifications
For reference, this is running on a fully patched Redhat 6.6 host.
Thanks!
Jason
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Tue Jun 23, 2015 2:46 pm
by tgriep
In the /usr/local/nagios/etc/nagios.cfg file, what are the following variables set to?
Code: Select all
enable_notifications
execute_service_checks
execute_host_checks
If they are set to 1, that will enable them at startup.
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Tue Jun 23, 2015 2:50 pm
by lmiltchev
Run the following commands and show us the output:
Code: Select all
/usr/local/nagios/bin/nagios | head -2
/usr/local/nagios/bin/ndo2db | head -2
service mysqld restart
service nagios stop
service ndo2db stop
service ndo2db start
service nagios start
service mysqld status
service nagios status
service ndo2db status
Were you using mod gearman or mk livestatus?
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Tue Jun 23, 2015 3:00 pm
by jcmanous
The output of the commands requested is listed below. Additionally, all values for the following settings were set to =1:
Code: Select all
enable_notifications
execute_service_checks
execute_host_checks
Code: Select all
[root]# /usr/local/nagios/bin/nagios | head -2
Nagios Core 4.0.8
[root]# /usr/local/nagios/bin/ndo2db | head -2
NDO2DB 2.0.0
[root]# service nagios stop
Stopping nagios: .done.
[root]# service ndo2db stop
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
[root]# service ndo2db start
Starting ndo2db: done.
[root]# service nagios start
Starting nagios: done.
[root]# service mysqld status
mysqld (pid 1509) is running...
[root]# service nagios status
nagios (pid 6719) is running...
[root]# service ndo2db status
ndo2db (pid 6702) is running...
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Tue Jun 23, 2015 3:15 pm
by lmiltchev
Were you using mod gearman or mk livestatus prior to the upgrade? Can you run the following commands and post the output in code wraps?
Code: Select all
tail -100 /var/log/messages
tail -100 /usr/local/nagios/var/nagios.log
Also, post the "/tmp/nagiosxi/upgrade.log".
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Wed Jul 22, 2015 8:55 am
by jcmanous
There have been a few days since I've last updated and another admin did some manual poking around on the system hence, those commands probably don't net what you want.
What's the easiet way to check on the status of those?
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Wed Jul 22, 2015 9:10 am
by jcmanous
I went back through the logs on the day I started this thread and this keeps coming up:
From /var/log/messages
Code: Select all
Jun 23 01:45:43 smart ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_systemcommands SET instance_id='1', start_time=FROM_UNIXTIME(1435038343), start_time_usec='163695', end_time=FROM_UNIXTIME
(0), end_time_usec='0', command_line='/bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/1435038343\.perfdata\.service', timeout='5', early_timeout='0', execution_time='0.000
000', return_code='0', output='', long_output='' ON DUPLICATE KEY UPDATE instance_id='1', start_time=FROM_UNIXTIME(1435038343), start_time_usec='163695', end_time=FROM_UNIXTIME(0), end_time_usec='0', comm
and_line='/bin/mv /usr/local/nagios/var/service-perfdata /usr/local/nagios/var/spool/xidpe/1435038343\.perfdata\.service', timeout='5', early_timeout='0', execution_time='0.000000', return_code='0', outpu
t='', long_output='''
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Wed Jul 22, 2015 9:31 am
by tgriep
It looks like the mysql database is corrupt. Run the following to repair it.
Code: Select all
service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f *
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Wed Jul 22, 2015 10:11 am
by jcmanous
Ran the code given and still have 3 of 6 services showing a blue "!"
Active Host Checks Disabled
Active Service Checks Disabled
Notifications are Disabled
Re: Problems post upgrade (2012R1.5b to 2014R2.7)
Posted: Wed Jul 22, 2015 10:38 am
by jcmanous
At this point I think the DB must be in such sad shape that these things aren't taking.
We only have one plugin that's custom (external) and that's the AD Authentication Plugin. That said, is there a quick and dirty method to back up the hosts and their services files to migrate to a fresh install?