Upgrade from 5.2.7 to 5.5.0 failed

[blueagle]

Finally! Upgrade has completed and I can see we're running 5.5.1 however I'm not sure if it completed successfully. I didn't get the "Upgrade Completed!" message as per the guide.
Most importantly the "Monitoring Engine" is not running and when I start it I get a message "An unexpected error has occurred". I've uploaded the latest upgrade.log.
Thanks for all your help so far.

[scottwilkerson]

Your config did not verify successfully.

Go to Configure, CCM -> Tools -> Config File Management
Click Delete Files
Write Configs
Verify Files

Work through any errors then repeat the process until the config verifies successfully.

Then Apply Configuration

[blueagle]

I've looked into it a bit more and I don't think its the configuration that's causing the issue. I believe "nagios" service isn't starting up.
I figured it out when I looked at nagios.log in /usr/local/nagios/var.
This file hasn't updated since 20th of July and that was when I had last attempted the update. The most recent log line in the log file:

[1532073086] Caught SIGSEGV, shutting down..

I've tried to start the nagios service manually but got this error:

[root@nagios var]# service nagios start
Starting nagios: /etc/init.d/nagios: line 87: 7935 Segmentation fault $NagiosBin -vp $NagiosCfgFile > "$TMPFILE"
[root@nagios var]# service nagios status
No lock file found in /var/run/nagios.lock

So I thought I'll stop and start all services manually as mentioned in this link:

http://www.scmgalaxy.com/tutorials/step ... xi-server/

Got exactly the same error message.

I've looked at the TMPFILE that's getting created in /tmp directory. Its blank!
Please advise what do I need to do to fix this?

[scottwilkerson]

from your log this:

Code: Select all

> Verifying configuration with Nagios Core
> Output: 
> Return Code: 139

and this

[root@nagios var]#

Code: Select all

service nagios start
Starting nagios: /etc/init.d/nagios: line 87:  7935 Segmentation fault      $NagiosBin -vp $NagiosCfgFile > "$TMPFILE"

Are both because it cannot verify the configuration.

Did you do the steps I outlined above?

[blueagle]

I started going through all the verification erros as you had suggested.
It was clear Nagios was just going through the list of services in alphabetical order and was complaining about the same thing for each service next in the alphabetical order.

Error: Service has no hosts and/or service_description (config file '/usr/local/nagios/etc/services/devops_mysql_connection_time.cfg', starting on line 16)
Error: Service escalation host name or description is NULL
Error: Could not register service escalation (config file '/usr/local/nagios/etc/serviceescalations.cfg', starting on line 16)

I've received the same error for 11 different services which would appear next in order. All of these have hosts and service description.
To fix I would de-activate the service, Delete Files, Write Configs, Verify Files and the next service would throw the same error.

In the Verification I'm seeing this constantly:

Cannot open/overwrite service configuration files (check the permissions)!

Nagios is only writing one cfg file under:
/usr/local/nagios/etc/services

When I select "Delete Files" then this file is deleted but I see this error in apache error_log:

PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 491

The exact same error appears in error_log when "Write Configs" is selected.

I've confirmed apache:apache is the owner of the directory. Even changed permissions to read/write/execute all (777) but that didn't help. SELinux is disabled. Its a CentOS 6.5 vm.

[scottwilkerson]

can you show the output of the following

Code: Select all

ls -al /usr/local/nagios/etc/services

[blueagle]

Here's the output:

[root@nagios ~]# ls -al /usr/local/nagios/etc/services
total 28
drwxrwxr-x 2 apache nagios 20480 Jul 24 09:41 .
drwxrwxr-x 8 apache nagios 4096 Jul 23 10:08 ..
-rw-rw-r-- 1 apache apache 1453 Jul 24 09:41 devops_nrpe_check_load.cfg

[scottwilkerson]

lets go up one level

Code: Select all

ls -al /usr/local/nagios/etc

[blueagle]

Here's the output:

[root@nagios ~]# ls -al /usr/local/nagios/etc
total 524
drwxrwxr-x 8 apache nagios 4096 Jul 23 10:08 .
drwxr-xr-x 9 root root 4096 Jul 27 2016 ..
-rw-rw-r-- 1 apache nagios 216563 Jun 11 2014 boundary.pem
-rw-rw-r-- 1 apache nagios 92 Jun 11 2014 boundary.yml
-rw-rw-r-- 1 apache nagios 1279 Apr 19 11:37 cgi.cfg
-rw-rw-r-- 1 apache nagios 31183 Jul 24 09:41 commands.cfg
-rw-rw-r-- 1 apache nagios 1851 Jul 24 09:41 contactgroups.cfg
-rw-rw-r-- 1 apache nagios 11830 Jul 24 09:41 contacts.cfg
-rw-rw-r-- 1 apache nagios 1969 Jul 24 09:41 contacttemplates.cfg
-rw-rw-r-- 1 apache nagios 796 Jul 24 09:41 hostdependencies.cfg
-rw-rw-r-- 1 apache nagios 794 Jul 24 09:41 hostescalations.cfg
-rw-rw-r-- 1 apache nagios 786 Jul 24 09:41 hostextinfo.cfg
-rw-rw-r-- 1 apache nagios 6177 Jul 24 09:41 hostgroups.cfg
drwxrwxr-x 2 apache nagios 32768 Jul 24 09:41 hosts
-rw-rw-r-- 1 apache nagios 21419 Jul 24 09:41 hosttemplates.cfg
drwxrwxr-x 2 apache nagios 4096 Apr 19 11:37 import
-rw-rw-r-- 1 apache nagios 7336 Mar 13 2014 marge.cttest.bskyb.com.cfg
-rw-rw-r-- 1 apache nagios 6583 May 16 2016 nagios.cfg
-rw-rw-r-- 1 apache nagios 6320 Jul 20 2015 nagios.cfg_my_jp
-rw-rw-r-- 1 apache nagios 6250 Jul 20 2015 nagios.cfg_pre_change_jp
-rw-rw-r-- 1 apache nagios 2229 Feb 10 2014 ndo2db.cfg
-rw-rw-r-- 1 apache nagios 4827 Feb 10 2014 ndomod.cfg
-rw-rw-r-- 1 apache nagios 7222 Feb 25 2014 nrpe.cfg
-rw-rw-r-- 1 apache nagios 7222 Feb 25 2014 nrpe.cfg.old
-rw-rw-r-- 1 apache nagios 5351 Aug 14 2014 nsca.cfg
drwxrwxr-x 4 apache nagios 4096 Jul 23 10:08 pnp
-rw-rw-r-- 1 apache nagios 0 Jan 27 2015 recurringdowntime.cfg
-rw-rw-r-- 1 apache nagios 210 Feb 10 2014 resource.cfg
-rw-rw-r-- 1 apache nagios 1633 Aug 14 2014 send_nsca.cfg
-rw-rw-r-- 1 apache nagios 802 Jul 24 09:41 servicedependencies.cfg
-rw-rw-r-- 1 apache nagios 1104 Jul 24 09:41 serviceescalations.cfg
-rw-rw-r-- 1 apache nagios 792 Jul 24 09:41 serviceextinfo.cfg
-rw-rw-r-- 1 apache nagios 1888 Jul 24 09:41 servicegroups.cfg
drwxrwxr-x 2 apache nagios 20480 Jul 24 09:41 services
-rw-rw-r-- 1 apache nagios 24305 Jul 24 09:41 servicetemplates.cfg
drwxrwxr-x 2 apache nagios 4096 Jul 23 10:08 static
drwxrwxr-x 2 apache nagios 4096 Jul 23 10:08 stormtest
-rw-rw-r-- 1 apache nagios 10898 Jul 24 09:41 timeperiods.cfg
[root@nagios ~]#

[scottwilkerson]

Code: Select all

su - nagios
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh

Please run the EXACT commands above and send us the entire output.