Upgrade from 5.4.0 to 5.4.4 failure

[gwakem]

Happy Monday!
Currently on 5.4.0 and trying to upgrade to 5.4.4 via the shell, but I'm getting consistent failures.
Here is the output of the log at the time of failure:

Code: Select all

make[1]: Leaving directory `/var/tmp/nagiosxi/subcomponents/nagioscore/nagios-4.2.4'
Warning: nagios.service changed on disk. Run 'systemctl daemon-reload' to reload units.
no crontab for nagios
Saving old crontab to /tmp/root.crontab.orig.5...
Removing XI cron jobs...
Copying over new XI directory...
Updating NagiosQL...
NAGIOSQL-POST
Patching NDOUtils...
NDOUtils already patched at level 103
Installing new PNP templates...
Updating init script...
Enabling large install tweaks...
Fixing config file permissions...
Fixing htpasswd permissions...
Checking group memberships...
Installing new XI templates...
URL: https://localhost/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies https://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&backend=1&username=nagiosxi&password=xxxxxxxx' -O nagiosql.login--2017-05-15 13:19:17--  https://localhost/nagiosxi/includes/components/ccm/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2017-05-15 13:19:17 ERROR 500: Internal Server Error.

NAGIOSQL LOGIN FAILED!

This system forces SSL logins. I have attempted the upgrade with SSL redirection rules disabled, and the CCM Admin > Settings > Paths > Server Protocol set to HTTP instead of HTTPS, but it always attempts HTTPS regardless. After this point, apache only displays a blank page and I can't do anything but restore from backup.

The "/usr/bin/wget --save-cookies nagiosql.cookies" that fails during the upgrade process works properly before the upgrade with no errors.

[tgriep]

Is the server setup use the LDAP / Active Directory integration component?

Try this KB article and see if it fixes the error when you try the upgrade.
https://support.nagios.com/kb/article/n ... login.html

Can you look in the Apache Error logs in the /var/log/httpd folder when the upgrade was done, search them for any errors and post them here so we can review them?

[gwakem]

This does use AD Integration, but the only thing I see in /usr/local/nagiosxi/html/includes/components/ is ldap_ad_integration. I attempted removing that after the upgrade failed (and then again before an upgrade attempt,) but that didn't fix the 500 errors in the logs and white screen. As far as logs, I removed all apache logs, restarted apache, ran the upgrade, and reviewed the logs, but there is nothing that stands out. It simply goes from showing 200's to 500's. I gzipped the logs in case you want to review them.

[tgriep]

Yes, we would need to see the log files and also your System Profile from the server as well.
Can you PM them to me?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and PM it back to me.

Before doing the upgrade, can you go in to the Core Config Manager and Apply the Config without any errors?

[gwakem]

The profile has been PM'd. and yes, I can apply with no issues.

[tgriep]

Thanks for the profile, I didn't find any settings or configurations that could cause the upgrade to fail.
If you still have the Apache Log files as well as the upgrade.log file for when you ran the upgrade, PM them to me so I can check them for errors.

The only thing you should do is to allocate more memory to the system, it is very tight and it looks like the system swapped in the past which slows the server down.
Try increasing the memory to 8 Gig.

[gwakem]

I increased the memory, although the fact that I have pretty much only been non stop testing the upgrade from different angles may have contributed to that. Also, I sent you the upgrade log and apache logs via PM.

[tgriep]

I received the file and shared it.
I saw one permission error in the logs so can you run the following commands as root and post the output?

Code: Select all

ls -l /usr/local/nagiosxi/html/includes/
ls -l /usr/local/nagiosxi/html/includes/phpmailer

Can you attach the following files here so we can check the servers ssl settings?

Code: Select all

/etc/hosts
/etc/httpd/conf/httpd.conf
/etc/httpd/conf.d/nagiosxi.conf
/etc/httpd/conf.d/ssl.conf

Thanks

[gwakem]

ls -l /usr/local/nagiosxi/html/includes/

Code: Select all

total 2776
-rwxr-x---  1 nagios nagios   5980 Jan 11 09:35 auth.inc.php
-rwxr-x---  1 nagios nagios    807 Jan 11 09:35 common.inc.php
drwxr-x--- 63 nagios nagios   4096 Apr 11 15:17 components
-rwxr-x---  1 nagios nagios   1060 Jan 11 09:35 components.inc.php
drwxr-x--- 65 nagios nagios   4096 Mar 17  2016 configwizards
-rwxr-x---  1 nagios nagios    619 Jan 11 09:35 configwizards.inc.php
-rwxr-x---  1 nagios nagios  21886 Jan 11 09:35 constants.inc.php
-rwxr-x---  1 nagios nagios  24962 Jan 11 09:35 constants-nagioscore.inc.php
drwxr-x---  3 nagios nagios   4096 Nov 30 09:57 css
drwxr-x--- 15 nagios nagios   4096 Apr  5  2016 dashlets
-rwxr-x---  1 nagios nagios    680 Jan 11 09:35 dashlets.inc.php
-rwxr-x---  1 nagios nagios 245009 Jan 11 09:35 dbauth.inc.php
-rwxr-x---  1 nagios nagios  16251 Jan 11 09:35 db.inc.php
-rwxr-x---  1 nagios nagios 104785 Jan 11 09:35 dbl.inc.php
-rwxr-x---  1 nagios nagios    763 Jan 11 09:35 errors.inc.php
drwxr-x---  2 nagios nagios   4096 Feb 18  2016 fonts
-rwxr-x---  1 nagios nagios     56 Jan 11 09:35 footer-child.inc.php
-rwxr-x---  1 nagios nagios   7349 Jan 11 09:35 footer.inc.php
drwxr-x---  5 nagios nagios    112 Feb 18  2016 fpdf
-rwxr-x---  1 nagios nagios    564 Jan 11 09:35 header-child.inc.php
-rwxr-x---  1 nagios nagios  20209 Jan 11 09:35 header.inc.php
drwxr-x---  6 nagios nagios   4096 Nov 30 09:57 js
drwxr-x---  3 nagios nagios     64 Feb 18  2016 lang
-rwxr-x---  1 nagios nagios    163 Jan 11 09:35 notificationmethods.inc.php
-rw-r-----  1 nagios nagios  17692 Jan 11 09:35 page-default-splash.php
-rwxr-x---  1 nagios nagios   2428 Jan 11 09:35 page-home-main.php
-rwxr-x---  1 nagios nagios    789 Jan 11 09:35 page-home.php
-rwxr-x---  1 nagios nagios    835 Jan 11 09:35 page-missing.php
-rwxr-x---  1 nagios nagios  30657 Jan 11 09:35 pageparts.inc.php
drwxr-x---  2 nagios nagios     74 Feb 18  2016 phpmailer
-rwxr-x---  1 nagios nagios   4054 Jan 11 09:35 utils-auditlog.inc.php
-rwxr-x---  1 nagios nagios  62681 Jan 11 09:35 utils-backend.inc.php
-rwxr-x---  1 nagios nagios  13188 Jan 11 09:35 utils-ccm.inc.php
-rwxr-x---  1 nagios nagios  31557 Jan 11 09:35 utils-commands.inc.php
-rwxr-x---  1 nagios nagios   3797 Jan 11 09:35 utils-components.inc.php
-rwxr-x---  1 nagios nagios  12064 Jan 11 09:35 utils-configwizards.inc.php
-rwxr-x---  1 nagios nagios 133425 Jan 11 09:35 utils-dashboards.inc.php
-rwxr-x---  1 nagios nagios 110053 Jan 11 09:35 utils-dashlets.inc.php
-rwxr-x---  1 nagios nagios   6502 Jan 11 09:35 utils-email.inc.php
-rwxr-x---  1 nagios nagios  21745 Jan 11 09:35 utils-events.inc.php
-rwxr-x---  1 nagios nagios   2036 Jan 11 09:35 utils-graphtemplates.inc.php
-rwxr-x---  1 nagios nagios  38305 Jan 11 09:35 utils.inc.php
-rwxr-x---  1 nagios nagios 822297 Jan 11 09:35 utilsl.inc.php
-rwxr-x---  1 nagios nagios   9328 Jan 11 09:35 utils-links.inc.php
-rwxr-x---  1 nagios nagios  75902 Jan 11 09:35 utils-menu.inc.php
-rwxr-x---  1 nagios nagios  27445 Jan 11 09:35 utils-metrics.inc.php
-rwxr-x---  1 nagios nagios   1998 Jan 11 09:35 utils-mibs.inc.php
-rwxr-x---  1 nagios nagios 246321 Jan 11 09:35 utils-nagioscore.inc.php
-rwxr-x---  1 nagios nagios   2080 Jan 11 09:35 utils-notificationmethods.inc.php
-rwxr-x---  1 nagios nagios 115801 Jan 11 09:35 utils-notifications.inc.php
-rwxr-x---  1 nagios nagios  38660 Jan 11 09:35 utils-objects.inc.php
-rwxr-x---  1 nagios nagios   2442 Jan 11 09:35 utils-perms.inc.php
-rw-r-----  1 nagios nagios   4103 Jan 11 09:35 utils-reports-export.inc.php
-rwxr-x---  1 nagios nagios  45445 Jan 11 09:35 utils-reports.inc.php
-rw-r-----  1 nagios nagios   6099 Jan 11 09:35 utils-rrdexport.inc.php
-rw-r-----  1 nagios nagios   1392 Jan 11 09:35 utils-rss.inc.php
-rwxr-x---  1 nagios nagios   5473 Jan 11 09:35 utils-status.inc.php
-rwxr-x---  1 nagios nagios   2379 Jan 11 09:35 utils-systat.inc.php
-rwxr-x---  1 nagios nagios  14349 Jan 11 09:35 utils-tables.inc.php
-rwxr-x---  1 nagios nagios    726 Jan 11 09:35 utils-themes.inc.php
-rw-r-----  1 nagios nagios   9530 Jan 11 09:35 utils-time.inc.php
-rw-r-----  1 nagios nagios   7751 Jun 21  2016 utils-timezones.inc.php
-rwxr-x---  1 nagios nagios   4299 Jan 11 09:35 utils-tools.inc.php
-rwxr-x---  1 nagios nagios   3738 Jan 11 09:35 utils-updatecheck.inc.php
-rwxr-x---  1 nagios nagios  31615 Jan 11 09:35 utils-users.inc.php
-rwxr-x---  1 nagios nagios  38085 Jan 11 09:35 utils-views.inc.php
-rwxr-x---  1 nagios nagios   6705 Jan 11 09:35 utils-wizards.inc.php
-rwxr-x---  1 nagios nagios  18071 Jan 11 09:35 utilsx.inc.php
-rwxr-x---  1 nagios nagios  34326 Jan 11 09:35 utilsx-json.php
-rwxr-x---  1 nagios nagios   4364 Jan 11 09:35 utils-xmlauditlog.inc.php
-rwxr-x---  1 nagios nagios  37057 Jan 11 09:35 utils-xmlobjects.inc.php
-rwxr-x---  1 nagios nagios  33960 Jan 11 09:35 utils-xmlreports.inc.php
-rwxr-x---  1 nagios nagios  62752 Jan 11 09:35 utils-xmlstatus.inc.php
-rwxr-x---  1 nagios nagios   7753 Jan 11 09:35 utils-xmlsysstat.inc.php
-rwxr-x---  1 nagios nagios   1947 Jan 11 09:35 utils-xmlusers.inc.php

ls -l /usr/local/nagiosxi/html/includes/phpmailer

Code: Select all

total 116
-rwxr-x--- 1 nagios nagios 74783 Jan 11 09:35 class.phpmailer.php
-rwxr-x--- 1 nagios nagios 11833 Jan 11 09:35 class.pop3.php
-rwxr-x--- 1 nagios nagios 28024 Jan 11 09:35 class.smtp.php

Hosts:

Code: Select all

127.0.0.1               localhost.localdomain localhost
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.20.12.198         xi-test.ingles-markets.com xi-test

[tgriep]

Those config files look like they are configured correctly and the permission error I saw looks to be a false alarm.

Can you login to the XI GUI and go to the Admin > System Config > System Settings menu and verify that the Program URL is setup to use https and is using the FQDN of the server

Code: Select all

https://FQDN/nagiosxi/

Next navigate to Configure > Core Config Manager > CCM Admin > Settings menu and click on the Paths tab, verify that the server protocol is https.

Go to the Admin > Reset Security Credentials menu, Click on the Update Credentials to reset passwords Nagiosxi uses for the backend, etc.

Then login to the XI server, run the following and post the output.

Code: Select all

cd /usr/local/nagiosxi/scripts
su nagios
./reconfigure_nagios.sh

I took another look at the Apache log files you sent earlier and I do not see the log change from 200 to 500 errors so they didn't help much.