Can no longer upgrade through UI on version 5.6

[nickap]

When we tried to upgrade to from 5.5.11 to 5.6.3 we ran into the issue where the instructions box is blank. Following the instructions in another post we were able to successfully complete the upgrade:

https://support.nagios.com/forum/viewto ... 16&t=53606

Today we attempted to upgrade from 5.6.3 to 5.6.4 and ran into the same issue again.

Is this a bug? We never had issues before version 5.6

[benjaminsmith]

Hi @nickap,

We're you able to upgrade successfully upgrade to 5.6.4?

It could be a number of things causing the failure such as incorrect configurations, disk space, permissions..etc. If you haven't upgraded yet or during your next upgrade, please open a support ticket and we can capture key log data during the upgrade to isolate the root cause.

Is this a bug? We never had issues before version 5.6

It's a known issue with the the web update. In most cases, it works fine, but for some systems, I would recommend using the upgrade script from the command line instead. Capturing the log files on your server during the upgrade would help us understand what is failing in your system.

[nickap]

Yes, I was able to upgrade to 5.6.4 using SSH and the commands for the manual process.

I'll try and capture the logs next time. Should I open a support ticket now for the future upgrade or when a new release comes out?

[benjaminsmith]

The next time you try to upgrade, do this:

First, make sure it verifies properly:

Code: Select all

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Then run this command to make sure you don't have any crashed tables:

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

- If any of the sizes say NULL, it means that you have a crashed DB table, and you should repair the DB before upgrading by following this guide:

https://support.nagios.com/kb/article.php?id=24

Then check to make sure you have enough free space:

Code: Select all

df -h
df -i

Then check the permissions on the files:

Code: Select all

ls -lhd /usr/local/nagiosxi/var
ls -lhd /usr/local/nagiosxi/var/upgrades
ls -lh /usr/local/nagiosxi/tmp/upgrade.log

Here's mine for comparision:

Code: Select all

[root@xid var]# ls -lhd /usr/local/nagiosxi/var/upgrades
drwxrwxr-x. 2 apache nagios 36 Jul 15 11:16 /usr/local/nagiosxi/var/upgrades
[root@xid var]# ls -lhd /usr/local/nagiosxi/var
drwxrwxr-x. 7 nagios nagios 4.0K Jul 15 11:24 /usr/local/nagiosxi/var
[root@xid var]# ls -lhd /usr/local/nagiosxi/var/upgrades
drwxrwxr-x. 2 apache nagios 36 Jul 15 11:16 /usr/local/nagiosxi/var/upgrades
[root@xid var]# ls -lh /usr/local/nagiosxi/tmp/upgrade.log
-rwxrwxr-- 1 nagios nagios 0 Jul 15 11:18 /usr/local/nagiosxi/tmp/upgrade.log

Now run this tail command (as root) and LEAVE IT RUNNING:

Code: Select all

tail -Fn0 /var/log/http/error_log /var/log/httpd/ssl_error_log /usr/local/nagiosxi/var/cmdsubsys.log /usr/local/nagiosxi/var/load_url.log /usr/local/nagiosxi/var/eventman.log /usr/local/nagiosxi/tmp/upgrade.log

Then try to upgrade from the web interface:

If the upgrade fails, please send us the ENTIRE output from the screen (attach in a separate file), and ALSO attach these files as well:

Code: Select all

/usr/local/nagiosxi/tmp/upgrade.log
/etc/sudoers
[code]
Then send us the full output of these commands as well:
[code]
echo "select * from xi_commands where command IN ('1100', '1120');" | mysql -uroot -pnagiosxi nagiosxi
chage -l nagios
chage -l apache
grep nag /etc/group
grep "User \|Group " /etc/httpd/conf/httpd.conf
df -h
df -i

Let me know if you have any questions.

[GldRush98]

I'm seeing this on multiple XI instances now too. Machines that I have upgraded through the UI since as far back as 2016 are now hanging any time you try and run the upgrade through the UI. No config issues, no crashed tables, plenty of disk space, matched ownership to your dirs.
/usr/local/nagiosxi/tmp/upgrade.log is empty every time.

[ssax]

Sometimes that happens, usually it's pretty good, it's most likely related to customization of the system (like hardening, setting a password on the nagios/apache accounts, expired passwords, sudoers issues, PAM issues, permissions issues, http/https misconfiguration or redirection issues, VPN routing/dns issues, etc), there are soooo many ways that the web interface upgrade could break that it's very hard to code for that.

Please follow benjaminsmith's instructions in his previous post and provide us with the full output of all commands requested, we need to be able to review the output in its entirety so that we can determine what the issue is so that we can move forward in troubleshooting (there may be things that may look normal to you but will stand out to us).

Just treat each one as a unique separate issue until we determine what the actual problem is for both, one could be failing for completely different reasons. Please send the commands and profiles from the multiple systems experiencing the issue so that we can compare them.

Thank you!

[nickap]

I tried to upgrade this morning and followed the instructions by benjamin. There are no errors with the configs, permissions are correct, and there is plenty of free space. When I tail the logs it keeps spamming this error on line 491 in nagiosim.inc.php (note I changed the IP address to 127.0.0.1 in the logs). Also the upgrade.log is 0KB and empty:

==> /usr/local/nagiosxi/var/cmdsubsys.log <==
PROCESSING COMMAND ID 10361...
PROCESS COMMAND: CMD=1120, DATA=a:1:{i:0;s:61:"https://assets.nagios.com/downloads/nag ... est.tar.gz";}
..
==> /var/log/httpd/ssl_error_log <==
[Thu Jul 25 07:32:09 2019] [error] [client 127.0.0.1] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 491, referer: https://myserver/nagiosxi/admin/updates.php?&=

==> /usr/local/nagiosxi/var/cmdsubsys.log <==
.
==> /usr/local/nagiosxi/var/eventman.log <==
.
==> /usr/local/nagiosxi/var/cmdsubsys.log <==
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 58049118 bytes) in /usr/local/nagiosxi/html/includes/utilsx.inc.php on line 179

==> /var/log/httpd/ssl_error_log <==
[Thu Jul 25 07:32:11 2019] [error] [client 127.0.0.1] PHP Warning: Invalid argument supplied for foreach() in /usr/local/nagiosxi/html/includes/components/nagiosim/nagiosim.inc.php on line 491, referer: https://myserver/nagiosxi/admin/?xiwindow=updates.php

[tgriep]

It looks like the PHP settings on your system needs to be increased.
To do that, follow this article.
https://support.nagios.com/kb/article/n ... e-611.html

This is what caused the upgrade to fail in the GUI.

Code: Select all

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 58049118 bytes) in /usr/local/nagiosxi/html/includes/utilsx.inc.php on line 179

[nickap]

I can confirm this resolved the issue. I increased the memory limit from 128MB to 512MB.

[scottwilkerson]

I can confirm this resolved the issue. I increased the memory limit from 128MB to 512MB.

Great!

Locking thread