Performed offline upgrade from 5.10.0 to 5.11.3 and now monitoring is not working

[Ken.Hall]

I performed a manual backup of the Nagios XI server and then ran the offline upgrade.sh script. Everything seemed to go fine as all the rpm packages installed without error. However upon logging in to verify everything is still in good working order, I am seeing all kinds of issues. First off, I noticed the hostgroup and service group grids would not display. When I check the system profile, it still says Nagios XI version 5.10.0. Checking the monitoring engine status says the process state is Not Running and when I try to restart it, it fails saying the request was not processed in a timely manner.

I am fairly new to nagios administration so I am at a loss as to what I can do to resolve this problem. Any help would be much appreciated.

[ajcoil]

Hi @Ken.Hall,

Thank you for reaching out!

I am sorry to hear that your are having issues with your XI server.

I would recommend the following the documentation to help guide you in upgrading an offline environment:
https://assets.nagios.com/downloads/nag ... onment.pdf

If you are still having issues, feel free to reach out!

[Ken.Hall]

That is the exact document I followed to perform the upgrade. As I mentioned, everything seemed to go smoothly and I received no errors during the upgrade. After the installation completed successfully, I logged into the Web UI and that is when I noticed problems. I have since rolled the machine back to last nights snapshot and it is back up and running on the 5.10.0 version. I am looking for a way to upgrade this server to the latest version without any issues now but all I have found is the steps outlined in the document you linked.

At this point I guess I would like to know if I need to do anything else prior to upgrading the server like stop the service or run any script before/after the upgrade that would make this endeavor a success.

To be clear, my steps were:
Run the backup script: /usr/local/nagiosxi/scripts/backup_xi.sh
Perform the steps in the document for an offline upgrade:
cd /tmp/rpms
tar xzf nagiosxi-5.11.3-1.el8.x86_64.tar.gz
cd nagiosxi
./upgrade

[jsimon]

Hi Ken,

When you run the upgrade, do you get a message telling you:

Code: Select all

---------------------------
Nagios XI Upgrade Complete!
---------------------------

If you don't see error output but you also don't get this message, there may be an error happening in the upgrade process. I would recommend trying the upgrade again, and seeing if you get this message. If not, check:

Code: Select all

 /usr/local/nagiosxi/tmp/upgrade.log

for error messages with the upgrade process. If you have errors in your log that you aren't able to resolve on your end, provide them here for advisable next steps!

[Ken.Hall]

This machine is behind a firewall that does not allow external access so the upgrade is being performed in an offline manner using the steps in the documentation as mentioned above. When I execute the ./upgrade command, it installs several rpms and that is all. the output from that shows no errors other than nagios.cfg being created as nagios.cfg.new which is expected. Literally, all I am seeing is what I have included below.


Installed products updated.

Upgraded:
nagiosxi-5.11.3-1.el8.x86_64 nagiosxi-mrtg-5.11.3-1.el8.x86_64 nagiosxi-nagioscore-5.11.3-1.el8.x86_64 nagiosxi-nagiosplugins-5.11.3-1.el8.x86_64 nagiosxi-nagvis-5.11.3-1.el8.x86_64
nagiosxi-ndoutils-5.11.3-1.el8.x86_64 nagiosxi-nrds-5.11.3-1.el8.x86_64 nagiosxi-nrpe-5.11.3-1.el8.x86_64 nagiosxi-nsca-5.11.3-1.el8.x86_64 nagiosxi-nxti-5.11.3-1.el8.x86_64
nagiosxi-pnp-5.11.3-1.el8.x86_64 nagiosxi-shellinabox-5.11.3-1.el8.x86_64 nagiosxi-wkhtmltox-5.11.3-1.el8.x86_64 nagiosxi-wmic-5.11.3-1.el8.x86_64

Complete!

I should also add that after the upgrade and startup of the nagios process, the system information still shows Nagios XI version: 5.10.0 as if it did not actually update. When I look for the logfile you mentioned, it does not exist. However, there is a /tmp/xi-upgrade.log with one line:
/var/tmp/rpm-tmp.z0BdS6: line 30: ./rpmupgrade: Permission denied

Why I would get any permission denied is unknown. I am performing this upgrade as root on the system.

[swolf]

Hi @Ken.Hall - I haven't run into your issue before, but if you're okay with using your terminal a bit more, that's where I'd go next.

The "Monitoring Engine" in the web interface is directly tied to the `nagios` systemd service - to debug your specific issue, I'd want to get back to the spot where the Monitoring Engine is stopped, and I would run

Code: Select all

systemctl restart nagios
systemctl status nagios

and see what the output is - it's a bit concerning to me that the web interface would be non-responsive rather than giving you an error, but from the terminal there's a pretty strong chance that you'll get a coherent error.

In general, you're roughly at the point where I would open a case with the Support Team proper.
You're in a somewhat hairy spot and they will probably be able to give better guidance and/or be able to collect better information than we can in public.

[Ken.Hall]

I guess I should have been clear on that. I am doing _everything_ from a terminal rather than upgrading from the web UI. The systemd service says that nagios is happily running. (just not upgraded)

● nagios.service - Nagios Core 4.4.13
Loaded: loaded (/usr/lib/systemd/system/nagios.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-11-16 07:18:45 PST; 20min ago
Docs: https://www.nagios.org/documentation
Process: 2316625 ExecStart=/usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg (code=exited, status=0/SUCCESS)
Process: 2316623 ExecStartPre=/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg (code=exited, status=0/SUCCESS)
Main PID: 2316626 (nagios)
Tasks: 26 (limit: 102126)
Memory: 30.5M
CGroup: /system.slice/nagios.service
├─2316626 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
├─2316627 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316628 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316629 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316630 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316631 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316632 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316633 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316634 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316635 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316636 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316637 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
├─2316638 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
└─2316658 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

Also noticed that when I login to the webUI (which works) I see that the monitoring engine process is not running. Attempts to restart it fail.