Questions regarding updating Nagios XI to version 5.7.4

[tony]

We are about to upgrade Nagios XI from version 5-6-14 to 5.7.4 (we also use mod-gearman and worker servers to off load active checks)

I have a question regarding mod-gearman version 3.0.7 which we use.

1. Does Nagios XI version 5.7.4 fully support mod-gearman version 3.0.7 and its worker servers


I also have a few questions regarding NDO:

I noticed in the change log the following statement
“Updated NDOutils to NDO 3.0.0 for performance increase and no longer using kmq or the ndo2db daemon”

Ndo2d caused us a few issues in the past with duplicate message queues and message queues filling up and
we had to change the system kernel settings - msgmni

1. Can you explain the reasons for moving from ndo2b to NDO – will this resolve the above issues and improve performance.
2. How does NDO run - on my test system I do not see a service \ daemon?
3. When stopping and starting Nagios is there any particular order to stopping and starting services
4. Will the upgrade stop and remove the ndo2b service?
Or do we have to manually delete it.?
5. Is the new NDO architecture described anywhere - https://assets.nagios.com/downloads/nag ... OUtils.pdf

[nickap]

I highly recommend you take a snapshot or backup just before upgrading. We've been unable to successfully upgrade to 5.7 since it's release over 5 months ago due to performance issues and bugs. Our support ticket is still open to this day.

[ssax]

Definitely take a VM snapshot (if it's a VM) and an XI backup, just in case:

https://assets.nagios.com/downloads/nag ... ios-XI.pdf

You will need to upgrade mod_gearman to the latest on the XI server and ALL of the workers for it to be compatible with XI 5.7+, you follow this guide to do that:

https://assets.nagios.com/downloads/nag ... ios_XI.pdf

NOTE: You will have issues if you do not upgrade gearman on the XI server and all of the workers.

1. The Kernel Message Queue is the slowest method of interprocess communication in the system. NDO3 was written to get rid of the KMQ and communicate directly with the DB. The intention is to get rid of the KMQ to help alleviate performance issues when the systems are larger. NOTE: Some systems have issues with the new NDO3 on 5.7+, if you have issues with host/service statuses not updating in the web interface, checks stuck in pending, or new hosts/services not showing up after an apply config you may need to downgrade NDO3 back to NDO2DB. If you see those issues, let us know and send us a copy of your profile so we can tailor the commands to your system.

2. NDO3 is a neb module so it doesn't use a service anymore (if you have an NDO2DB service check you can disable it). Nagios Core communicates directly with the NDO3 neb module via the broker line which communicates directly with the DB instead of utilizing the KMQ.

3.

Stopping:

Code: Select all

systemctl stop nagios
systemctl stop mod-gearman-worker
systemctl stop gearmand

Starting:

Code: Select all

systemctl start gearmand
systemctl start mod-gearman-worker
systemctl start nagios

4. Yes, it should.

5. Not currently, I have submitted a feature request for it to be added:

FR: XI - Add NDO3 to architecture documentation

Please keep in mind that the decision to implement the enhancement is at the discretion of our development team.

[tony]

Thanks for the helpful answers to my queries. I just need to clarify the following.

1. What is the latest version of Mod Gearman compatible with Nagios 5.7.4.
We are running version 3.0.7 which we installed in June.
This version is still referenced in the ModGearmanInstall.sh script listed in the guide
https://assets.nagios.com/downloads/nag ... ios_XI.pdf

2. Under what circumstances have, you seen issues with the new NDO3 on 5.7+ and have had to downgrade NDO3 back to NDO2DB.
Have you seen many upgrades failing.
Does this only occur on large systems and if so what would you classify as a large system – how many service checks?

3. How do you monitor the health status of NDO 3.

NDO3 is a now a neb module therefore it does not use a service anymore.
How can you monitor a NEB module, is there a process we can monitor?

[ssax]

1. Here are the latest when using our setup script:

Code: Select all

[root@xig ~]# rpm -qa |grep gearman
mod_gearman-3.3.0-1.x86_64
gearmand-1.1.19-1.x86_64
gearmand-server-0.33-7.x86_64

2. The issues listed are the issues we've seen:

NOTE: Some systems have issues with the new NDO3 on 5.7+, if you have issues with host/service statuses not updating in the web interface, checks stuck in pending, or new hosts/services not showing up after an apply config

You'll need to upgrade and test to find out if your system hits any of them, we were unable to replicate it on our own internal systems during testing which is why it only occurs on some systems. The workaround is to downgrade NDO3 back to NDO2DB via the instructions I previously sent that were tailored to your system.

3. There isn't a way that I'm aware of currently, the neb module is called directly from nagios core so there is no service/queue to check in the new NDO3 at this point in time. I've submitted a feature request to development on your behalf with a link back to this thread:

Code: Select all

FR: XI - NDO3 - Add the functionality to be able to monitor it to see if it's working.

Please keep in mind that the decision to implement the enhancement is at the discretion of our development team.

Thank you!

[tony]

Regarding your reply - "The workaround is to downgrade NDO3 back to NDO2DB via the instructions I previously sent that were tailored to your system."

You have not sent me the instructions to downgrade NDO3 back to NDO2DB.
I would like to have these instructions available in case we have the issues described and we can then quickly resolve the issues.

How do you reinstall ndo2db
- Where do you get the software
- How do you compile and reinstall
- How do you start the service

How do you integrate ndo2db with nagios

How do you uninstall NDO3
How do you stop nagios using NDO3

[ssax]

Woops, sorry about that, I reused that output so ignore that sentence, we'll need your profile:

If you see those issues, let us know and send us a copy of your profile so we can tailor the commands to your system.

Please PM me a copy of your profile and I'll send you the instructions, you can download it from Admin > System Profile by clicking the Download Profile button.

[jwgill]

I just updated from 5.6.14 to 5.7.4 and probably need the same instructions. I see events happening in nagios.log, but nothing is getting to the webpage for display, and the monitoring engine shows nothing to be running.

I did get everything working at one point, but only after repairing the database, and restarting httpd. Once the system reboots, everything stops and it's all broken again.

This last time, nothing I do will get it to start. working. I'm glad I took a pre-upgrade snapshot.

[tony]

My manager would like us to have the instructions to downgrade before we start the upgrade.

However we have 2 production Nagios XI systems and 1 DEV one.

Can you not provide the instructions without me sending a profile as I would have to send you 2 production profiles and 1 DEV profile.

Is there not a generic set of instructions

[rhasg]

I've used the NDO downgrade instructions in the second post here: https://support.nagios.com/forum/viewto ... 16&t=60392

All seems to work fine with 5.7.4 after downgrading NDO.