Nagios Support Forum

Posted: **Thu Mar 03, 2022 4:56 am**

Hello community!

We have a "little" problem with the NagiosXI upgrade process (5.6.7 -> 5.8.7). We have two servers in different environments, one in Production env. (PROD) and other in integration env. (INT). None of the servers have internet access even the installation was a normal one (no offline installation). We've requestes access to some URLs and We've tested the upgrade process in INT to test everything we need to upgrade in a safe way the PROD server.

PROD server:
32 CPUs
128 GB RAM

INT server:
8 CPUs
32 GB RAM

Both servers have their database installed locally, and despite of having different hardware resources, regarding software they are supposed to be mirror servers. We use INT as the last test before publishing new features to PROD.

We are suffering poblems inserting the data in nagios database using ndoutils, ndo2db is too slow and is not able to insert all the data in the DB. I'm pretty sure that we didn't reach the limit of the server, it has enough resources to handle the double amount of assets. We have an old server that is handling the double amount of assets, however is inserting less objects macros. Based on the realase notes, NDOutils3 has improved performance inserting data in the database, so it is the solution for our main problem.

cat /etc/sysctl.conf | grep kernel

Code: Select all

kernel.randomize_va_space = 2
kernel.msgmnb = 796432000
kernel.msgmax = 796432000
kernel.shmmax = 4294967295
kernel.shmall = 268435456

ipcs -q

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x49000040 32802      nagios     600        796288000    777625

Meanwhile we were debuging the problem with NDOutils, we realized that our PROD server has a non updated ndoutils package and ipcs queues are maxed out.

Code: Select all

UPGRADE: Nagios Core upgraded OK.
UPGRADE: NDO is being upgraded...
UPGRADE: NDO manually downgraded -- skipping
UPGRADE: Nagios Plugins is being upgraded...
Applying Nagios XI patches to Nagios Plugins...
checking for a BSD-compatible install... /bin/install -c
checking whether build environment is sane... yes

I'd like to figure out why the NDOutils component was not updated during the NagiosXI upgrade:

nagiosxi/subcomponents/ndo/upgrade

Code: Select all

function version_gt ()
{
    # Copied from ndo-3 db-mgmt.sh
    # thanks stackoverflow #16989598/bash-comparing-version-numbers
    test "$(printf '%s\n' "$@" | sort -V | head -n 1)" != "$1"
}

if [ -f '/usr/local/nagios/bin/ndo2db' ] && [ "$1" != "-f" ]; then

...
# Grab version from database
ndoversion=$(mysql "$ndoname" -u "$ndouser" -p"$ndopass" -h "$ndohost" -P "$ndoport" -se "SELECT version FROM nagios_dbversion WHERE name='ndoutils';")
if version_gt "$ndoversion" "2.1.3"; then
    # Downgraded system
    echo "UPGRADE: NDO manually downgraded -- skipping"
    exit 0
fi
...

Code: Select all

 mysql -u$user -D nagios -se "SELECT version FROM nagios_dbversion WHERE name='ndoutils';" -p
Enter password:
version
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1
2.0.1

Could we do a manual upgrade of that component without affecting the rest of the components?, how risky is that process?

And last but no least, with NDOUtils3.x how can we manage it?

Thanks in advance.

BR,
Juanma.

Posted: **Thu Mar 03, 2022 5:25 pm**

Hello @jmsanesteban.sgre

Thanks for reaching out, and you are correct that executing the installer in the subcomponents

Code: Select all

systemctl stop nagios
cd /tmp
nagiosxi/subcomponents/ndo/upgrade

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is commented out:

#broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg
Make sure this line is uncommented:
broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

Let us know how that looks.

Thanks,
Perry

Posted: **Wed Mar 09, 2022 4:11 am**

Hi community!

We didn't upgrade ndoutils yet, we are evaluating it but we should answer several questions before update it:

Why the SQL query to show the version returned 12 results for ndoutils version?
NDOUtils is an isolated software, so in case of problems could we downgrade manually without impacting the whole soft installation?
I saw some posts talking about performance problems in big installations using NDO3, apart from install the package and change the broker, is there any other conf to change to avoid those performance problems?
In Older ndo2db versions, we use message queues, I can't see a Nagios message queues in the servers I've installed NDO3, how can I troubleshooting message problems without it?

Server using NDO2DB 2.1.3

Code: Select all

 ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x91000040 6          nagios     600        520054784    507866

Server using NDOUtils version 3.0.7

Code: Select all

 ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

Thanks in advance.

BR,
Juanma.

Posted: **Fri Mar 11, 2022 3:41 pm**

Hello @jmsanesteban

Thanks for following up.

Question:

Why the SQL query to show the version returned 12 results for ndoutils version?

I am not sure on this and will need to check into this.

Question:

NDOUtils is an isolated software, so in case of problems could we downgrade manually without impacting the whole soft installation?

Yes, you are correct and here are the instructions:
We see that you are currently on 5.8.3 Nagios XI version and want to have you download the matching installer and run through the subcomponents install for ndoutils.

Code: Select all

wget https://assets.nagios.com/downloads/nagiosxi/xxx/xi-x.x.x.tar.gz
.../nagiosxi/subcomponents/ndo/
tar -xvf ndoutils-2.1.3.tar.gz
cd ndoutils-2.1.3
./install-sh

Code: Select all

systemctl enable ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

Here is a support article that provides a really good outline as well for your reference:
https://support.nagios.com/kb/article.php?id=406

Question:

I saw some posts talking about performance problems in big installations using NDO3, apart from install the package and change the broker, is there any other conf to change to avoid those performance problems?

For the most part we have corrected most of the issue surrounding this and performance is not a problem on the latest version.

In Older ndo2db versions, we use message queues, I can't see a Nagios message queues in the servers I've installed NDO3, how can I troubleshooting message problems without it?

you are correct that the 'ipcs' is not going to pick up message queues with NDO3. NDOutil no longer uses a message queue as it is direct "fed".

Please let us know the results,
Perry

Posted: **Tue Mar 15, 2022 5:43 am**

Hi all,

Thanks for your comments and answers

Definitely the problem were those 12 lines with ndoutils versions that were in the table. It could be great to update the "upgrade" script to avoid that problem. the query returned "2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 2.0.1 " , but in order to do that it makes sense to understand how a system could add those lines. Unfortunately as I've updated that component now I only have one entry with the correct version, sorry because I can't help to find the root cause.

Thanks all for your comments, time and effort.

BR,
Juanma.

Posted: **Tue Mar 15, 2022 3:29 pm**

Hello @jmsanesteban.sgre

Thanks for following up, please let us know if you need anything further,

Perry

Posted: **Sat Mar 19, 2022 4:08 am**

Hi all,

No problem, everything is ok from my side, but as I said, it could be great to avoid this problem to other users in the future, so understanding how this problem appeared will help to update "upgrade" script to handle that situation conuld be perfect.

Thanks.

BR,
Juanma.

Posted: **Mon Mar 21, 2022 8:52 am**

Thanks for the feedback, @jmsanesteban and will have dev take a look at updating the script.

I will go ahead and lock this one,
Perry

We're moving to a new support system!

The Nagios Answer Hub is a place where you can get help with technical questions from our experts. There, you can quickly open tickets and join discussion boards.

Request Nagios Answer Hub access here: https://info.nagios.com/answer-hub-access-new-users

After completing the access form, you will be given access to a portal where new tickets can be created. We will keep the old customer forum sections and ticket system available for current cases to be resolved.

Nagios Support Forum

Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Re: Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Re: Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Re: Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Re: Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Re: Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Re: Upgrade process fail. Missing components. 5.6.7 -> 5.8.7

Re: Upgrade process fail. Missing components. 5.6.7 -> 5.8.7