NDO-3 query failed after each apply configuration

[jweijters]

@benjaminsmith

Further investigation on this issue we found that this problem could be created because of the scheduled_endtime is after 01-19-2038 (January 19th 2038 ) it looks the signed integer bug for the date see https://en.wikipedia.org/wiki/Year_2038_problem

I seems that mariadb can only handle :
https://mariadb.com/kb/en/from_unixtime/

Timestamps in MariaDB have a maximum value of 2147483647, equivalent to 2038-01-19 05:14:07. This is due to the underlying 32-bit limitation. Using the function on a timestamp beyond this will result in NULL being returned. Use DATETIME as a storage type if you require dates beyond this.

in nagios 5.6.14 we see that the the query also failed with a date being "Thu Dec 31 22:59:59 UTC 2099" however
it failed because the scheduled_endtime = 4102441199


kind regards

Joris Weijters

[WillemDH]

Hello,

As requested, we changed the order of the broke_module configuration, but that didn't seem to help.

Code: Select all

broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg
broker_module=/usr/lib64/mod_gearman/mod_gearman_nagios4.o config=/etc/mod_gearman/module.conf eventhandler=no

So we tried downgrading NDO3 to NDO2db with teh following procedure:

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db

During the install we get:

Code: Select all

*** Configuration summary for ndoutils 2.1.3 2017-04-13 ***:
 
 General Options:
 -------------------------
 NDO2DB user:      nagios
 NDO2DB group:     nagios
 NDO2DB tcp port:  5668
 

Review the options above for accuracy.  If they look
okay, type 'make all' to compile the NDO utilities,
or type 'make' to get a list of make options.
 
Makefile:49: *** missing separator.  Stop.

Unfortunately we are stuck at enabling the ndo2db service and we get the following error:

Code: Select all

Failed to execute operation: No such file or directory

It seems like the systemd configuration file is missing. Where can we get the correct ndo2db systemd config file?

We tested on our QA where the install script continued beyond the 'missing separator' issue and continued with:

Code: Select all

*** Configuration summary for ndoutils 2.1.3 2017-04-13 ***:
 
 General Options:
 -------------------------
 NDO2DB user:      nagios
 NDO2DB group:     nagios
 NDO2DB tcp port:  5668
 

Review the options above for accuracy.  If they look
okay, type 'make all' to compile the NDO utilities,
or type 'make' to get a list of make options.
 
cd ./src && make
make[1]: Entering directory `/tmp/nagiosxi/subcomponents/ndoutils/ndoutils-2.1.3/src'
gcc -fPIC -fPIC -g -O2 -I/usr/include/mysql -DHAVE_CONFIG_H  -c -o io.o io.c
gcc -fPIC -fPIC -g -O2 -I/usr/include/mysql -DHAVE_CONFIG_H  -c -o utils.o utils.c
gcc -fPIC -g -O2 -I/usr/include/mysql -DHAVE_CONFIG_H  -o file2sock file2sock.c io.o utils.o   -lm -lnsl
gcc -fPIC -g -O2 -I/usr/include/mysql -DHAVE_CONFIG_H  -o log2ndo log2ndo.c io.o utils.o   -lm -lnsl
make ndo2db-2x
make[2]: Entering directory `/tmp/nagiosxi/subcomponents/ndoutils/ndoutils-2.1.3/src'
gcc -fPIC -g -O2 -I/usr/include/mysql -DHAVE_CONFIG_H  -c -o db.o db.c
gcc -fPIC -g -O2 -I/usr/include/mysql -DHAVE_CONFIG_H  -D BUILD_NAGIOS_2X -c -o dbhandlers-2x.o dbhandlers.c
gcc -fPIC -g -O2 -I/usr/include/mysql -DHAVE_CONFIG_H   -D BUILD_NAGIOS_2X -o ndo2db-2x queue.c ndo2db.c dbhandlers-2x.o  io.o utils.o db.o    -lnsl -L/usr/lib64/mysql -lmysqlclient -lpthread -lz -lm -ldl -lssl -lcrypto -lm
make[2]: Leaving directory `/tmp/nagiosxi/subcomponents/ndoutils/ndoutils-2.1.3/src'

So what's going wrong with the install.sh and and the missing separator on our PR system?

Please advice how to continue downgrading NDO3 as we currently cannot apply configuration without risking nagios to crash.

Tx

Willem

[ssax]

Please attach the Makefile from here so we can see what it's failing on:

Code: Select all

/tmp/nagiosxi/subcomponents/ndoutils/ndoutils-2.1.3

[PeterDK]

Hi,

I've attached the Makefile.in file I got from the ndoutils-2.1.3.tar.gz
Had to rename it to txt, otherwise it wasn't allowed.

Kind regards,
Peter

[ssax]

That file looks like it wasn't updated when the ./configure was run by the script, I would try doing this again as root:

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi
./init.sh
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db

If you have an offloaded database you will need to edit your /usr/local/nagios/etc/ndo2db.cfg file and update these before running the next command to start it up:
- You can get the info from your /usr/local/nagios/etc/ndo.cfg or from /usr/local/nagiosxi/html/config.inc.php

Code: Select all

db_host
db_port
db_user
db_pass

Then run this command to start it up.:

Code: Select all

systemctl start ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure all occurrences of this line are commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios

If that doesn't resolve it, please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/

[PeterDK]

Hi,

two forum post are getting mixed up, so I suggest we lock this one?

I will continue https://support.nagios.com/forum/viewto ... 16&t=61378 for the load issues.

The solution to this one was indeed:

Code: Select all

systemctl stop nagios

vim /usr/local/nagios/var/retention.dat  -> remove all blocks with a high end_time. I searched for "end_time=2" and "end_time=4" for example, as those are mostly after year 2038.

systemctl start nagios

After that I reenabled downtime until somewhere in 2037.

Now all the NDO-3 warnings are gone from the logs.

Kind regards,
Peter

[jweijters]

After a good 1/2 day of running Nagios 5.8.1 our Nagios also crashes after a apply config

Code: Select all

[1611818636] SERVICE ALERT: mulc-mahtmgw01;Disk: /;UNKNOWN;HARD;10;ERROR: Description/Type table : No response from remote 
[1611818646] NDO-3: Ended flapping thread
[1611818646] NDO-3: Ended acknowledgement thread
[1611818646] NDO-3: Ended notification thread
[1611818646] NDO-3: Ended statechange thread
[1611818646] NDO-3: Ended event_handler thread
[1611818646] Caught SIGSEGV, shutting down...

We have fixed the faulty downtime. So that isn't the problem
I I look at the database cache we see the folowing:

Capture.JPG

After the reboot Nagios starts. We get a lot of inserts, then it slows down, then a lot of queries that host the cash, see the first highlight
When we ran the apply config, ( second highlight, the inserts start, however before the query Nagios already crashed.

Kind regards,
Joris Weijters

[PeterDK]

Correct. The downtime cleanup wasn't the issue for Nagios load and crashes.
We downgraded to NDO2 yesterday and everything seems to be back to normal.
See https://support.nagios.com/forum/viewto ... 16&t=61378 for more details.

[jweijters]

Hi Peter,

I also downgraded NDO-3 to NDO-2, I see this only as a workaround, but it works.


Kind regards,

Joris Weijters

[benjaminsmith]

Hi,

I also downgraded NDO-3 to NDO-2, I see this only as a workaround, but it works.

Glad to hear that is working for you now. The upgrade script will not automatically force an update to ndo3, if you'd like to continue to use ndo2 on newer versions.

--Benjamin