Page 1 of 1
NDO-3 Crashing
Posted: Fri Jan 22, 2021 1:48 pm
by rdurga
Running NAGIOS XI 5.7.5 on RHEL 7, 64 bit manual install. We are having problems with NDO-3 crashing. When the issue occurs non of the Nagios XI checks from other servers fail to this box, but Nagios XI stops sending notifications. We found the following in the logs.
[1611308094] NDO-3: The following query failed while MySQL appears to be connected:
[1611308094] NDO-3: INSERT INTO nagios_configfiles (instance_id, configfile_type, configfile_path) VALUES (1, 0, '/usr/local/nagios/etc/nagios.cfg')
[1611308095] NDO-3: ndo_return = 1 (Statement not prepared)
[1611308095] NDO-3: ndo_process_nagios_config(ndo-startup.c:149): Unable to bind parameters
[1611308095] NDO-3: NDO startup thread failed at ndo_write_config_files() - disabling NDO.
[1611308095] Caught SIGSEGV, shutting down...
Re: NDO-3 Crashing
Posted: Fri Jan 22, 2021 2:19 pm
by tgriep
Run the following as root to repair the MYSQL database to be sure it is clean.
Code: Select all
mysqlcheck -f -r -u root -pnagiosxi --databases nagios --use-frm
Attached to the post is the updated NDO3 broker.
Copy the ndo-master.zip file to the XI server and run the following to compile, install it and to restart nagios.
Code: Select all
unzip ndo-master.zip
cd ndo-master
./configure
make all
make install
Save the file and then restart nagios.
Re: NDO-3 Crashing
Posted: Mon Jan 25, 2021 10:28 am
by hbouma
Hi,
I am helping Ron with this. I think the issue may be due to the fact that we have an offloaded database and also have downgraded from NDO3 back to NDO2DB. When performing the downgrade, this server gives the following error and fails.
Code: Select all
sysctl: setting key "kernel.msgmni": Invalid argument
kernel.msgmni = 512000
NOTE: If prompted, enter the MySQL root password
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
We have run the provided Database command a few times and get the following output each time:
Code: Select all
nagios.nagios_acknowledgements
warning : Number of rows changed from 0 to 388
status : OK
nagios.nagios_commands
warning : Number of rows changed from 0 to 173
status : OK
nagios.nagios_commenthistory
warning : Number of rows changed from 0 to 205505
status : OK
nagios.nagios_comments
warning : Number of rows changed from 0 to 7
status : OK
nagios.nagios_configfiles
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_configfilevariables
warning : Number of rows changed from 0 to 133
status : OK
nagios.nagios_conninfo
warning : Number of rows changed from 0 to 5036
status : OK
nagios.nagios_contact_addresses
warning : Number of rows changed from 0 to 492
status : OK
nagios.nagios_contact_notificationcommands
warning : Number of rows changed from 0 to 2084
status : OK
nagios.nagios_contactgroup_members
warning : Number of rows changed from 0 to 1601
status : OK
nagios.nagios_contactgroups
warning : Number of rows changed from 0 to 95
status : OK
nagios.nagios_contactnotificationmethods
warning : Number of rows changed from 0 to 96295
status : OK
nagios.nagios_contactnotifications
warning : Number of rows changed from 0 to 96293
status : OK
nagios.nagios_contacts
warning : Number of rows changed from 0 to 478
status : OK
nagios.nagios_contactstatus
warning : Number of rows changed from 0 to 478
status : OK
nagios.nagios_customvariables
warning : Number of rows changed from 0 to 6775
status : OK
nagios.nagios_customvariablestatus
warning : Number of rows changed from 0 to 6775
status : OK
nagios.nagios_dbversion
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_downtimehistory
warning : Number of rows changed from 0 to 434626
status : OK
nagios.nagios_eventhandlers
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_externalcommands
warning : Number of rows changed from 0 to 2744756
status : OK
nagios.nagios_flappinghistory
warning : Number of rows changed from 0 to 22642
status : OK
nagios.nagios_host_contactgroups
warning : Number of rows changed from 0 to 872
status : OK
nagios.nagios_host_contacts
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_host_parenthosts OK
nagios.nagios_hostchecks OK
nagios.nagios_hostdependencies OK
nagios.nagios_hostescalation_contactgroups OK
nagios.nagios_hostescalation_contacts OK
nagios.nagios_hostescalations OK
nagios.nagios_hostgroup_members
warning : Number of rows changed from 0 to 1661
status : OK
nagios.nagios_hostgroups
warning : Number of rows changed from 0 to 155
status : OK
nagios.nagios_hosts
warning : Number of rows changed from 0 to 710
status : OK
nagios.nagios_hoststatus
warning : Number of rows changed from 0 to 710
status : OK
nagios.nagios_instances
warning : Number of rows changed from 0 to 2
status : OK
nagios.nagios_logentries
warning : Number of rows changed from 0 to 7675451
status : OK
nagios.nagios_notifications
warning : Number of rows changed from 0 to 346568
status : OK
nagios.nagios_objects
warning : Number of rows changed from 0 to 35173
status : OK
nagios.nagios_processevents
warning : Number of rows changed from 0 to 6981
status : OK
nagios.nagios_programstatus
warning : Number of rows changed from 0 to 1
status : OK
nagios.nagios_runtimevariables
warning : Number of rows changed from 0 to 18
status : OK
nagios.nagios_scheduleddowntime OK
nagios.nagios_service_contactgroups
warning : Number of rows changed from 0 to 6364
status : OK
nagios.nagios_service_contacts
warning : Number of rows changed from 0 to 75
status : OK
nagios.nagios_service_parentservices OK
nagios.nagios_servicechecks OK
nagios.nagios_servicedependencies OK
nagios.nagios_serviceescalation_contactgroups OK
nagios.nagios_serviceescalation_contacts OK
nagios.nagios_serviceescalations OK
nagios.nagios_servicegroup_members
warning : Number of rows changed from 0 to 550
status : OK
nagios.nagios_servicegroups
warning : Number of rows changed from 0 to 90
status : OK
nagios.nagios_services
warning : Number of rows changed from 0 to 6349
status : OK
nagios.nagios_servicestatus
warning : Number of rows changed from 0 to 6348
status : OK
nagios.nagios_statehistory
warning : Number of rows changed from 0 to 1147350
status : OK
nagios.nagios_systemcommands
warning : Number of rows changed from 0 to 65
status : OK
nagios.nagios_timedeventqueue OK
nagios.nagios_timedevents OK
nagios.nagios_timeperiod_timeranges
warning : Number of rows changed from 0 to 3400
status : OK
nagios.nagios_timeperiods
warning : Number of rows changed from 0 to 488
status : OK
Re: NDO-3 Crashing
Posted: Mon Jan 25, 2021 6:02 pm
by tgriep
Can you get the following file from the Nagios server and upload it here so we can view it?
It looks like there is a syntax error in it.
Re: NDO-3 Crashing
Posted: Tue Jan 26, 2021 10:40 am
by hbouma
This issue doesn't appear to the /etc/syscfg.conf file. it appears to be due to the fact that we have offloaded the DB to a separate server.
It is failing at this part of the post-install program from ndo2db. After I commented it out and reran the install file, the downgrade worked great.
Code: Select all
# See if database already exists
#echo "NOTE: If prompted, enter the MySQL root password"
#if mysql nagios -u root -p"************" -e 'SELECT COUNT(*) FROM nagios_hosts' &>/dev/null; then
# echo "NDOUtils database already exists - skipping creation..."
#else
# sqlfile="create_db.sql"
# Remove default charset on el8 and change latin1 to utf8 in db file
# if [ "$dist" == "el8" ]; then
# sed -i 's/character set latin1 /character set utf8 /' "$pkgname"/db/mysql.sql
# sed -i 's/"charset" => "latin1"/"charset" => "utf8"/' ../../nagiosxi/basedir/html/config.inc.php
# sqlfile="create_db_mysql8.sql"
# fi
# Run SQL to create database and add user
# mysql -u root -p"$mysqlpass" < mods/$sqlfile
##parse values in case mysql is offloaded
# ndopass=$(sed -n '/^db_pass=/ s///p' /usr/local/nagios/etc/ndo2db.cfg)
# ndohost=$(sed -n '/^db_host=/ s///p' /usr/local/nagios/etc/ndo2db.cfg)
# ndouser=$(sed -n '/^db_user=/ s///p' /usr/local/nagios/etc/ndo2db.cfg)
# Install the database
# -- Apply some changes for newer MySQL restrictions if the ndo version isn't updated
# (
# cd ./"$pkgname"/db
# sed -i "s/\`command_line\` varchar(255)/\`command_line\` TEXT/" mysql.sql
# sed -i "s/'0000-00-00 00:00:00'/'1970-01-01 00:00:01'/" mysql.sql
# ./installdb -u "$ndouser" -p "$ndopass" -h "$ndohost" -d nagios
# )
Re: NDO-3 Crashing
Posted: Tue Jan 26, 2021 2:24 pm
by tgriep
I see what the issue was now. Thanks.