Nagios service constantly exited

[Shades007]

Hi Support,

For the past 3 days, we encountered Nagios service keep exiting service and then monitoring engine not started even after Nagios service has been restarted.

Errors seen in nagios.logs just before the services exited as below.

Caught SIGSEGV, shutting down...
[1635437251] NDO-3: Started timed_event thread
[1635437251] NDO-3: Started event_handler thread
[1635437251] NDO-3: Started host_check thread
[1635437251] NDO-3: Started service_check thread
[1635437251] NDO-3: Started comment thread
[1635437251] NDO-3: Started flapping thread
[1635437251] NDO-3: Started downtime thread
[1635437251] NDO-3: Started service_status thread
[1635437251] NDO-3: Started host_status thread
[1635437251] NDO-3: Started contact_status thread
[1635437251] NDO-3: Started acknowledgement thread
[1635437251] NDO-3: Started statechange thread
[1635437251] NDO-3: Started notification thread
[1635437253] NDO-3: Ended contact_status thread
[1635437266] NDO-3: Ended host_check thread
[1635437268] NDO-3: Ended host_status thread


Appreciate the feedback and advise. Thank you.....

[benjaminsmith]

Hi @Shades007,

There's a backend application called NDO3 that writes all the check output from the nagios service into the database, and that is crashing. The likely cause is the database tables that were not configured or updated properly.

If this is a production system, the immediate resolution would be to change from NDO3 to NDO2 so this does not impact your ability to monitor.

# Convert NDO3 to NDO2

Code: Select all

systemctl stop nagios
cd /tmp
rm -rf /tmp/nagiosxi
wget https://assets.nagios.com/downloads/nagiosxi/5/xi-5.6.14.tar.gz
tar zxf xi-5.6.14.tar.gz
cd /tmp/nagiosxi/subcomponents/ndoutils
./install
systemctl enable ndo2db

Then edit your /usr/local/nagios/etc/nagios.cfg and make sure this line is uncommented:

Code: Select all

broker_module=/usr/local/nagios/bin/ndomod.o config_file=/usr/local/nagios/etc/ndomod.cfg

Make sure this line is commented:

Code: Select all

#broker_module=/usr/local/nagios/bin/ndo.so /usr/local/nagios/etc/ndo.cfg

Then start the nagios service:

Code: Select all

systemctl start nagios
systemctl start ndo2db

Otherwise, we can review the profile and database settings to troubleshoot the system. Please attach the system profile and the output to the following command. Thanks, Benjamin

Code: Select all

mysqldump --no-data --database nagios -u username -phassword -h xxx.xxx.xxx.xxx >nagios.sql

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button

[Shades007]

hi ben,

have pm you the profile.


Thanks..

[benjaminsmith]

Hi,

Thanks for the profile. Can you send over the following output file (nagos.sql)? I'd like to compare the database settings with a known good system.

Code: Select all

mysqldump --no-data --database nagios -u username -phassword -h xxx.xxx.xxx.xxx > nagios.sql

On a default system,that command would be:

Code: Select all

 mysqldump --no-data --database nagios -u ndoutils -pn@gweb -h localhost  > /tmp/nagios.sql

Thanks, Ben

[Shades007]

Hi Ben,

Please find as requested.


-- MySQL dump 10.14 Distrib 5.5.65-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: nagios
-- ------------------------------------------------------
-- Server version 5.5.65-MariaDB

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Current Database: `nagios`
--

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `nagios` /*!40100 DEFAULT CHARACTER SET latin1 */;

USE `nagios`;

--
-- Table structure for table `alc`
--

DROP TABLE IF EXISTS `alc`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `alc` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ipaddress` varchar(255) NOT NULL,
`region` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `bdc`
--

DROP TABLE IF EXISTS `bdc`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `bdc` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ipaddress` varchar(255) NOT NULL,
`region` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `hla`
--

DROP TABLE IF EXISTS `hla`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `hla` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ipaddress` varchar(255) NOT NULL,
`region` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=28 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `hlib`
--

DROP TABLE IF EXISTS `hlib`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `hlib` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ipaddress` varchar(255) NOT NULL,
`region` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `hlisb`
--

DROP TABLE IF EXISTS `hlisb`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `hlisb` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ipaddress` varchar(255) NOT NULL,
`region` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=8 DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `limit_t1`
--

DROP TABLE IF EXISTS `limit_t1`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `limit_t1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`central_t1` int(3) NOT NULL,
`north_t1` int(3) NOT NULL,
`ecaost_t1` int(3) NOT NULL,
`south_t1` int(3) NOT NULL,
`sabah_t1` int(3) NOT NULL,
`sarawak_t1` int(3) NOT NULL,
`timestamp` datetime NOT NULL,
`by` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `limit_t2`
--

DROP TABLE IF EXISTS `limit_t2`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `limit_t2` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`central_t2` int(3) NOT NULL,
`north_t2` int(3) NOT NULL,
`ecaost_t2` int(3) NOT NULL,
`south_t2` int(3) NOT NULL,
`sabah_t2` int(3) NOT NULL,
`sarawak_t2` int(3) NOT NULL,
`timestamp` datetime NOT NULL,
`by` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `limit_t3`
--

DROP TABLE IF EXISTS `limit_t3`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `limit_t3` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`central_t3` int(3) NOT NULL,
`north_t3` int(3) NOT NULL,
`ecaost_t3` int(3) NOT NULL,
`south_t3` int(3) NOT NULL,
`sabah_t3` int(3) NOT NULL,
`sarawak_t3` int(3) NOT NULL,
`timestamp` datetime NOT NULL,
`by` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `limit_total`
--

DROP TABLE IF EXISTS `limit_total`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `limit_total` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`central_total` int(3) NOT NULL,
`north_total` int(3) NOT NULL,
`ecaost_total` int(3) NOT NULL,
`south_total` int(3) NOT NULL,
`sabah_total` int(3) NOT NULL,
`sarawak_total` int(3) NOT NULL,
`timestamp` datetime NOT NULL,
`by` varchar(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8mb4;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `nagios_acknowledgements`
--

DROP TABLE IF EXISTS `nagios_acknowledgements`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `nagios_acknowledgements` (
`acknowledgement_id` int(11) NOT NULL AUTO_INCREMENT,
`instance_id` smallint(6) NOT NULL DEFAULT '0',
`entry_time` datetime NOT NULL DEFAULT '1970-01-01 00:00:01',
`entry_time_usec` int(11) NOT NULL DEFAULT '0',
`acknowledgement_type` smallint(6) NOT NULL DEFAULT '0',
`object_id` int(11) NOT NULL DEFAULT '0',
`state` smallint(6) NOT NULL DEFAULT '0',
`author_name` varchar(1024) NOT NULL DEFAULT '',
`comment_data` varchar(4096) NOT NULL DEFAULT '',
`is_sticky` smallint(6) NOT NULL DEFAULT '0',
`persistent_comment` smallint(6) NOT NULL DEFAULT '0',
`notify_contacts` smallint(6) NOT NULL DEFAULT '0',
PRIMARY KEY (`acknowledgement_id`),
UNIQUE KEY `instance_id` (`instance_id`,`entry_time`,`entry_time_usec`)
) ENGINE=MyISAM AUTO_INCREMENT=6100 DEFAULT CHARSET=utf8 COMMENT='Current and historical host and service acknowledgements';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `nagios_commands`
--

DROP TABLE IF EXISTS `nagios_commands`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `nagios_commands` (
`command_id` int(11) NOT NULL AUTO_INCREMENT,
`instance_id` smallint(6) NOT NULL DEFAULT '0',
`config_type` smallint(6) NOT NULL DEFAULT '0',
`object_id` int(11) NOT NULL DEFAULT '0',
`command_line` text NOT NULL,
PRIMARY KEY (`command_id`),
UNIQUE KEY `instance_id` (`instance_id`,`object_id`,`config_type`)
) ENGINE=MyISAM AUTO_INCREMENT=681 DEFAULT CHARSET=utf8 COMMENT='Command definitions';
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Table structure for table `nagios_commenthistory`
--

DROP TABLE IF EXISTS `nagios_commenthistory`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `nagios_commenthistory` (
`commenthistory_id` int(11) NOT NULL AUTO_INCREMENT,
`instance_id` smallint(6) NOT NULL DEFAULT '0',
`entry_time` datetime NOT NULL DEFAULT '1970-01-01 00:00:01',
`entry_time_usec` int(11) NOT NULL DEFAULT '0',
`comment_type` smallint(6) NOT NULL DEFAULT '0',
`entry_type` smallint(6) NOT NULL DEFAULT '0',
`object_id` int(11) NOT NULL DEFAULT '0',

[benjaminsmith]

Hi,

Thanks for posting that, it's not quite the full output and I'm seeing some tables that do not exist on my test system. I noticed the system is pretty large, over 40K host and service checks, and looking at the top output it's hitting some load issues.

Please follow the steps in the article below to increase the max connections for the database.
https://support.nagios.com/kb/article.php?id=513

To increase the limits on the number of processes, edit the /etc/security/limits.conf file and add the following to the bottom of the file.

Code: Select all

*          soft     nproc          262144
*          hard     nproc          262144

Save the change and reboot the server for the change to take effect.
See: https://www.thegeekdiary.com/how-to-set ... -rhel-567/

After making those changes, let me know if you notice any improvement. Also, can you send over a current profile along with the full output of the database dump. Thanks!

To Download a System Profile
To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button

[safuanmansor]

Hi benjaminsmith,

Why such exact value for the limit. Any specific reason or formula to calculate it?

Regrads,
Safuan

[benjaminsmith]

Hi,

We usually recommend increasing those on large systems so you are not hitting the limits.

If you want to review the code, it's here:

https://github.com/NagiosEnterprises/na ... 3781-L3800

After making those changes (especially the max db max connections), can you send over a fresh profile? Thanks.

[benjaminsmith]

Hi,

We usually recommend increasing those on large systems so you are not hitting the limits.

If you want to review the code, it's here:

https://github.com/NagiosEnterprises/na ... 3781-L3800

After making those changes (especially the max db max connections), can you send over a fresh profile? Thanks.

[safuanmansor]

Hi ben,

Current system is set with 50000 max connection for DB and already 200k for limit setting.
Just curious if the new limit suggested is enough as it was question by the technical board team.

Thanks,
Safuan