Delay in Nagios monitoring after ndo2db startup

[corkyman]

We have a configuration where there are two Nagios XI servers working as a primary and a backup, both sharing the same enterprise MySQL server. As instructed Nagios support we keep the nagios and ndo2db services down on the backup server and start them up only when the primary server fails. What we observe is about 5 min delay before Nagios starts running the checks. Is that normal and we just have to live with it in the failover situation? Or anything could be done to speed up the processing of checks when primary fails and the backup takes over?

[dwhitfield]

Performance is a pretty thorny issue dealing with the way MySQL specifically handles processes, general process handling, disk I/O, RAM, and processor.

That said, we do have a couple of documents that should help you:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
https://assets.nagios.com/downloads/nag ... zation.pdf

Please let us know if you'd like to explore anything in those documents in more detail.

Let's assume you've been through those documents on the backup server. So that we can do a little more digging, can you PM me your Profile on the backup server? You can download it by going to Admin > System Config > System Profile and click the Download Profile (not Show Profile) button in the top right corner. If for whatever reason you cannot download the profile, please put the output of Show Profile in the thread (that will at least get us some info).

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

[avandemore]

This thread https://support.nagios.com/forum/viewto ... 16&t=40432 and the other referenced thread therein discuss how to increase start up speed.

Essentially you need super fast disk io, but there are things you can do to mitigate this. Lots of RAM, rrdcached, Linux RAMDisk, etc.

Most likely Core is actually doing checks ASAP, and the changes just aren't reflected in in XI. Do these instances share retention.dat eg DRBD(Not Master-Master that would cause corruption)?

[corkyman]

The profile is uploaded. We do use ram disk to increase performance. Is there any way to confirm that it is just a perceived delay and that core is processing checks asap?

[avandemore]

Sure, you can view the status directly in Core. Just go to http://<Your Nagios Install>/nagios/ instead of http://<Your Nagios Install>/nagiosxi/

[avandemore]

You also have some issues in your log:

Code: Select all

mysql_pconnect(): Too many connections in

In /etc/my.cnf under the mysqld section, you can set max_connections= to some higher value. I believe it defaults to 151 if not previously set.

Code: Select all

PHP Fatal error:  Allowed memory size of 367001600 bytes exhausted

It seems your PHP size limit is 350MB, a curious size. You can edit /etc/php.ini and memory_limit to a higher value and restart apache.

[corkyman]

Thank you. We made the changes. Please close this incident.

[avandemore]

Good to hear this resolved the issue. Locking thread.