Page 1 of 1
Delay in Nagios monitoring after ndo2db startup
Posted: Thu Dec 08, 2016 8:41 am
by corkyman
We have a configuration where there are two Nagios XI servers working as a primary and a backup, both sharing the same enterprise MySQL server. As instructed Nagios support we keep the nagios and ndo2db services down on the backup server and start them up only when the primary server fails. What we observe is about 5 min delay before Nagios starts running the checks. Is that normal and we just have to live with it in the failover situation? Or anything could be done to speed up the processing of checks when primary fails and the backup takes over?
Re: Delay in Nagios monitoring after ndo2db startup
Posted: Thu Dec 08, 2016 12:20 pm
by dwhitfield
Performance is a pretty thorny issue dealing with the way MySQL specifically handles processes, general process handling, disk I/O, RAM, and processor.
That said, we do have a couple of documents that should help you:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf
https://assets.nagios.com/downloads/nag ... zation.pdf
Please let us know if you'd like to explore anything in those documents in more detail.
Let's assume you've been through those documents on the backup server. So that we can do a little more digging, can you PM me your Profile on the backup server? You can download it by going to Admin > System Config > System Profile and click the Download Profile (not Show Profile) button in the top right corner. If for whatever reason you cannot download the profile, please put the output of Show Profile in the thread (that will at least get us some info).
After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.
Re: Delay in Nagios monitoring after ndo2db startup
Posted: Thu Dec 08, 2016 12:22 pm
by avandemore
This thread
https://support.nagios.com/forum/viewto ... 16&t=40432 and the other referenced thread therein discuss how to increase start up speed.
Essentially you need super fast disk io, but there are things you can do to mitigate this. Lots of RAM, rrdcached, Linux RAMDisk, etc.
Most likely Core is actually doing checks ASAP, and the changes just aren't reflected in in XI. Do these instances share retention.dat eg DRBD(Not Master-Master that would cause corruption)?
Re: Delay in Nagios monitoring after ndo2db startup
Posted: Wed Dec 28, 2016 2:32 pm
by corkyman
The profile is uploaded. We do use ram disk to increase performance. Is there any way to confirm that it is just a perceived delay and that core is processing checks asap?
Re: Delay in Nagios monitoring after ndo2db startup
Posted: Wed Dec 28, 2016 2:37 pm
by avandemore
Sure, you can view the status directly in Core. Just go to http://<Your Nagios Install>/nagios/ instead of http://<Your Nagios Install>/nagiosxi/
Re: Delay in Nagios monitoring after ndo2db startup
Posted: Wed Dec 28, 2016 2:48 pm
by avandemore
You also have some issues in your log:
Code: Select all
mysql_pconnect(): Too many connections in
In
/etc/my.cnf under the mysqld section, you can set
max_connections= to some higher value. I believe it defaults to 151 if not previously set.
Code: Select all
PHP Fatal error: Allowed memory size of 367001600 bytes exhausted
It seems your PHP size limit is 350MB, a curious size. You can edit
/etc/php.ini and
memory_limit to a higher value and restart apache.
Re: Delay in Nagios monitoring after ndo2db startup
Posted: Thu Jan 05, 2017 12:35 pm
by corkyman
Thank you. We made the changes. Please close this incident.
Re: Delay in Nagios monitoring after ndo2db startup
Posted: Thu Jan 05, 2017 12:42 pm
by avandemore
Good to hear this resolved the issue. Locking thread.