Host / Service checks very slow

[emjga]

Folks

I have about 3500 odd hosts and about 7000 Services to check.

Both Hosts and Services is very slow with the checks even though they are supposed to be checked every 10 min

i.e
Service Information
Last Updated: Thu Feb 27 15:59:04 UTC 2014
Updated every 90 seconds

Current Status: OK (for 0d 5h 57m 26s)
Status Information: Active,02/27/2014,09:03,1,XXXXXX,000.29.158.100,HGOCCY1-D-Z00102
Performance Data: RGS_State=1
Current Attempt: 1/3 (HARD state)
Last Check Time: 02-27-2014 15:03:37
Check Type: ACTIVE
Check Latency / Duration: 341.112 / 6.021 seconds
Next Scheduled Check: 02-27-2014 15:13:37
Last State Change: 02-27-2014 10:01:38

As you can see the last check time was just under 1hr ago
I am typical seeing one check every 4 - 8 hrs


I have MRTG graphs installed for nagiosstat
It is only reporting Average Service Check Latency of 4 seconds

I have looked at turning on "use_large_installation_tweaks" however it has not helped in any way.

The System load is quite high
load average: 12.35, 10.80, 10.76

4 CPU , with 16GB Ram

Does any body have any suggestions


Thanks

Matt

[lmiltchev]

This is quite a large install. What is the disk I/O? Can you run "top" and show the output?

I would recommend utilizing ram disk, if you haven't done it already. You can probably use the following post as a reference:

http://albertolarripa.com/2013/03/03/nagios-tuning/

We have an official document on how to utilize ram disk, but it is for Nagios XI. I will link it below anyway, in case you want to review this one as well:

http://assets.nagios.com/downloads/nagi ... giosXI.pdf

[emjga]

lmiltchev Thanks for your reply.

On further checking with nagios in Debug mode
I noticed that pnp4nagios (Performance graphs) was realy slowing nagios down.

On disabling process_performance_data=0 , the checks went back to every 12 min on average
I need to spend some time looking at getting pnp4nagios to run faster.
A very qucik check on there website sugests unsing "Bulk Mode with NPCD"

At the moment it looks like pnp4nagios is configured for Synchronous mode

Matt

[abrist]

At the moment it looks like pnp4nagios is configured for Synchronous mode

This setting will really slow down the server as it treats perfdata as a blocking process. I would suggest implementing bulk or bulk with npcd.