Distributed Server not updating the Central Nagios Server

[binmats]

Hi,

I have setup a distributed Nagios environment with around 500 servers and 10000 services. I have two distributed servers and one Central Nagios server with NSCA.

Everything was working fine at the starting. But after a few months the central nagios is not updating correctly when distributed updated. Some times, we are facing a time difference of 1-2 hours between the servers. Some times some server on the distributed servers also updating late by half an hour.

I am using oscp script to send to send data to central server through send_nsca plugin.
The time is same on all servers.
I am not seeing any error logs on nsca debug file also.
I was thinking, as the number of services are more, the logs are updating one by one only on central server.

I am polling the host and service on 3 minutes interval.
I tried to enable freshness check also. Again the issue is there.
Besides, I am using pnp4nagios and ndouitls for storing data on MySQL.

I have tried to change the reaper frequency and enabled the freshness check. But I am not able to fix it correctly.

If you can help me to tune the nagios with some performance tuning parameters, it would be very helpful to me.

Thanks in advance.

[mguthrie]

I'm guessing that your check latency is getting farther and farther behind on the slave servers, can you post your output from nagiosstats on all 3 servers? On a source install this is at: /usr/local/nagios/bin/nagiosstats

[crfriend]

First and foremost, heed mguthrie's advice; that'll tell you if your schedulers are "stalling".

Besides, I am using pnp4nagios and ndouitls for storing data on MySQL.

How is the performance on your MySQL server? Are you seeing lots of "slow queries"? If so, that may be stalling the schedulers on your active servers (i.e. those that actually perform the checks). Does turning off the nodutils on those systems help the situation? (Also, what version of ndoutils are you on? Some of the earlier versions can benefit mightily from adding an index or two.)