Nagios 4 Load issues

[liquidcool]

Just uploaded the load frequency that I have been talking about

[liquidcool]

hi emislivec,

I have been able to start a debug capture as the load was going up ....

The file is pretty large. A few hundred MB in size. I will compress it, but where can I send it once done ?

[Box293]

Do you have a dropbox that it can be downloaded from?

[liquidcool]

I don't have drop box, but I do have MS onedrive. Could put it up there.

If you PM me the email addresses of the people that are looking at this so I can send the share invite to them, I will get it up on there.

It actually is 1.7GB in size (uncompressed)

[liquidcool]

I downgraded the version to 4.0.2 this morning. I am still seeing the same load fluctuations. Nothing has changed.

[sreinhardt]

I've read through most of this, and it looks like no one has addressed, that we specifically patched issues like this with 4.0.8. Specifically the autorescheduler has been corrected(evening out the schedule), issues with the scheduling algorithm have worked themselves out(leveling out those peaks without autoscheduling), and lots of additional memory movement has been done away with(just overall load reduction). I want to clarify that I don't know if everything is fixed(when is it ever ), but the vast majority of things that would effect a pure core system should be resolved with an update and enabling of autorescheduling in the nagios.cfg. Had you tried anything newer than 4.0.6?

[liquidcool]

sreinhardt,

All the issue that I have raised have been while running 4.0.8. I only downgraded to 4.0.2 because one user said he did that and it all started working. It did not for me.

[liquidcool]

sreinhardt,

I have been doing more testing with this. I also had a feeling that there was something up with the auto scheduler. So what I did was disable it it for about a day. As a result the load went very high and was still doing the same fluctuations, at this higher load. Also for a few days before hand I disabled use_retained_scheduling_info (This still did not change what it was doing). Even disabling the auto_reschedule_checks and then re-enabling it a few minutes later did not nothing.

Last night I re-enabled auto_reschedule_checks. All of a sudden the load drop immensely, and stayed low since then. Also I have not seen any of those distinct fluctuations.

Looking back at when this started happening I think I noticed a trend. We started adding a lot of checks onto the server. We talking thousands. Probably taking it from about 15k-20k to about 35k. When these new checks were added and reloading / restarting the nagios service each time, the load started to increase (which we expected) but the load fluctuations came back.

So after disabling and then re-enabling auto_reschedule_checks, it seems to have reset anything it may have held about the schedules it was adjusting and started working, taking the load down to about the same as it was when we started adding the more checks.

[liquidcool]

I have another update to this.

I have built a completely new server. This time running CENTOS 6.6 with kernel version 3.14.

It is running with the exact same specs, and config's.

Strangely it is still doing the same load fluctuations, thought this time every 40 minutes (peak to peak)

I have attached a snapshot of the load graph.

[liquidcool]

Another Update.

I have run the system now with the standard kernel from CENTOS 6.6 (2.6.32-504.1.3.el6.x86_64)

And this is the load fluctuations we get now.

We can almost start calling this nagios art.

So almost a correlation between linux kernel's (config of them) and possibly the auto scheduler.