Search found 14 matches

by dnelson
Tue Feb 18, 2014 9:41 am
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Solution - Disabled embedded perl

On the advice of an individual that was experiencing the same issues as I was, I disabled the embedded Perl interpreter (set 'enable_embedded_perl=0' in nagios.cfg). Restarted Nagios and waited. It's now been 5+ days since doing this and service and host latencies are stable in the 200 and 400 mS ra...
by dnelson
Wed Jan 08, 2014 9:58 am
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Re: Latencies increase drastically after 33 hourse of uptime

Hi abrist, Thanks for the link. The link you provided appears to be for Nagios 4.x and differs slightly from Nagios 3.x. I'm including both here to avoid any confusion. Nagios 3.x large installation tweaks: http://nagios.sourceforge.net/docs/3_0/largeinstalltweaks.html Nagios 4.x large installation ...
by dnelson
Tue Jan 07, 2014 3:34 pm
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Use the source, Luke.

Before submitting a case w/ the devs regarding the use_large_installation_tweaks and child_processes_fork_twice tunables, I needed to verify that the problem existed in 3.5.1 (I had been using 3.3.1). Sure enough, I can reproduce the problem with 3.5.1. Next, I went reading in the source code to see...
by dnelson
Mon Jan 06, 2014 3:52 pm
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

use_large_installation_tweaks and child_processes_fork_twice

Reading up on what large_installation_tweaks does, ( http://nagios.sourceforge.net/docs/3_0/largeinstalltweaks.html ), I interpreted that if set, child_processes_fork_twice would be automatically set to 0. This seemed to be reinforced in the docs for "child_process_fork_twice" as it stated...
by dnelson
Mon Jan 06, 2014 1:54 pm
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Small Nagios Config

Over the new year's break, I configured a simple Nagios setup consisting of ~650 host checks each with a simple nrpe service check. Without the need for SNMP checks, I also set 'enable_environment_macros=0' since that was brought up in an earlier post I warranted some investigation. After ~38.5 hour...
by dnelson
Thu Jan 02, 2014 1:38 pm
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Re: Latencies increase drastically after 33 hours of uptime

I have a few different core systems running 3.5.1 and I have not noticed this behavior. The checks/5min are low on those servers though (around a 1000). This behavior in the past has been caused by: 1) Latency/lack of resources (ram/disk io/load) 2) System ulimits (open files is usually the culprit...
by dnelson
Mon Dec 30, 2013 9:54 am
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Latencies climb with a generic OS build

It turns out that a fresh generic OS build behaves the same as one w/ Gluster and Linux HA (heartbeat) installed. After ~33 hours of daemon uptime, both service and host latencies increase at a rate of approx 160 seconds / day. Also, kernel time behaves similarly in that it gradually rises and when ...
by dnelson
Fri Dec 27, 2013 4:45 pm
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Re: Latencies increase drastically after 33 hours of uptime

The shutdown and rebuild of the 2nd node looks to have had an interesting effect on the service latencies of the 1st node. The service latency times due to the daemon restart at 9:42 AM yesterday is not surprising. However, when I shutdown the 2nd node at 13:39 PM yesterday to begin rebuilding the O...
by dnelson
Fri Dec 27, 2013 3:27 pm
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Re: Latencies increase drastically after 33 hourse of uptime

Do you know if this kernel was a release candidate, or a full release? Which one are you running - 2.6.18 or 2.6.32? Was the kernel upgraded or any other major changes done before you started experiencing the issue? Hi abrist, AFAIK, these kernels are the full release. They are the ones released th...
by dnelson
Thu Dec 26, 2013 4:48 pm
Forum: Open Source Nagios Projects
Topic: Latencies increase drastically after 33 hourse of uptime
Replies: 22
Views: 6426

Re: Latencies increase drastically after 33 hourse of uptime

Hi slansing, Very interesting data sets, I hope your above proposal is not going to be too much work, are you running anything else on the same kernel configuration as this nagios server? You had mentioned HA, is there a backup server you could bring online to see if it has the same quirky latency? ...