High check latency after migration

[cornea]

I migrated nagios 3.3.1 from one server to another, but I got some issues about latency.
Below is the output. Is there anyone can help us to resolve?

# nagiostats
[/color]
Nagios Stats 3.3.1
Copyright (c) 2003-2008 Ethan Galstad (http://www.nagios.org)
Last Modified: 07-25-2011
License: GPL

CURRENT STATUS DATA
------------------------------------------------------
Status File: /usr/local/nagios/var/status.dat
Status File Age: 0d 0h 0m 19s
Status File Version: 3.3.1

Program Running Time: 0d 1h 9m 42s
Nagios PID: 15146
Used/High/Total Command Buffers: 0 / 0 / 4096

Total Services: 3502
Services Checked: 3502
Services Scheduled: 3502
Services Actively Checked: 3502
Services Passively Checked: 0
Total Service State Change: 0.000 / 28.680 / 0.104 %
Active Service Latency: 285.395 / 346.288 / 290.122 sec
Active Service Execution Time: 0.045 / 16.721 / 0.440 sec
Active Service State Change: 0.000 / 28.680 / 0.104 %
Active Services Last 1/5/15/60 min: 204 / 1668 / 3483 / 3502
Passive Service Latency: 0.000 / 0.000 / 0.000 sec
Passive Service State Change: 0.000 / 0.000 / 0.000 %
Passive Services Last 1/5/15/60 min: 0 / 0 / 0 / 0
Services Ok/Warn/Unk/Crit: 3495 / 5 / 0 / 2
Services Flapping: 0
Services In Downtime: 0

Total Hosts: 834
Hosts Checked: 834
Hosts Scheduled: 834
Hosts Actively Checked: 834
Host Passively Checked: 0
Total Host State Change: 0.000 / 19.210 / 0.331 %
Active Host Latency: 284.618 / 295.795 / 289.067 sec
Active Host Execution Time: 0.044 / 10.030 / 0.074 sec
Active Host State Change: 0.000 / 19.210 / 0.331 %
Active Hosts Last 1/5/15/60 min: 79 / 340 / 834 / 834
Passive Host Latency: 0.000 / 0.000 / 0.000 sec
Passive Host State Change: 0.000 / 0.000 / 0.000 %
Passive Hosts Last 1/5/15/60 min: 0 / 0 / 0 / 0
Hosts Up/Down/Unreach: 833 / 1 / 0
Hosts Flapping: 0
Hosts In Downtime: 0

Active Host Checks Last 1/5/15 min: 115 / 400 / 1266
Scheduled: 113 / 396 / 1253
On-demand: 2 / 4 / 13
Parallel: 113 / 396 / 1258
Serial: 0 / 0 / 0
Cached: 2 / 4 / 9
Passive Host Checks Last 1/5/15 min: 0 / 0 / 0
Active Service Checks Last 1/5/15 min: 350 / 1803 / 5324
Scheduled: 350 / 1803 / 5324
On-demand: 0 / 0 / 0
Cached: 0 / 0 / 0
Passive Service Checks Last 1/5/15 min: 0 / 0 / 0

External Commands Last 1/5/15 min: 0 / 0 / 0

#/usr/local/nagios/bin/nagios -s ../etc/nagios.cfg

Nagios Core 3.3.1
Copyright (c) 2009-2011 Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 07-25-2011
License: GPL

Website: http://www.nagios.org
Timing information on object configuration processing is listed
below. You can use this information to see if precaching your
object configuration would be useful.

Object Config Source: Config files (uncached)

OBJECT CONFIG PROCESSING TIMES (* = Potential for precache savings with -u option)
----------------------------------
Read: 0.081451 sec
Resolve: 0.012336 sec *
Recomb Contactgroups: 0.000877 sec *
Recomb Hostgroups: 0.036682 sec *
Dup Services: 0.018501 sec *
Recomb Servicegroups: 0.000140 sec *
Duplicate: 0.000006 sec *
Inherit: 0.004831 sec *
Recomb Contacts: 0.000003 sec *
Sort: 0.000002 sec *
Register: 0.050974 sec
Free: 0.004534 sec
============
TOTAL: 0.210339 sec * = 0.073380 sec (34.89%) estimated savings


RETENTION DATA TIMES
----------------------------------
Read and Process: 0.374611 sec
============
TOTAL: 0.374611 sec


Timing information on configuration verification is listed below.

CONFIG VERIFICATION TIMES (* = Potential for speedup with -x option)
----------------------------------
Object Relationships: 0.063920 sec
Circular Paths: 0.000078 sec *
Misc: 0.003401 sec
============
TOTAL: 0.067399 sec * = 0.000078 sec (0.1%) estimated savings


EVENT SCHEDULING TIMES
-------------------------------------
Get service info: 0.040919 sec
Get host info info: 0.009469 sec
Get service params: 0.000037 sec
Schedule service times: 0.133915 sec
Schedule service events: 0.057062 sec
Get host params: 0.000003 sec
Schedule host times: 0.031260 sec
Schedule host events: 0.041126 sec
============
TOTAL: 0.313791 sec


Projected scheduling information for host and service checks
is listed below. This information assumes that you are going
to start running Nagios with your current config files.

HOST SCHEDULING INFORMATION
---------------------------
Total hosts: 834
Total scheduled hosts: 834
Host inter-check delay method: SMART
Average host check interval: 300.00 sec
Host inter-check delay: 0.36 sec
Max host check spread: 30 min
First scheduled check: Fri Mar 29 16:11:42 2013
Last scheduled check: Fri Mar 29 16:16:41 2013


SERVICE SCHEDULING INFORMATION
-------------------------------
Total services: 3502
Total scheduled services: 3502
Service inter-check delay method: SMART
Average service check interval: 312.08 sec
Inter-check delay: 0.09 sec
Interleave factor method: SMART
Average services per host: 4.20
Service interleave factor: 5
Max service check spread: 5 min
First scheduled check: Fri Mar 29 16:12:42 2013
Last scheduled check: Fri Mar 29 16:17:41 2013


CHECK PROCESSING INFORMATION
----------------------------
Check result reaper interval: 5 sec
Max concurrent service checks: 300


PERFORMANCE SUGGESTIONS
-----------------------
I have no suggestions - things look okay.

[slansing]

Are you using MK_livestatus, or offloading checks to DNX/Mod Gearman? How does the server's load look when this latency issue occurs? Can you also post a copy of your nagios.cfg file?

Also, be sure that the Nagios server, and the apache portion's times are synced, otherwise you can see high latency.

[cornea]

I do not use MK_livestatus.
On the other server, I use the same configuration, and the latency is low.

What does "Nagios server, and the apache portion's times are synced" mean? They are on the same server.

Are you using MK_livestatus, or offloading checks to DNX/Mod Gearman? How does the server's load look when this latency issue occurs? Can you also post a copy of your nagios.cfg file?

Also, be sure that the Nagios server, and the apache portion's times are synced, otherwise you can see high latency.

[abrist]

Check the timezone settings in php.ini:

Code: Select all

cat /etc/php.ini | grep timezone

Does it match the system timezone?

Code: Select all

cat /etc/sysconfig/clock

Is the date/time right?

Code: Select all

date

Do you use ntp?

[cornea]

I do not use NTP.
I check timezone and date/time. They are correct.

[slansing]

Can you send us a screenshot of the following page's:

Admin > System Status

and

Admin > Monitoring Engine Status

As well as a copy of your system profile found at:

Admin > System Profile

Thank you!