Nagios Support Forum

Hi,

I have a nagios server installed monitoring aroung 90 hosts and around 500 services.
The CPU utilization always comes around 90%, and most of the CPU load is taken by nagios processes:

See the output below:

[root@msmontest01 ~]# ps -eo pcpu,pid,user,args | sort -k 1 -r | head -25
%CPU PID USER COMMAND
9.0 25194 nagios [nagios]
8.4 30282 nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
8.0 25189 nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
7.0 25192 nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
5.0 25195 nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
5.0 25187 nagios /usr/local/nagios/libexec/check_nt -H ILREVAPP3.JDADELIVERS.COM -p 1248 -s NScl1kd8sz87 -v USEDDISKSPACE -l c -w 80 -c 90
3.5 25186 nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
2.3 25085 nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
1.7 25082 nagios /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
1.6 25086 nagios /usr/local/nagios/libexec/check_ping -H ILSRGAPP1.JDADELIVERS.COM -w 200,80% -c 300,90% -p 5
1.5 25061 nagios [nagios]
1.2 25083 nagios /usr/local/nagios/libexec/check_ping -H ILREVWEB6.JDADELIVERS.COM -w 200,80% -c 300,90% -p 5
1.0 25193 nagios [nagios]
1.0 25190 nagios [nagios]
0.1 24389 root /usr/bin/perl /usr/local/JDA/bin/icmp_monitor_mod
0.0 993 ssadm -bash
0.0 989 ssadm sshd: ssadm@pts/0
0.0 985 root sshd: ssadm [priv]
0.0 8248 apache /usr/sbin/httpd
0.0 615 root /sbin/udevd -d
0.0 5 root [khelper]
0.0 582 root [kauditd]
0.0 556 root [kjournald]
0.0 539 root [ksnapd]

Here, I can see multiple nagios daemons running. Only the main nagios daemon runs permenantly and the other daemons only stays for few seconds.
Is this normal? And why the Utilization gets so high? When I restart nagios, the utilization becomes normal, but it builds up gradually, and reaches 90% after around 24 hours.
My Cpu info:

[root@msmontest01 ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X7560 @ 2.27GHz
stepping : 4
cpu MHz : 2394.000
cache size : 24576 KB
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc up ida nonstop_tsc pni cx16 popcnt lahf_lm
bogomips : 4788.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: [8]

Thanks in advance.

Regards,
Ajmal

Hello,
Can you provide the output of dstat during a peak and quiet cpu usage period?

The dstat output while 100% utilization.

[root@msmontest01 ~]# dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 23 74 1 0 0|4060B 239k| 0 0 | 164B 1017B| 482 287
3 97 0 0 0 0| 0 512k| 11k 11k| 0 0 |1186 678
13 87 0 0 0 0| 16k 0 | 15k 6666B| 0 0 |1109 469
1 99 0 0 0 0| 0 0 |6612B 1315B| 0 0 |1024 78
1 99 0 0 0 0| 0 0 | 696B 626B| 0 0 |1014 77
3 97 0 0 0 0|4096B 0 |6486B 6047B| 0 0 |1101 449
2 98 0 0 0 0| 0 592k|2993B 3113B| 0 0 |1069 199
7 91 0 0 1 1| 64k 1296k|9688B 10k| 0 0 |1274 958
4 96 0 0 0 0|4096B 1388k|5801B 4969B| 0 0 |1147 347
1 99 0 0 0 0| 0 0 | 430B 637B| 0 0 |1011 79
1 99 0 0 0 0| 0 0 | 424B 449B| 0 0 |1007 66
0 100 0 0 0 0| 0 0 | 473B 429B| 0 0 |1009 69
1 99 0 0 0 0| 0 32k| 488B 533B| 0 0 |1010 70
1 99 0 0 0 0| 0 436k| 339B 517B| 0 0 |1019 85
0 100 0 0 0 0| 0 0 | 60B 420B| 0 0 |1004 70
1 99 0 0 0 0| 0 0 | 300B 449B| 0 0 |1008 75


While normal ytilization after restarting:
[root@msmontest01 ~]# dstat
You did not select any stats, using -cdngy by default.
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 23 74 1 0 0|4112B 239k| 0 0 | 207B 1017B| 482 287
5 3 90 0 1 1| 0 0 |3271B 3578B| 0 0 |1043 187
2 1 97 0 0 0| 0 0 |4420B 3818B| 0 0 |1057 189
1 2 97 0 0 0| 0 0 |3635B 3134B| 0 0 |1049 200
1 2 97 0 0 0| 0 8192B|2748B 2634B| 0 0 |1049 182
2 2 94 2 0 0| 0 760k|5033B 4109B| 0 0 |1097 263
1 1 98 0 0 0| 0 0 |2716B 2499B| 0 0 |1038 162
2 1 97 0 0 0| 0 0 |3821B 3538B| 0 0 |1058 191
1 2 97 0 0 0| 0 0 |3167B 2721B| 0 0 |1049 174
2 1 96 0 0 1| 0 0 |3065B 2718B| 0 0 |1050 169
6 10 74 10 0 0| 0 2448k|2331B 2249B| 0 0 |1162 391
2 1 97 0 0 0| 0 0 |3548B 3091B| 0 0 |1045 194
3 2 95 0 0 0| 0 0 |5631B 4785B| 0 0 |1067 234
2 3 95 0 0 0| 0 0 |6908B 5634B| 0 0 |1075 276
3 2 95 0 0 0| 0 72k|5921B 5426B| 0 0 |1088 243
2 3 94 0 0 1| 0 0 |6044B 5540B| 0 0 |1081 241
1 2 95 2 0 0| 0 660k|4188B 4002B| 0 0 |1100 232
You have new mail in /var/spool/mail/root

It is normal for multiple nagios processes to be running. The parent process will spawn children processes to run checks. As long as there are not multiple parent processes this is normal.

What type of checks are your running? some checks are very CPU intensive and some are not.

Also, what is your load average in top?

-Yancy