Nagios 2014R2.3 on VM HIGH LOAD SPIKES

[carlos.atos]

Hello,

@Abrist :
this is the last TOP from today (load of 52) working with only 1 vCPU, here you can see that there are a lot of httpd processes.

TOP 02-02-15.PNG

@lgroschen & WillemDH :
I had updated the VM to the maximun vCPUs available (4) and the ram is still on 4Gb. I had shut down the other VMs in this host, so les say that Nagios has the full hardware, lets see how does it work.

I will give it 24 hours to see what happen

Cheers,

[WillemDH]

Looking at your top, imo it's not the httpd processes, but the many php processes. When I compare this with my top on XI server with 600 hosts, I also see httpd processes, but I do not se any php process. i'm curious what Nagios support has to say about that. are you running any custom php checks or scripts?

[scottwilkerson]

I would have to agree with WillemDH

Can we see which of the php processes are running (these are usually the crons on Nagios XI)

Code: Select all

ps -ef|grep php

[carlos.atos]

Hello Guys,

As supposed with 4 vCPUs , the performance of the VM is superb, but of corse it is 4 times faster, but I doubt it will have improved the processes running on here, the higher peaks are reaching 2( in load scale).If I consider that I had increased the vCPUs by 4, so this load could be roughly equivalent to 8, am I wrong? .. but in other case it works, I will try to downgrade it to two vCPUs.

this is the load graph since I stopped the 1vCPUs VM, and upgraded up to 4 vCPUs

localhost-current_load Feb03.jpg

@ scottwilkerson :

this is the outcome form the ps- ef|grep php

Code: Select all

 [root@localhost ~]# ps -ef|grep php
nagios   10874 10866  0 10:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   10878 10869  0 10:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios   10880 10870  0 10:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios   10881 10868  0 10:05 ?        00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios   10884 10874  0 10:05 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios   10886 10878  0 10:05 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios   10887 10881  1 10:05 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios   10891 10880  1 10:05 ?        00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
root     11109 22240  0 10:05 pts/0    00:00:00 grep php

@WillemDH

I'm not running any custom php script, only the provided by the nagios wizards.

So what do you think?

Cheers,

[WillemDH]

As supposed with 4 vCPUs , the performance of the VM is superb, but of corse it is 4 times faster, but I doubt it will have improved the processes running on here, the higher peaks are reaching 2( in load scale).If I consider that I had increased the vCPUs by 4, so this load could be roughly equivalent to 8, am I wrong? .. but in other case it works, I will try to downgrade it to two vCPUs.

Well I think you might be wrong. As far as i know load is not divided by the number of cores in this case, so with a load with peaks of 2, it seems like 2 cpu's would be enough. If you can spare the extra cpu's, I would suggest keeping 4, taking into account your Nagios setup would only grow over time?

Although when I read this:

hello I have configured 8 host for ping services and 2 SNMP routers for network traffic.

I would think a load of 2 is still relatively high. But let's see what Nagios support thinks about that..

Grtz

Willem

[scottwilkerson]

Largely this looks much better, I'd like to see where the load levels out, and also if you are experiencing any disk I/O problems.

In virtual environments, the Load can often go up even though you have more than enough CPU's because the system is waiting it's turn to read/write to disk.

Your XI system could be doing just fine but if another VM is monopolizing the disk you will still have higher than normal load.

[carlos.atos]

Hi Guys,

Yesterday I made a copy of this same VM but with 2vCPUs running in the Same Exsi Host. ( only these two VMs are running in the Host) and then i left them running the whole night,

the performance of the 2vCPUs VM is slightly higher than the one with 4vCPUS , as WillemDH said 2 vCPUS would be enough.

Juut at the moment I was going to send you and update, I saw a very odd thing, the load on the 4vCPUs VM started to increase with an spike up to 20 :s

Load 4vCPUs FEB-04.PNG

Load 2vCPUs FEB-04.PNG

I'm not sure what happening here, but I think maybe I could have an I/O disks issues, thus the server is quite old and this happened having the second VM (with 2vCPUs working) maybe there are some processes running at the same time .Could be this correlated due the VMs are monitoring the same Machines?, no idea really

Lets see what happens until tomorrow.

cheers

[WillemDH]

You could check in vmware if the load on other vm's on the same ESX server have similar load spikes. Next time you should also immediately do a top to see what procs are consuming cpu resources. It could also be a Nagios scheduled backup or a snapshot of the machine (maybe taken by external backup software?). I see similar load spikes when taking a snapshot.

[carlos.atos]

Hello WillemDH, I did the TOP Captures at the moment, but I forgot to upload them, sorry

here they are:

TOP Nagios 4vCPU feb-04-2.PNG

TOP Nagios 2vCPU feb-04-2.PNG

and checking the load on the ESXi Host , the host and the VMs present the same CPU performance and load Spikes.

and about Nagios snapshotsI haven't configured none of that, they are by default.
there is a way to check the ocurrence of the snapshot/backup scheduled jobs?

Cheers

[scottwilkerson]

You had run this before, but I think after the system was running normally, being php seems to be consuming much of the CPU, when it is overloaded again, I would run the following and see if you have any duplicate crons running

Code: Select all

ps -ef|grep php|grep -v /bin/sh