Page 3 of 6
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Mon Feb 02, 2015 6:19 am
by carlos.atos
Hello,
@Abrist :
this is the last TOP from today (load of 52) working with only 1 vCPU, here you can see that there are a lot of httpd processes.
TOP 02-02-15.PNG
@lgroschen & WillemDH :
I had updated the VM to the maximun vCPUs available (4) and the ram is still on 4Gb. I had shut down the other VMs in this host, so les say that Nagios has the full hardware, lets see how does it work.
I will give it 24 hours to see what happen
Cheers,
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Mon Feb 02, 2015 7:02 am
by WillemDH
Looking at your top, imo it's not the httpd processes, but the many php processes. When I compare this with my top on XI server with 600 hosts, I also see httpd processes, but I do not se any php process. i'm curious what Nagios support has to say about that. are you running any custom php checks or scripts?
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Mon Feb 02, 2015 9:17 am
by scottwilkerson
I would have to agree with WillemDH
Can we see which of the php processes are running (these are usually the crons on Nagios XI)
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Tue Feb 03, 2015 5:18 am
by carlos.atos
Hello Guys,
As supposed with 4 vCPUs , the performance of the VM is superb, but of corse it is 4 times faster, but I doubt it will have improved the processes running on here, the higher peaks are reaching 2( in load scale).If I consider that I had increased the vCPUs by 4, so this load could be roughly equivalent to 8, am I wrong? .. but in other case it works, I will try to downgrade it to two vCPUs.
this is the load graph since I stopped the 1vCPUs VM, and upgraded up to 4 vCPUs
localhost-current_load Feb03.jpg
@ scottwilkerson :
this is the outcome form the ps- ef|grep php
Code: Select all
[root@localhost ~]# ps -ef|grep php
nagios 10874 10866 0 10:05 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 10878 10869 0 10:05 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 10880 10870 0 10:05 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 10881 10868 0 10:05 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 10884 10874 0 10:05 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 10886 10878 0 10:05 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 10887 10881 1 10:05 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios 10891 10880 1 10:05 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
root 11109 22240 0 10:05 pts/0 00:00:00 grep php
@WillemDH
I'm not running any custom php script, only the provided by the nagios wizards.
So what do you think?
Cheers,
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Tue Feb 03, 2015 12:58 pm
by WillemDH
As supposed with 4 vCPUs , the performance of the VM is superb, but of corse it is 4 times faster, but I doubt it will have improved the processes running on here, the higher peaks are reaching 2( in load scale).If I consider that I had increased the vCPUs by 4, so this load could be roughly equivalent to 8, am I wrong? .. but in other case it works, I will try to downgrade it to two vCPUs.
Well I think you might be wrong. As far as i know load is not divided by the number of cores in this case, so with a load with peaks of 2, it seems like 2 cpu's would be enough. If you can spare the extra cpu's, I would suggest keeping 4, taking into account your Nagios setup would only grow over time?
Although when I read this:
hello I have configured 8 host for ping services and 2 SNMP routers for network traffic.
I would think a load of 2 is still relatively high. But let's see what Nagios support thinks about that..
Grtz
Willem
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Tue Feb 03, 2015 2:00 pm
by scottwilkerson
Largely this looks much better, I'd like to see where the load levels out, and also if you are experiencing any disk I/O problems.
In virtual environments, the Load can often go up even though you have more than enough CPU's because the system is waiting it's turn to read/write to disk.
Your XI system could be doing just fine but if another VM is monopolizing the disk you will still have higher than normal load.
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Wed Feb 04, 2015 12:50 pm
by carlos.atos
Hi Guys,
Yesterday I made a copy of this same VM but with 2vCPUs running in the Same Exsi Host. ( only these two VMs are running in the Host) and then i left them running the whole night,
the performance of the 2vCPUs VM is slightly higher than the one with 4vCPUS , as WillemDH said 2 vCPUS would be enough.
Juut at the moment I was going to send you and update, I saw a very odd thing, the load on the 4vCPUs VM started to increase with an spike up to 20 :s
Load 4vCPUs FEB-04.PNG
Load 2vCPUs FEB-04.PNG
I'm not sure what happening here, but I think maybe I could have an I/O disks issues, thus the server is quite old and this happened having the second VM (with 2vCPUs working) maybe there are some processes running at the same time .Could be this correlated due the VMs are monitoring the same Machines?, no idea really
Lets see what happens until tomorrow.
cheers
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Wed Feb 04, 2015 4:06 pm
by WillemDH
You could check in vmware if the load on other vm's on the same ESX server have similar load spikes. Next time you should also immediately do a top to see what procs are consuming cpu resources. It could also be a Nagios scheduled backup or a snapshot of the machine (maybe taken by external backup software?). I see similar load spikes when taking a snapshot.
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Thu Feb 05, 2015 6:37 am
by carlos.atos
Hello WillemDH, I did the TOP Captures at the moment, but I forgot to upload them, sorry
here they are:
TOP Nagios 4vCPU feb-04-2.PNG
TOP Nagios 2vCPU feb-04-2.PNG
and checking the load on the ESXi Host , the host and the VMs present the same CPU performance and load Spikes.
and about Nagios snapshotsI haven't configured none of that, they are by default.
there is a way to check the ocurrence of the snapshot/backup scheduled jobs?
Cheers
Re: Nagios 2014R2.3 on VM HIGH LOAD SPIKES
Posted: Thu Feb 05, 2015 9:15 am
by scottwilkerson
You had run this before, but I think after the system was running normally, being php seems to be consuming much of the CPU, when it is overloaded again, I would run the following and see if you have any duplicate crons running