Page 1 of 1
CPU-usage gets higher and higher
Posted: Mon Mar 30, 2020 8:59 am
by noweda
Hello support,
our NagiosXI server seems to have a problem with CPU usage. After a restart of the server, the usage get's higher and higher. A chart out of VMware for our NagiosXI VM is attached:
CPU-usage_vm_last_month.png
What can we do?
Nagios XI - System Info
System
Nagios XI version: 5.6.12
Release info: ux010162.noweda.de 2.6.32-754.27.1.el6.x86_64 x86_64
CentOS release 6.10 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
Server Name: ux010162.noweda.de
Server Address: 188.1.40.162
Server Port: 443
Best regards
Christoph
Re: CPU-usage gets higher and higher
Posted: Mon Mar 30, 2020 2:41 pm
by jdunitz
You can try to diagnose this yourself, or you can send us a system profile and we can take a look.
To start with, when the CPU utilization is high, run the "top" command and take note of which processes are taking the most CPU.
That will be a good clue for where the problem lies.
Also, looking at the error and other logs:
/var/log/messages
/var/log/httpd/access_log
/var/log/httpd/error_log
Those may give you some information about what is going on with your system, as well as the logs in /usr/local/nagios/var and /usr/local/nagiosxi/var.
If you'd rather have us look at your logs and configuration, please generate a system profile, PM to us.
Included is a screenshot of where to go to generate the profile.
Thanks!
Re: CPU-usage gets higher and higher
Posted: Wed Apr 01, 2020 11:34 am
by jdunitz
Looking at top from your system, I can see that there are a number of instances of a backup process I don't recognize.
Code: Select all
top - 16:21:19 up 12 days, 19:48, 2 users, load average: 4.55, 5.05, 5.44
Tasks: 428 total, 5 running, 423 sleeping, 0 stopped, 0 zombie
Cpu(s): 35.3%us, 34.0%sy, 0.0%ni, 30.4%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 24723620k total, 23626580k used, 1097040k free, 325988k buffers
Swap: 2064380k total, 79124k used, 1985256k free, 16850200k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11859 mysql 20 0 6309m 172m 6900 S 58.4 0.7 3836:44 mysqld
62769 apache 20 0 488m 66m 4960 S 15.6 0.3 10:15.81 httpd
98436 apache 20 0 483m 60m 6016 R 11.7 0.3 7:11.17 httpd
101897 apache 20 0 483m 62m 4864 R 11.7 0.3 8:04.92 httpd
8420 nagios 20 0 103m 1424 1236 S 7.8 0.0 890:32.71 backup_nagios.s
118320 nagios 20 0 103m 1380 1236 S 7.8 0.0 1439:47 backup_nagios.s
9545 nagios 20 0 103m 1388 1236 S 5.8 0.0 749:48.36 backup_nagios.s
40323 nagios 20 0 56460 8028 1036 S 5.8 0.0 26:50.11 ndo2db
48458 nagios 20 0 103m 1448 1236 S 5.8 0.0 622:38.58 backup_nagios.s
66345 nagios 20 0 103m 1444 1236 S 5.8 0.0 75:14.46 backup_nagios.s
67331 apache 20 0 484m 62m 6068 S 5.8 0.3 8:08.67 httpd
91164 nagios 20 0 103m 1440 1236 S 5.8 0.0 509:47.56 backup_nagios.s
94681 nagios 20 0 103m 1376 1236 S 5.8 0.0 1025:17 backup_nagios.s
95245 nagios 20 0 103m 1444 1236 S 5.8 0.0 174:57.72 backup_nagios.s
891 nagios 20 0 103m 1376 1236 S 3.9 0.0 1570:21 backup_nagios.s
Is that something of your own creation, or a third-party script you downloaded somewhere?
Whatever it is, it seems to be chewing up quite a lot of CPU.
Also, your nagios log file isn't readable, or is missing.
What do you get if you do a "ls -la /usr/local/nagios/var/nagios.log"?
You may have other problems also, but let's start there.
Thanks!
Re: CPU-usage gets higher and higher
Posted: Mon Apr 06, 2020 1:08 am
by noweda
Hi,
thanks for your reply! Here's the output for nagios.log:
root@ux010162:~# ls -la /usr/local/nagios/var/nagios.log
-rw-r--r-- 1 nagios nagios 2561130 Apr 6 08:01 /usr/local/nagios/var/nagios.log
The script seems to be a custom script generated by one of our oracle admins - I didn't know it until now... I will check that.
Best regards
Christoph
Re: CPU-usage gets higher and higher
Posted: Mon Apr 06, 2020 3:05 pm
by ssax
It looks like they are not finishing properly and probably causing that as @jdunitz found, let us know what your team says.
Thank you
Re: CPU-usage gets higher and higher
Posted: Wed Apr 08, 2020 7:54 am
by noweda
Hi,
our oracle admin stopped the custom script and it's lower the usage.
This issue is resolved now.
Thanks for your support.
Sincerely,
Re: CPU-usage gets higher and higher
Posted: Wed Apr 08, 2020 8:04 am
by scottwilkerson
noweda wrote:Hi,
our oracle admin stopped the custom script and it's lower the usage.
This issue is resolved now.
Thanks for your support.
Sincerely,
Great!
Locking thread