CPU-usage gets higher and higher

[noweda]

Hello support,

our NagiosXI server seems to have a problem with CPU usage. After a restart of the server, the usage get's higher and higher. A chart out of VMware for our NagiosXI VM is attached:

CPU-usage_vm_last_month.png

What can we do?

Nagios XI - System Info
System
Nagios XI version: 5.6.12
Release info: ux010162.noweda.de 2.6.32-754.27.1.el6.x86_64 x86_64
CentOS release 6.10 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
Server Name: ux010162.noweda.de
Server Address: 188.1.40.162
Server Port: 443

Best regards
Christoph

[jdunitz]

You can try to diagnose this yourself, or you can send us a system profile and we can take a look.

To start with, when the CPU utilization is high, run the "top" command and take note of which processes are taking the most CPU.
That will be a good clue for where the problem lies.

Also, looking at the error and other logs:
/var/log/messages
/var/log/httpd/access_log
/var/log/httpd/error_log

Those may give you some information about what is going on with your system, as well as the logs in /usr/local/nagios/var and /usr/local/nagiosxi/var.

If you'd rather have us look at your logs and configuration, please generate a system profile, PM to us.
Included is a screenshot of where to go to generate the profile.

Thanks!

[jdunitz]

Looking at top from your system, I can see that there are a number of instances of a backup process I don't recognize.

Code: Select all

top - 16:21:19 up 12 days, 19:48,  2 users,  load average: 4.55, 5.05, 5.44
Tasks: 428 total,   5 running, 423 sleeping,   0 stopped,   0 zombie
Cpu(s): 35.3%us, 34.0%sy,  0.0%ni, 30.4%id,  0.2%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  24723620k total, 23626580k used,  1097040k free,   325988k buffers
Swap:  2064380k total,    79124k used,  1985256k free, 16850200k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 11859 mysql     20   0 6309m 172m 6900 S 58.4  0.7   3836:44 mysqld
 62769 apache    20   0  488m  66m 4960 S 15.6  0.3  10:15.81 httpd
 98436 apache    20   0  483m  60m 6016 R 11.7  0.3   7:11.17 httpd
101897 apache    20   0  483m  62m 4864 R 11.7  0.3   8:04.92 httpd
  8420 nagios    20   0  103m 1424 1236 S  7.8  0.0 890:32.71 backup_nagios.s
118320 nagios    20   0  103m 1380 1236 S  7.8  0.0   1439:47 backup_nagios.s
  9545 nagios    20   0  103m 1388 1236 S  5.8  0.0 749:48.36 backup_nagios.s
 40323 nagios    20   0 56460 8028 1036 S  5.8  0.0  26:50.11 ndo2db
 48458 nagios    20   0  103m 1448 1236 S  5.8  0.0 622:38.58 backup_nagios.s
 66345 nagios    20   0  103m 1444 1236 S  5.8  0.0  75:14.46 backup_nagios.s
 67331 apache    20   0  484m  62m 6068 S  5.8  0.3   8:08.67 httpd
 91164 nagios    20   0  103m 1440 1236 S  5.8  0.0 509:47.56 backup_nagios.s
 94681 nagios    20   0  103m 1376 1236 S  5.8  0.0   1025:17 backup_nagios.s
 95245 nagios    20   0  103m 1444 1236 S  5.8  0.0 174:57.72 backup_nagios.s
   891 nagios    20   0  103m 1376 1236 S  3.9  0.0   1570:21 backup_nagios.s

Is that something of your own creation, or a third-party script you downloaded somewhere?
Whatever it is, it seems to be chewing up quite a lot of CPU.

Also, your nagios log file isn't readable, or is missing.
What do you get if you do a "ls -la /usr/local/nagios/var/nagios.log"?


You may have other problems also, but let's start there.

Thanks!

[noweda]

Hi,

thanks for your reply! Here's the output for nagios.log:

root@ux010162:~# ls -la /usr/local/nagios/var/nagios.log
-rw-r--r-- 1 nagios nagios 2561130 Apr 6 08:01 /usr/local/nagios/var/nagios.log

The script seems to be a custom script generated by one of our oracle admins - I didn't know it until now... I will check that.

Best regards
Christoph

[ssax]

It looks like they are not finishing properly and probably causing that as @jdunitz found, let us know what your team says.

Thank you

[noweda]

Hi,

our oracle admin stopped the custom script and it's lower the usage.

This issue is resolved now.

Thanks for your support.

Sincerely,

[scottwilkerson]

Hi,

our oracle admin stopped the custom script and it's lower the usage.

This issue is resolved now.

Thanks for your support.

Sincerely,

Great!

Locking thread