High CPU Load due to multiple Nagis processes

[aditya.c.anand]

Hello Team,

We are facing a high utilization of CPU on the NagiosXI production instance for past 2 days due to multiple java and mysqld (nagios) process.
We tried to kill multiple java processes, but still the utilization is high.

Can you please suggest anything here, as this is impacting our Nagios application.

[npolovenko]

Hello, @aditya.c.anand. Please run the following commands in order and let me know if it resolves the issue:

service nagios stop
service ndo2db stop
service mysqld stop
service crond stop
service httpd stop
killall -9 nagios
killall -9 ndo2db
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
service mysqld start
service ndo2db start
service nagios start
service httpd start
service crond start

[aditya.c.anand]

Hello Team,

we performed the steps, but for few hours i.e in off business hours where the utilization was very low the load was down, but again it went high.

Kindly suggest as multiple java processes are utilizing the load and due to heavy load we can see performance issue on the tool.

Please suggest.

[npolovenko]

@aditya.c.anand, You said you're running Java processes on the XI server? We don't recommend running any third party applications on the Nagios server to avoid interferences. Please run the following commands and then upload the nagios.txt file from the /tmp/ folder:

ps -aef > /tmp/nagios.txt
top >> /tmp/nagios.txt

[aditya.c.anand]

Hello Team,

Please find the attached file.
We are not able to acknowledge the alerts properly due to heavy load.
Apply configurations is taking time and most importantly putting severs and it's services are taking way longer time than it was in the past.
The above all performance issues are due to heavy load on host.

Kindly help.

[aditya.c.anand]

Hello team,

The issue has started all of a sudden from 6th of November. Prior to this, everything was fine and nothing has been changed from server side of application side.
Kindly help why this issue came abruptly and this is hitting us brutally.

Regards,
Aditya Anand

[aditya.c.anand]

Hello Team,

The maximum java processes are getting initiated by check_as400 script and are taking maximum CPU utilization.
Can you please help here.

Regards,
Aditya Anand

[npolovenko]

@aditya.c.anand, Yeah, I was wondering where all these java processes come from. Looks like many of them are hanging for a few minutes. How many services are using this script at a given moment? (check_as400) Did you add more services recently?
I suggest running a few service checks that utilize this plugin from the command line with one of these verbose flags and look for any errors:

-d
Be verbose (debug)
NOTE: Needs to be one of the first arguments to work
-D
Be verbose and dump screen outputs (debug)
NOTES: Needs to be one of the first arguments to work

Also, make sure that you're using the latest version of this plugin:
https://github.com/cjt74392/check_as400

Finally, I suggest reaching out to the author of this plugin directly by opening an issue on his GitHub page:
https://github.com/cjt74392/check_as400/issues