Not able to place servers in maintenance mode

[raamardhani7]

Hi Team,

We are facing two major issues:
1) Users are unable to put the servers in Scheduled downtime, when even the load was fine
2) load is high
3) Notifications are not sent by the server in last more than 24 hours.

Below are our observations:

1) XI Server A is monitoring XI Server B. XI Server B is throwing the error in Service Status Details for the following services:

I/O Wait
Load
Nagios XI Daemons
Nagios XI Jobs

On the Admin tab XI System Component Status, all service are running green.

2) mysqld and splunk was using high CPU so we killed splunk and restarted Nagios Xi services and repaired the database

3) Server is able to send the test mail, but notifications are not sent.

4) Please find the attached output and command as mentioned in below link:
https://support.nagios.com/forum/viewto ... =6&t=41667

5)we found few idle process running on Nagios


Kindly help on this and let us know, if we can schedule a call to discuss this. If yes, please share the contact details and time.

[lmiltchev]

Are you having issues with Server A, B or both? Let's focus on one server (and one issue) at a time.

Log in the problem server, go to Admin > System Config > System Profile, click on "Download Profile" and PM me (or anyone on the Nagios Support team) the profile.zip file. We will need to review your configuration files, and various logs in order to identify the issue that you are having.

[raamardhani7]

Hi All,

We have shared the profile personally, kindly have a look and suggest us a solution.

Attached message is appearing randomly on Nagios GUI when we are performing any task.

[tgriep]

I see that there was a ticket opened for this issue, do you want to continue in the ticket or here in the forum for support?

[awilson]

I'd like to continue in the forum. The ticket was an attempted workaround because I could not upload the zip file. I'll close the ticket.

Thanks!
Alan

[tgriep]

It looks like there are a lot of defunct processes on the server and that could be driving up the load so let's stop them from running and restart them.

To do that, run the following as root on the Nagios server.

Code: Select all

service npcd stop
service nagios stop
service ndo2db stop
if grep --quiet pgsql /usr/local/nagiosxi/html/config.inc.php; then service postgresql stop; fi;
service mysqld restart
rm -rf /usr/local/nagios/var/rw/nagios.cmd
rm -rf /usr/local/nagios/var/nagios.lock
rm -rf /usr/local/nagios/var/ndo.sock
rm -rf /usr/local/nagios/var/ndo2db.lock
rm -rf /usr/local/nagiosxi/var/reconfigure_nagios.lock
rm -rf /var/lib/mrtg/mrtg_l
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
pkill -9 -u nagios
if grep --quiet pgsql /usr/local/nagiosxi/html/config.inc.php; then service postgresql start; fi;
service httpd restart
service ndo2db start
service nagios start
service npcd start
service crond restart

After this, the load should be lower on the server.

One thing I would like to look as is this Postgress Log file. Can you get this file from the server and add it to the post?

Code: Select all

/var/lib/pgsql/data/pg_log/postgresql-Mon.log

[awilson]

Hi. The postgresql log file is attached to the ticket. High CPU Load - Server unresponsive via Nagios XI #411506

[scottwilkerson]

I am locking this thread as it is now being worked via support ticket.