nagios services not getting killed

[nagiosconnect]

Hi all,
There was a ghost host and i tried to restart the nagios service to over come that problem.
But i when i tried to stop the nagios i got the error as below
Warning - nagios did not exit in a timely manner

I have increased the time in the stop script to - for i in {1..900} - but still i am getting the same issue.
At present i am able to start the nagios service, But all my services are failing. I tried a forceful check on the existing servicce and i am getting the below error
your request was not processed in a timely manner. it may still execute, as the server may be temporarily busy.

I can see around 700+ processor in my system under nagios. killall is not removing the processors.
I believe the server is not getting shutdown in a properly
Please guide me on how can i bring back al the services.

[tgriep]

What version of Nagios XI are you running on your server?
Can you login to the server as root, run the following commands and post the output?

Code: Select all

chage -l nagios
ps -ef --cols=300
df -h
df -i

Have you tried to reboot the system?

[nagiosconnect]

Thank you for the time. we are using NagiosXI5.4.3,
I have not restarted the server. Please find the details you have requsted.
It takes a long time to reflect in the GUI after we commit the changes.

Please find the answers

[root@307 ~]# chage -l nagios
Last password change : Mar 21, 2017
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
========================================

[root@307 ~]# ps -ef | grep nagios |wc -l
873
-==============================================
ps -ef --cols=300 | grep nagios | more

nagios 61635 60699 0 May29 ? 00:00:00 [nagios] <defunct>
nagios 61636 60699 0 May29 ? 00:00:00 [nagios] <defunct>
nagios 61637 60699 0 May29 ? 00:00:00 [nagios] <defunct>
nagios 61889 61443 0 02:33 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 61890 61443 0 02:33 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 61891 61443 0 02:33 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 61892 61443 0 02:33 ? 00:00:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 61893 61443 0 02:33 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 61894 61443 0 02:33 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 62149 1 0 May25 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 62250 27799 0 02:34 ? 00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios 62251 62250 0 02:34 ? 00:00:16 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios 62357 62149 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 62358 62149 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 62359 62149 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 62360 62149 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 62362 62149 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 62363 62149 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 62420 1 0 00:26 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 62430 1 0 May24 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 62432 62430 0 May24 ? 00:00:00 [nagios] <defunct>
nagios 62433 62430 0 May24 ? 00:00:00 [nagios] <defunct>
nagios 62434 62430 0 May24 ? 00:00:00 [nagios] <defunct>
nagios 62435 62430 0 May24 ? 00:00:00 [nagios] <defunct>
nagios 62436 62430 0 May24 ? 00:00:00 [nagios] <defunct>
nagios 62438 62430 0 May24 ? 00:00:00 [nagios] <defunct>
nagios 62769 62420 0 00:27 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 62770 62420 0 00:27 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 62771 62420 0 00:27 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 62772 62420 0 00:27 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 62773 62420 0 00:27 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 62774 62420 0 00:27 ? 00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 63233 1 0 May25 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 63235 63233 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 63236 63233 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 63238 63233 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 63239 63233 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 63240 63233 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 63241 63233 0 May25 ? 00:00:00 [nagios] <defunct>
nagios 63318 27799 0 00:28 ? 00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios 63319 63318 0 00:28 ? 00:00:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios 64220 62430 0 May24 ? 00:01:07 [nagios] <defunct>
nagios 65320 61184 0 May25 ? 00:01:22 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
............................ more are there


[root@307 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
15G 6.0G 7.9G 44% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 194M 103M 82M 56% /boot
/dev/mapper/rootvg-home_lv
2.0G 78M 1.8G 5% /local_home
/dev/mapper/rootvg-opt_lv
6.0G 388M 5.3G 7% /opt
/dev/mapper/rootvg-tmp_lv
2.0G 1.1G 824M 57% /tmp
/dev/mapper/rootvg-usr_lv
2.0G 1.4G 485M 75% /usr/local
/dev/mapper/rootvg-var_lv
4.0G 3.3G 483M 88% /var
/dev/mapper/rootvg-history_lv
2.0G 68M 1.9G 4% /history_logs
/dev/mapper/rootvg-itm_lv
3.0G 374M 2.5G 14% /opt/IBM/ITM
/dev/mapper/appvg-apps_lv
247G 580M 234G 1% /apps

[lmiltchev]

Can you PM me (or any of us on the Nagios Support team) your profile (profile.zip)? You can obtain it by going to Admin > System Profile > Download Profile. Thanks!