Lock File Moved after Config Failure

[blariv]

Hi all,

I have been having this problem for a while now. I thought it was addressed in an earlier post of mine concerning my monitoring engine not starting. I have been digging around in the forum and found this post. It is exactly my issue. I tried following the fixes but when I made all the changes my engine would just hang. Attached is my system profile.

https://support.nagios.com/forum/viewto ... ck#p273193

Thanks,

Brian

[npolovenko]

Hello, @blariv. Please run the following commands and show us the location of the nagios.lock file in each directory.

Code: Select all

cat /etc/rc.d/init.d/nagios | grep NagiosRunFile=
cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint.sh | grep lockfile=
cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint_specific.sh | grep lockfile=
cat /usr/local/nagios/etc/nagios.cfg | grep lock

[blariv]

I had changed these before to all be the "/var/run/nagios.lock" that are now commented out

[root@nagiosxi libexec]# cat /etc/rc.d/init.d/nagios | grep NagiosRunFile=
#NagiosRunFile=/var/run/nagios.lock
NagiosRunFile=${prefix}/var/nagios.lock

[root@nagiosxi libexec]# cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint.sh | grep lockfile=
#lockfile="/var/run/nagios.lock"
lockfile="/usr/local/nagios/var/nagios.lock"

[root@nagiosxi libexec]# cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint_specific.sh | grep lockfile=
#lockfile="/var/run/nagios.lock"
lockfile="/usr/local/nagios/var/nagios.lock"

[root@nagiosxi libexec]# cat /usr/local/nagios/etc/nagios.cfg | grep lock
lock_file=/usr/local/nagios/var/nagios.lock
#lock_file=/var/run/nagios.lock

[npolovenko]

@blariv, Is the nagios process starting with this lock file configuration? Or is it hanging? Is the lock file path getting reverted to the one you commented out?

[blariv]

its starts fine with this config, however if we happen to have a failed "apply configuration" due to misconfigured hosts or services the lock file location in /usr/local/nagios/etc/nagios.cfg changes to "lock_file=/var/run/nagios.lock". the engine will no longer start until I change the lock file location back to "lockfile="/usr/local/nagios/var/nagios.lock", delete the old locks, reload the deamon and then start nagios.

I can repeat this at any time as well if you need logs.

[npolovenko]

@blariv, This is strange. Are you on CentOS 6 or CentOS 7?

What's the location of the NagiosRunFile in /etc/init.d/nagios ?(if you have one on your server)


Please change all nagios.lock file locations to /var/run/nagios.lock.

Then run the following commands in order:

Code: Select all

service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start

And if nagios process "hangs" again, please send me the /usr/local/nagios/var/nagios.log.
Also, when you mentioned that your engine just hangs, did you mean it's not starting, or Apply Configuration hangs?

[blariv]

very strange indeed.

we are running RHEL7

init.d
NagiosRunFile=${prefix}/var/nagios.lock

the nagios service just hangs, not the apply config.

I will give this a shot and post results.

[npolovenko]

Sounds good.

[blariv]

same problem

see attached log and screenshot

[npolovenko]

@blariv, Please keep the lock file set to /var/run/ for now. Just to be sure, it's currently set to /var/run in all locations that were mentioned in this thread?

What's the output of all the following commands?

ls -ld /var/run/

ls -l /var/run/

ps -ef | grep nagios.cfg

/usr/local/nagiosxi/scripts/manage_services.sh status nagios