Page 1 of 2

nagios restart does not apply .cfg changes

Posted: Sun Oct 14, 2018 4:16 pm
by hbackus
Nagios Core 4.4.1
Changes made to .cfg file in /usr/local/nagios/etc/objects/ are not applied with /etc/init.d/nagios restart

If I make a change to .cfg file such as "hosts.cfg" that has new, removed or modified "define host { " definition; the change does not take effect until I reboot the Nagois Core server.

The nagios check says all is OK, "/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg"
Things look okay - No serious problems were detected during the pre-flight check

and restart says it executed OK " /etc/init.d/nagios restart"
Running configuration check... Stopping nagios
Starting nagios: Running configuration check... done.

But the changes do not show up in the web page or and I do not get email notifications until I do "reboot" on the server.

Got any ideas on where I should look for the problem?

Re: nagios restart does not apply .cfg changes

Posted: Mon Oct 15, 2018 12:01 pm
by cdienger
What do the object_cache_fileand status_file options in nagios.cfg look like? Are these files getting updated after making changes to the config? The nagios user should have read & write access to these files.

Re: nagios restart does not apply .cfg changes

Posted: Thu Oct 18, 2018 1:48 pm
by hbackus
Nagios user has access to the nagios.cfg file

-rw-rw-r--. 1 nagios nagios 45913 Sep 19 12:14 /usr/local/nagios/etc/nagios.cfg

The object cache file is define in the nagios.cfg file
object_cache_file=/usr/local/nagios/var/objects.cache

-rw-r--r--. 1 nagios nagios 136742 Oct 4 16:12 /usr/local/nagios/var/objects.cache

as well as the status file
status_file=/usr/local/nagios/var/status.dat

-rw-rw-r--. 1 nagios nagios 223112 Oct 18 13:46 /usr/local/nagios/var/status.dat

Any other suggestion on where I should look?

Re: nagios restart does not apply .cfg changes

Posted: Thu Oct 18, 2018 4:20 pm
by npolovenko
@ hbackus, Did this issue start after you upgraded to Core 4.4.1 from an older version? Please upload the init script here so I could examine it.
/etc/initd.nagios

Re: nagios restart does not apply .cfg changes

Posted: Fri Oct 19, 2018 4:47 pm
by hbackus
Could not attach the file, sorry but I had to paste it into this reply:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.10.19 16:26:28 =~=~=~=~=~=~=~=~=~=~=~=
cat nagios
#!/bin/sh
#
# chkconfig: 345 99 01
# description: Nagios network monitor
# processname: nagios
# File : nagios
#
# Author : Jorge Sanchez Aymar ([email protected])
#
# Changelog :
#
# 1999-07-09 Karl DeBisschop <[email protected]>
# - setup for autoconf
# - add reload function
# 1999-08-06 Ethan Galstad <[email protected]>
# - Added configuration info for use with RedHat's chkconfig tool
# per Fran Boon's suggestion
# 1999-08-13 Jim Popovitch <[email protected]>
# - added variable for nagios/var directory
# - cd into nagios/var directory before creating tmp files on startup
# 1999-08-16 Ethan Galstad <[email protected]>
# - Added test for rc.d directory as suggested by Karl DeBisschop
# 2000-07-23 Karl DeBisschop <[email protected]>
# - Clean out redhat macros and other dependencies
# 2003-01-11 Ethan Galstad <[email protected]>
# - Updated su syntax (Gary Miller)
#
# Description: Starts and stops the Nagios monitor
# used to provide network services status.
#
### BEGIN INIT INFO
# Provides:nagios
# Required-Start:$local_fs $syslog $network
# Required-Stop:$local_fs $syslog $network
# Default-Start:2 3 4 5
# Default-Stop:0 1 6
# Short-Description:Starts and stops the Nagios monitoring server
# Description:Starts and stops the Nagios monitoring server
### END INIT INFO

# Our install-time configuration.
NagiosBin=/usr/local/nagios/bin/nagios
NagiosCfgFile=/usr/local/nagios/etc/nagios.cfg
NagiosCfgtestFile=/usr/local/nagios/var/nagios.configtest
NagiosStatusFile=/usr/local/nagios/var/status.dat
NagiosRetentionFile=/usr/local/nagios/var/retention.dat
NagiosCommandFile=/usr/local/nagios/var/rw/nagios.cmd
NagiosRunFile=/var/run/nagios.lock
NagiosVarDir=/usr/local/nagios/var
NagiosCGIDir=/usr/local/nagios/sbin
NagiosUser=nagios
NagiosGroup=nagios
checkconfig="true"

# Load any extra environment variables for Nagios and its plugins.
if test -f /etc/sysconfig/nagios; then
. /etc/sysconfig/nagios
fi

# Automate addition of RAMDISK based on environment variables
USE_RAMDISK=${USE_RAMDISK:-0}
if test "$USE_RAMDISK" -ne 0 && test "$RAMDISK_SIZE"X != "X"; then
ramdisk=`mount |grep "${RAMDISK_DIR} type tmpfs"`
if [ "$ramdisk"X == "X" ]; then
if [ ! -d ${RAMDISK_DIR} ]; then
mkdir -p -m 0755 ${RAMDISK_DIR}
if [ -d ${RAMDISK_DIR} ]; then
NagiosUserUID=`id -u ${NagiosUser}`
NagiosGroupGID=`id -g ${NagiosGroup}`
mount -t tmpfs -o uid=${NagiosUserUID},gid={$NagiosGroupGID},size=${RAMDISK_SIZE}m tmpfs ${RAMDISK_DIR}
su ${NagiosUser} -c "mkdir -m 0755 ${RAMDISK_DIR}/checkresults"
else
echo "ERROR: Unable to create ramdisk directory (${RAMDISK_DIR})"
exit 1
fi
fi
fi
fi


check_config ()
{
if test "$checkconfig" != "true"; then
return 0
fi

echo -n "Running configuration check... "

/bin/rm -f "$NagiosCfgtestFile";
if test -e "$NagiosCfgtestFile"; then
echo "ERROR: Could not delete '$NagiosCfgtestFile'"
exit 8
fi
if ! su $NagiosUser -c "touch $NagiosCfgtestFile"; then
echo "ERROR: Could not create or update '$NagiosCfgtestFile'"
exit 8
fi

TMPFILE=$(mktemp /tmp/.configtest.XXXXXXXX)
$NagiosBin -vp $NagiosCfgFile > "$TMPFILE"
WARN=`grep ^"Total Warnings:" "$TMPFILE" |awk -F: '{print \$2}' |sed s/' '//g`
ERR=`grep ^"Total Errors:" "$TMPFILE" |awk -F: '{print \$2}' |sed s/' '//g`

if test "$WARN" = "0" && test "${ERR}" = "0"; then
echo "OK - Configuration check verified" > $NagiosCfgtestFile
/bin/rm "$TMPFILE"
return 0
elif test "${ERR}" = "0"; then
# Write the errors to a file we can have a script watching for.
echo "WARNING: Warnings in config files - see log for details: $NagiosCfgtestFile" > $NagiosCfgtestFile
egrep -i "(^warning|^error)" "$TMPFILE" >> $NagiosCfgtestFile
/bin/rm "$TMPFILE"
return 0
else
# Write the errors to a file we can have a script watching for.
echo "ERROR: Errors in config files - see log for details: $NagiosCfgtestFile" > $NagiosCfgtestFile
egrep -i "(^warning|^error)" "$TMPFILE" >> $NagiosCfgtestFile
cat "$TMPFILE"
exit 8
fi
}


status_nagios ()
{
if test -x $NagiosCGIDir/daemonchk.cgi; then
if $NagiosCGIDir/daemonchk.cgi -l $NagiosRunFile > /dev/null 2>&1; then
return 0
fi
else
if ps -p $NagiosPID > /dev/null 2>&1; then
return 0
fi
fi

return 1
}

printstatus_nagios ()
{
if status_nagios; then
echo "nagios (pid $NagiosPID) is running..."
else
echo "nagios is not running"
fi
}

killproc_nagios ()
{
/bin/kill -s "$1" $NagiosPID
}

pid_nagios ()
{
if test ! -f $NagiosRunFile; then
echo "No lock file found in $NagiosRunFile"
return 1
fi

NagiosPID=`head -n 1 $NagiosRunFile`
}

# Check that nagios exists.
if [ ! -f $NagiosBin ]; then
echo "Executable file $NagiosBin not found. Exiting."
exit 1
fi

# Check that nagios.cfg exists.
if [ ! -f $NagiosCfgFile ]; then
echo "Configuration file $NagiosCfgFile not found. Exiting."
exit 1
fi

# See how we were called.
case "$1" in

start)
echo -n "Starting nagios: "

check_config

if test -f $NagiosRunFile; then
NagiosPID=`head -n 1 $NagiosRunFile`
if status_nagios; then
echo "another instance of nagios is already running."
exit 0
fi
fi

su $NagiosUser -c "touch $NagiosVarDir/nagios.log $NagiosRetentionFile"
/bin/rm -f $NagiosCommandFile
touch $NagiosRunFile
$NagiosBin -d $NagiosCfgFile

echo "done."
;;

stop)
echo -n "Stopping nagios: "

pid_nagios
if [ $? -eq 1 ]; then
exit 0
fi
killproc_nagios TERM

# now we have to wait for nagios to exit and remove its
# own NagiosRunFile, otherwise a following "start" could
# happen, and then the exiting nagios will remove the
# new NagiosRunFile, allowing multiple nagios daemons
# to (sooner or later) run - John Sellens
#echo -n 'Waiting for nagios to exit .'
for i in {1..90}; do
if status_nagios > /dev/null; then
echo -n '.'
sleep 1
else
break
fi
done
if status_nagios > /dev/null; then
echo ""
echo "Warning - nagios did not exit in a timely manner - Killing it!"
killproc_nagios KILL
else
echo "done."
fi

/bin/rm -f $NagiosStatusFile $NagiosRunFile $NagiosCommandFile
;;

status)
pid_nagios
if [ $? -eq 1 ]; then
exit 3
fi
printstatus_nagios
;;

checkconfig)
check_config

if [ $? -eq 0 ]; then
echo "OK."
else
echo "CONFIG ERROR! Check your Nagios configuration."
exit 1
fi
;;

restart)
check_config

$0 stop
$0 start
;;

reload|force-reload)
check_config

if test ! -f $NagiosRunFile; then
$0 start
else
pid_nagios
if status_nagios > /dev/null; then
echo -n "Reloading nagios configuration... "
killproc_nagios HUP
echo "done"
else
$0 stop
$0 start
fi
fi
;;

configtest)
$NagiosBin -vp $NagiosCfgFile
;;

*)
echo "Usage: nagios {start|stop|restart|reload|force-reload|status|checkconfig|configtest}"
exit 1
;;

esac

# End of this script
[root@nagios init.d]#

Re: nagios restart does not apply .cfg changes

Posted: Mon Oct 22, 2018 2:32 pm
by ssax
Do you have too many nagios processes running?

Code: Select all

ps aux | grep nagios.cfg | grep -v grep
There should be two listed, no more than two otherwise you have a problem.

What could be happening is that when you restart the service it's still using an old process that contains outdated information and then rebooting the server could start them up fresh and pull the correct information since you don't have those old nagios processes running.

Thank you

Re: nagios restart does not apply .cfg changes

Posted: Mon Oct 22, 2018 3:44 pm
by hbackus
Thanks for the reply

# ps aux | grep nagios.cfg | grep -v grep
nagios 1468 0.0 0.0 16636 3816 ? Ss 15:35 0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 1476 0.0 0.0 15948 784 ? S 15:35 0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
#

Just two processes running.

Since I did not have this problem until the upgrade to Ver. 4.4.1, how would I be able to identify old and new process related to Nagios?

Re: nagios restart does not apply .cfg changes

Posted: Mon Oct 22, 2018 4:21 pm
by npolovenko
@hbackus, Please open the /usr/local/nagios/etc/nagios.cfg file and make sure that the:
lock_file=/var/run/nagios.lock
Then restart nagios with:
service nagios restart
Show me the output of the following command:
ls -l /var/run/nagios.lock
And after you try to restart nagios please send me the output of:
tail -f 200 /usr/local/nagios/var/nagios.log

Re: nagios restart does not apply .cfg changes

Posted: Sat Dec 01, 2018 5:30 pm
by hbackus
Lock file location in nagios.cfg file:
lock_file=/usr/local/nagios/var/nagios.lock

# ls -l /usr/local/nagios/var/nagios.lock
-rw-r--r--. 1 root root 5 Dec 1 16:21 /usr/local/nagios/var/nagios.lock

# tail -f 200 /usr/local/nagios/var/nagios.log
tail: cannot open `200' for reading: No such file or directory
==> /usr/local/nagios/var/nagios.log <==
[1543702874] WARNING: The normal_check_interval attribute is deprecated and will be removed in future versions. Please use check_interval instead.
[1543702874] WARNING: The retry_check_interval attribute is deprecated and will be removed in future versions. Please use retry_interval instead.
[1543702874] Successfully launched command file worker with pid 1476
[1543702952] SERVICE ALERT: hsb-app-prod05;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.15 ms
[1543702972] SERVICE ALERT: bhds-edge-router;PING;CRITICAL;SOFT;1;PING CRITICAL - Packet loss = 40%, RTA = 39.94 ms
[1543702980] SERVICE ALERT: el-108;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 27.33 ms
[1543703031] SERVICE ALERT: bhds-edge-router;PING;OK;SOFT;1;PING OK - Packet loss = 0%, RTA = 40.00 ms
[1543703039] SERVICE ALERT: el-108;PING;OK;SOFT;1;PING OK - Packet loss = 0%, RTA = 27.30 ms
[1543703071] SERVICE ALERT: hsb-app-prod03;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.36 ms
[1543703120] SERVICE ALERT: bc-app07;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.17 ms
[1543703128] SERVICE ALERT: bc-app13-vm;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.47 ms
[1543703136] SERVICE ALERT: hsb-app-prod05;PING;OK;SOFT;1;PING OK - Packet loss = 0%, RTA = 37.15 ms

Re: nagios restart does not apply .cfg changes

Posted: Mon Dec 03, 2018 4:41 pm
by scottwilkerson
on 10/19/2018 you posted your nagios init script which had the lock file here

Code: Select all

NagiosRunFile=/var/run/nagios.lock
This needs to match the location you see in your nagios.cfg