nagios restart does not apply .cfg changes

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
User avatar
hbackus
Posts: 36
Joined: Tue May 17, 2011 5:02 pm

nagios restart does not apply .cfg changes

Post by hbackus »

Nagios Core 4.4.1
Changes made to .cfg file in /usr/local/nagios/etc/objects/ are not applied with /etc/init.d/nagios restart

If I make a change to .cfg file such as "hosts.cfg" that has new, removed or modified "define host { " definition; the change does not take effect until I reboot the Nagois Core server.

The nagios check says all is OK, "/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg"
Things look okay - No serious problems were detected during the pre-flight check

and restart says it executed OK " /etc/init.d/nagios restart"
Running configuration check... Stopping nagios
Starting nagios: Running configuration check... done.

But the changes do not show up in the web page or and I do not get email notifications until I do "reboot" on the server.

Got any ideas on where I should look for the problem?
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: nagios restart does not apply .cfg changes

Post by cdienger »

What do the object_cache_fileand status_file options in nagios.cfg look like? Are these files getting updated after making changes to the config? The nagios user should have read & write access to these files.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
hbackus
Posts: 36
Joined: Tue May 17, 2011 5:02 pm

Re: nagios restart does not apply .cfg changes

Post by hbackus »

Nagios user has access to the nagios.cfg file

-rw-rw-r--. 1 nagios nagios 45913 Sep 19 12:14 /usr/local/nagios/etc/nagios.cfg

The object cache file is define in the nagios.cfg file
object_cache_file=/usr/local/nagios/var/objects.cache

-rw-r--r--. 1 nagios nagios 136742 Oct 4 16:12 /usr/local/nagios/var/objects.cache

as well as the status file
status_file=/usr/local/nagios/var/status.dat

-rw-rw-r--. 1 nagios nagios 223112 Oct 18 13:46 /usr/local/nagios/var/status.dat

Any other suggestion on where I should look?
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: nagios restart does not apply .cfg changes

Post by npolovenko »

@ hbackus, Did this issue start after you upgraded to Core 4.4.1 from an older version? Please upload the init script here so I could examine it.
/etc/initd.nagios
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
hbackus
Posts: 36
Joined: Tue May 17, 2011 5:02 pm

Re: nagios restart does not apply .cfg changes

Post by hbackus »

Could not attach the file, sorry but I had to paste it into this reply:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.10.19 16:26:28 =~=~=~=~=~=~=~=~=~=~=~=
cat nagios
#!/bin/sh
#
# chkconfig: 345 99 01
# description: Nagios network monitor
# processname: nagios
# File : nagios
#
# Author : Jorge Sanchez Aymar (jsanchez@lanchile.cl)
#
# Changelog :
#
# 1999-07-09 Karl DeBisschop <kdebisschop@infoplease.com>
# - setup for autoconf
# - add reload function
# 1999-08-06 Ethan Galstad <egalstad@nagios.org>
# - Added configuration info for use with RedHat's chkconfig tool
# per Fran Boon's suggestion
# 1999-08-13 Jim Popovitch <jimpop@rocketship.com>
# - added variable for nagios/var directory
# - cd into nagios/var directory before creating tmp files on startup
# 1999-08-16 Ethan Galstad <egalstad@nagios.org>
# - Added test for rc.d directory as suggested by Karl DeBisschop
# 2000-07-23 Karl DeBisschop <kdebisschop@users.sourceforge.net>
# - Clean out redhat macros and other dependencies
# 2003-01-11 Ethan Galstad <egalstad@nagios.org>
# - Updated su syntax (Gary Miller)
#
# Description: Starts and stops the Nagios monitor
# used to provide network services status.
#
### BEGIN INIT INFO
# Provides:nagios
# Required-Start:$local_fs $syslog $network
# Required-Stop:$local_fs $syslog $network
# Default-Start:2 3 4 5
# Default-Stop:0 1 6
# Short-Description:Starts and stops the Nagios monitoring server
# Description:Starts and stops the Nagios monitoring server
### END INIT INFO

# Our install-time configuration.
NagiosBin=/usr/local/nagios/bin/nagios
NagiosCfgFile=/usr/local/nagios/etc/nagios.cfg
NagiosCfgtestFile=/usr/local/nagios/var/nagios.configtest
NagiosStatusFile=/usr/local/nagios/var/status.dat
NagiosRetentionFile=/usr/local/nagios/var/retention.dat
NagiosCommandFile=/usr/local/nagios/var/rw/nagios.cmd
NagiosRunFile=/var/run/nagios.lock
NagiosVarDir=/usr/local/nagios/var
NagiosCGIDir=/usr/local/nagios/sbin
NagiosUser=nagios
NagiosGroup=nagios
checkconfig="true"

# Load any extra environment variables for Nagios and its plugins.
if test -f /etc/sysconfig/nagios; then
. /etc/sysconfig/nagios
fi

# Automate addition of RAMDISK based on environment variables
USE_RAMDISK=${USE_RAMDISK:-0}
if test "$USE_RAMDISK" -ne 0 && test "$RAMDISK_SIZE"X != "X"; then
ramdisk=`mount |grep "${RAMDISK_DIR} type tmpfs"`
if [ "$ramdisk"X == "X" ]; then
if [ ! -d ${RAMDISK_DIR} ]; then
mkdir -p -m 0755 ${RAMDISK_DIR}
if [ -d ${RAMDISK_DIR} ]; then
NagiosUserUID=`id -u ${NagiosUser}`
NagiosGroupGID=`id -g ${NagiosGroup}`
mount -t tmpfs -o uid=${NagiosUserUID},gid={$NagiosGroupGID},size=${RAMDISK_SIZE}m tmpfs ${RAMDISK_DIR}
su ${NagiosUser} -c "mkdir -m 0755 ${RAMDISK_DIR}/checkresults"
else
echo "ERROR: Unable to create ramdisk directory (${RAMDISK_DIR})"
exit 1
fi
fi
fi
fi


check_config ()
{
if test "$checkconfig" != "true"; then
return 0
fi

echo -n "Running configuration check... "

/bin/rm -f "$NagiosCfgtestFile";
if test -e "$NagiosCfgtestFile"; then
echo "ERROR: Could not delete '$NagiosCfgtestFile'"
exit 8
fi
if ! su $NagiosUser -c "touch $NagiosCfgtestFile"; then
echo "ERROR: Could not create or update '$NagiosCfgtestFile'"
exit 8
fi

TMPFILE=$(mktemp /tmp/.configtest.XXXXXXXX)
$NagiosBin -vp $NagiosCfgFile > "$TMPFILE"
WARN=`grep ^"Total Warnings:" "$TMPFILE" |awk -F: '{print \$2}' |sed s/' '//g`
ERR=`grep ^"Total Errors:" "$TMPFILE" |awk -F: '{print \$2}' |sed s/' '//g`

if test "$WARN" = "0" && test "${ERR}" = "0"; then
echo "OK - Configuration check verified" > $NagiosCfgtestFile
/bin/rm "$TMPFILE"
return 0
elif test "${ERR}" = "0"; then
# Write the errors to a file we can have a script watching for.
echo "WARNING: Warnings in config files - see log for details: $NagiosCfgtestFile" > $NagiosCfgtestFile
egrep -i "(^warning|^error)" "$TMPFILE" >> $NagiosCfgtestFile
/bin/rm "$TMPFILE"
return 0
else
# Write the errors to a file we can have a script watching for.
echo "ERROR: Errors in config files - see log for details: $NagiosCfgtestFile" > $NagiosCfgtestFile
egrep -i "(^warning|^error)" "$TMPFILE" >> $NagiosCfgtestFile
cat "$TMPFILE"
exit 8
fi
}


status_nagios ()
{
if test -x $NagiosCGIDir/daemonchk.cgi; then
if $NagiosCGIDir/daemonchk.cgi -l $NagiosRunFile > /dev/null 2>&1; then
return 0
fi
else
if ps -p $NagiosPID > /dev/null 2>&1; then
return 0
fi
fi

return 1
}

printstatus_nagios ()
{
if status_nagios; then
echo "nagios (pid $NagiosPID) is running..."
else
echo "nagios is not running"
fi
}

killproc_nagios ()
{
/bin/kill -s "$1" $NagiosPID
}

pid_nagios ()
{
if test ! -f $NagiosRunFile; then
echo "No lock file found in $NagiosRunFile"
return 1
fi

NagiosPID=`head -n 1 $NagiosRunFile`
}

# Check that nagios exists.
if [ ! -f $NagiosBin ]; then
echo "Executable file $NagiosBin not found. Exiting."
exit 1
fi

# Check that nagios.cfg exists.
if [ ! -f $NagiosCfgFile ]; then
echo "Configuration file $NagiosCfgFile not found. Exiting."
exit 1
fi

# See how we were called.
case "$1" in

start)
echo -n "Starting nagios: "

check_config

if test -f $NagiosRunFile; then
NagiosPID=`head -n 1 $NagiosRunFile`
if status_nagios; then
echo "another instance of nagios is already running."
exit 0
fi
fi

su $NagiosUser -c "touch $NagiosVarDir/nagios.log $NagiosRetentionFile"
/bin/rm -f $NagiosCommandFile
touch $NagiosRunFile
$NagiosBin -d $NagiosCfgFile

echo "done."
;;

stop)
echo -n "Stopping nagios: "

pid_nagios
if [ $? -eq 1 ]; then
exit 0
fi
killproc_nagios TERM

# now we have to wait for nagios to exit and remove its
# own NagiosRunFile, otherwise a following "start" could
# happen, and then the exiting nagios will remove the
# new NagiosRunFile, allowing multiple nagios daemons
# to (sooner or later) run - John Sellens
#echo -n 'Waiting for nagios to exit .'
for i in {1..90}; do
if status_nagios > /dev/null; then
echo -n '.'
sleep 1
else
break
fi
done
if status_nagios > /dev/null; then
echo ""
echo "Warning - nagios did not exit in a timely manner - Killing it!"
killproc_nagios KILL
else
echo "done."
fi

/bin/rm -f $NagiosStatusFile $NagiosRunFile $NagiosCommandFile
;;

status)
pid_nagios
if [ $? -eq 1 ]; then
exit 3
fi
printstatus_nagios
;;

checkconfig)
check_config

if [ $? -eq 0 ]; then
echo "OK."
else
echo "CONFIG ERROR! Check your Nagios configuration."
exit 1
fi
;;

restart)
check_config

$0 stop
$0 start
;;

reload|force-reload)
check_config

if test ! -f $NagiosRunFile; then
$0 start
else
pid_nagios
if status_nagios > /dev/null; then
echo -n "Reloading nagios configuration... "
killproc_nagios HUP
echo "done"
else
$0 stop
$0 start
fi
fi
;;

configtest)
$NagiosBin -vp $NagiosCfgFile
;;

*)
echo "Usage: nagios {start|stop|restart|reload|force-reload|status|checkconfig|configtest}"
exit 1
;;

esac

# End of this script
[root@nagios init.d]#
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: nagios restart does not apply .cfg changes

Post by ssax »

Do you have too many nagios processes running?

Code: Select all

ps aux | grep nagios.cfg | grep -v grep
There should be two listed, no more than two otherwise you have a problem.

What could be happening is that when you restart the service it's still using an old process that contains outdated information and then rebooting the server could start them up fresh and pull the correct information since you don't have those old nagios processes running.

Thank you
User avatar
hbackus
Posts: 36
Joined: Tue May 17, 2011 5:02 pm

Re: nagios restart does not apply .cfg changes

Post by hbackus »

Thanks for the reply

# ps aux | grep nagios.cfg | grep -v grep
nagios 1468 0.0 0.0 16636 3816 ? Ss 15:35 0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 1476 0.0 0.0 15948 784 ? S 15:35 0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
#

Just two processes running.

Since I did not have this problem until the upgrade to Ver. 4.4.1, how would I be able to identify old and new process related to Nagios?
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: nagios restart does not apply .cfg changes

Post by npolovenko »

@hbackus, Please open the /usr/local/nagios/etc/nagios.cfg file and make sure that the:
lock_file=/var/run/nagios.lock
Then restart nagios with:
service nagios restart
Show me the output of the following command:
ls -l /var/run/nagios.lock
And after you try to restart nagios please send me the output of:
tail -f 200 /usr/local/nagios/var/nagios.log
Attachments
nagios.7z
(2.58 KiB) Downloaded 198 times
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
hbackus
Posts: 36
Joined: Tue May 17, 2011 5:02 pm

Re: nagios restart does not apply .cfg changes

Post by hbackus »

Lock file location in nagios.cfg file:
lock_file=/usr/local/nagios/var/nagios.lock

# ls -l /usr/local/nagios/var/nagios.lock
-rw-r--r--. 1 root root 5 Dec 1 16:21 /usr/local/nagios/var/nagios.lock

# tail -f 200 /usr/local/nagios/var/nagios.log
tail: cannot open `200' for reading: No such file or directory
==> /usr/local/nagios/var/nagios.log <==
[1543702874] WARNING: The normal_check_interval attribute is deprecated and will be removed in future versions. Please use check_interval instead.
[1543702874] WARNING: The retry_check_interval attribute is deprecated and will be removed in future versions. Please use retry_interval instead.
[1543702874] Successfully launched command file worker with pid 1476
[1543702952] SERVICE ALERT: hsb-app-prod05;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.15 ms
[1543702972] SERVICE ALERT: bhds-edge-router;PING;CRITICAL;SOFT;1;PING CRITICAL - Packet loss = 40%, RTA = 39.94 ms
[1543702980] SERVICE ALERT: el-108;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 27.33 ms
[1543703031] SERVICE ALERT: bhds-edge-router;PING;OK;SOFT;1;PING OK - Packet loss = 0%, RTA = 40.00 ms
[1543703039] SERVICE ALERT: el-108;PING;OK;SOFT;1;PING OK - Packet loss = 0%, RTA = 27.30 ms
[1543703071] SERVICE ALERT: hsb-app-prod03;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.36 ms
[1543703120] SERVICE ALERT: bc-app07;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.17 ms
[1543703128] SERVICE ALERT: bc-app13-vm;PING;WARNING;SOFT;1;PING WARNING - Packet loss = 20%, RTA = 37.47 ms
[1543703136] SERVICE ALERT: hsb-app-prod05;PING;OK;SOFT;1;PING OK - Packet loss = 0%, RTA = 37.15 ms
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: nagios restart does not apply .cfg changes

Post by scottwilkerson »

on 10/19/2018 you posted your nagios init script which had the lock file here

Code: Select all

NagiosRunFile=/var/run/nagios.lock
This needs to match the location you see in your nagios.cfg
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
Locked