Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

CCM stuck on "Changes detected!" after Apply Configuration

https://support.nagios.com/forum/viewtopic.php?t=51593

Page 1 of 2

CCM stuck on "Changes detected!" after Apply Configuration

Posted: Tue Dec 11, 2018 4:59 pm
by stefanw
Hi all --

Issue summary: Somehow I've gotten XI / CCM into a state where any CCM config change causes "Changes detected! Apply Configuration for new changes to take effect." for an object type to appear even after running "Apply Configuration" one or more times.

Environment:
- RHEL 6.10 in AWS, 2.6.32-754.6.3.el6.x86_64 no outstanding patches
- XI 5.5.8 upgraded from 5.5.7 today (2018-12-11)
- MySQL 5.7.16 on AWS RDS

It's almost a blank slate with everything stripped out except what I believe are the out-of-the-box default config objects. (This XI instance is going to get configs imported from a legacy Core server.)

I only get the Changes detected! Apply Configuration for new changes to take effect. on certain sections of CCM. Right now it's showing up on:
- Hosts
- Services
- Service Templates
There's only one Host defined in CCM (localhost) and it's stuck on Status Not Applied
Of the 13 Services, I've edited only one and that's the one that is stuck on Not Applied
None of the Service Templates show Not Applied but the Changes detected! shows up there also.

I've done each of the following multiple times in various orders:
- reconfigure_nagios.sh
- repair_databases.sh
- reset_config_perms.sh
- revert to a config snapshot from prior to this issue appearing
- Config File Management --> Delete --> Write --> Verify

I've also read through every XI log file with:

Code: Select all

find /usr/local/nagiosxi/var -type f -mmin -90 ! -size 0c | xargs ls -rt | xargs less
I was tempted to do restore_defaults.sh but I haven't - yet. It's an option.

Thanks in advance for your help!




This is the output from /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Code: Select all

Checking objects...
	Checked 12 services.
	Checked 1 hosts.
	Checked 0 host groups.
	Checked 0 service groups.
	Checked 2 contacts.
	Checked 2 contact groups.
	Checked 126 commands.
	Checked 4 time periods.
	Checked 0 host escalations.
	Checked 0 service escalations.
Checking for circular paths...
	Checked 1 hosts
	Checked 0 service dependencies
	Checked 0 host dependencies
	Checked 4 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check

xiversion:

Code: Select all

full=5.5.8
major=5
minor=5.8
releasedate=2018-12-11
release=5508

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Tue Dec 11, 2018 5:14 pm
by stefanw
I forgot to mention - yes I did go through KB articles and all the relevant topics here that I could find. There were some references to this same kind of issue but nothing that completely matched my situation.

Also I'm not seeing any problems with the nagios account escalating privilege - nothing in /var/log/secure looks unusual or failed.

Also nothing jumps out from:

Code: Select all

find /var/log -type f -mmin -90 ! -size 0c | xargs ls -rt | xargs less
...except for a few innocuous-looking PHP "Undefined index" errors:

Code: Select all

[Tue Dec 11 16:13:37 2018] [error] [client x.x.x.x] PHP Notice:  Undefined index: count in /usr/local/nagiosxi/html/includes/components/ccm/page_templates/ccm_table.php on line 152, referer: https://my-server/nagiosxi/includes/components/ccm/xi-index.php
[Tue Dec 11 16:24:35 2018] [error] [client x.x.x.x] PHP Notice:  Undefined index: config_name in /usr/local/nagiosxi/html/includes/components/ccm/classes/data.class.php on line 1588, referer: https://my-server/nagiosxi/includes/components/ccm/?cmd=modify&type=servicetemplate&id=57&page=1&returnUrl=index.php%3Fcmd%3Dview%26type%3Dservicetemplate%26page%3D1
[Tue Dec 11 16:24:35 2018] [error] [client x.x.x.x] PHP Notice:  Undefined index: service_description in /usr/local/nagiosxi/html/includes/components/ccm/classes/data.class.php on line 1589, referer: https://my-server/nagiosxi/includes/components/ccm/?cmd=modify&type=servicetemplate&id=57&page=1&returnUrl=index.php%3Fcmd%3Dview%26type%3Dservicetemplate%26page%3D1

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Tue Dec 11, 2018 6:22 pm
by stefanw
More tidbits that might help...

This issue of "Apply Configuration" not registering in the UI did not just start when I upgraded from 5.5.7 to 5.5.8 today. It showed up earlier today prior to the upgrade.

In addition the "Not Applied" Status on the only Host in CCM appears to be an issue related to the XI UI/database interaction but not related to Core. When I make a change to the Host and Apply Configuration that change does show up in the Core config file on disk as normal.

Example: localhost didn't have a Notes entry until I entered "Foo, bar" just now. After hitting Apply Configuration that new Note shows up just fine, but the Status in CCM remains "Not Applied".

Code: Select all

% grep -v '^#' /usr/local/nagios/etc/hosts/localhost.cfg

define host {
    host_name    localhost
    use          linux-server
    alias        localhost
    address      127.0.0.1
    contacts     nagiosadmin
    notes        Foo, bar
    register     1
}
Changing it again gives:

Code: Select all

% grep -v '^#' /usr/local/nagios/etc/hosts/localhost.cfg

define host {
    host_name    localhost
    use          linux-server
    alias        localhost
    address      127.0.0.1
    contacts     nagiosadmin
    notes        This was applied!
    register     1
}
...but the UI still shows:
Screen Shot 2018-12-11 at 6.19.51 PM.png

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Wed Dec 12, 2018 12:25 pm
by npolovenko
Hello, @stefanw. Please run the following commands on your nagios server and share the output in the thread:
cat /etc/rc.d/init.d/nagios | grep NagiosRunFile=
cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint.sh | grep lockfile=
cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint_specific.sh | grep lockfile=
cat /usr/local/nagios/etc/nagios.cfg | grep lock

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Wed Dec 12, 2018 1:10 pm
by stefanw
Here you are...

Code: Select all

% cat /etc/rc.d/init.d/nagios | grep NagiosRunFile
NagiosRunFile=/var/run/nagios.lock
                if $NagiosCGIDir/daemonchk.cgi -l $NagiosRunFile > /dev/null 2>&1; then
        if test ! -f $NagiosRunFile; then
                echo "No lock file found in $NagiosRunFile"
        NagiosPID=`head -n 1 $NagiosRunFile`
                if test -f $NagiosRunFile; then
                        NagiosPID=`head -n 1 $NagiosRunFile`
                touch $NagiosRunFile
                        chown $NagiosUser:$NagiosGroup $NagiosRunFile
                # own NagiosRunFile, otherwise a following "start" could
                # new NagiosRunFile, allowing multiple nagios daemons
                rm -f $NagiosStatusFile $NagiosRunFile $NagiosCommandFile
                if test ! -f $NagiosRunFile; then

Code: Select all

% cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint.sh | grep lockfile
lockfile="/var/run/nagios.lock"
sed -i "s|lock_file=.*|lock_file=$lockfile|" /usr/local/nagios/etc/nagios.cfg

Code: Select all

% cat /usr/local/nagiosxi/scripts/nom_restore_nagioscore_checkpoint_specific.sh | grep lockfile
lockfile="/var/run/nagios.lock"
sed -i "s|lock_file=.*|lock_file=$lockfile|" /usr/local/nagios/etc/nagios.cfg

Code: Select all

% cat /usr/local/nagios/etc/nagios.cfg | grep lock
lock_file=/var/run/nagios.lock

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Wed Dec 12, 2018 4:18 pm
by stefanw
The plot thickens!

At some point overnight the issue ("Changes detected!" and a status "Not Applied" even though it was written) went away! I will now go through all the nightly XI cron jobs and see if there's some database operation happening.

However as soon as I made a minor edit to a Host, did Apply Configuration with no errors, and went back to view the Hosts in CCM it's back like before. As like yesterday the change I made was indeed written out to disk in the expected Core config file.

I'll keep investigating, but I'm running out of things to try other than /usr/local/nagiosxi/scripts/restore_defaults.sh which I haven't ruled out.

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Wed Dec 12, 2018 5:40 pm
by ssax
Please go to Configure > Core Config Manager > Tools > Config File Management:
- Click the Delete Files button (don't worry, it's safe, they will be rewritten)
- Then click the Write Configs button
- Then Apply Configuration and see if it goes away

If it doesn't go away, please PM me a copy of your profile, you can download it from Admin > System Profile > Download Profile.

Additionally, please send me the output of these commands as well (please put it in a txt file):

Code: Select all

ls -l /usr/local/nagios/etc
ls -l /usr/local/nagios/etc/hosts
ls -l /usr/local/nagios/etc/services

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Wed Dec 12, 2018 5:41 pm
by npolovenko
@stefanw, Please run the following script as root:
/usr/local/nagiosxi/scripts/repair_databases.sh
If that doesn't work please send in your system profile so that I can review it.
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and send it in a private message.

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Wed Dec 12, 2018 6:37 pm
by stefanw
npolovenko wrote:@stefanw, Please run the following script as root:
/usr/local/nagiosxi/scripts/repair_databases.sh
I have tried the /usr/local/nagiosxi/scripts/repair_databases.sh several times but that doesn't change anything.

I thought I was perhaps onto something when (after you asked about the nagios.lock file) I went groping around for references to it with

Code: Select all

find /etc /usr/local -type f -print0 | xargs -0 grep -i nagios.lock
and I found...
/usr/local/nagiosxi/html/includes/components/ccm/config.inc.php contained two instances of:

Code: Select all

$CFG['lock_file'] = '/usr/local/nagios/var/nagios.lock';
which is a different path than /var/run/nagios.lock everywhere else. Unfortunately after editing that file (both lines) that didn't help either.

ssax wrote:Please go to Configure > Core Config Manager > Tools > Config File Management:
- Click the Delete Files button (don't worry, it's safe, they will be rewritten)
- Then click the Write Configs button
- Then Apply Configuration and see if it goes away
Same there also. Delete / Write / Apply doesn't change anything.

System Profile and the ls -l outputs coming by PM.

Re: CCM stuck on "Changes detected!" after Apply Configurati

Posted: Wed Dec 12, 2018 10:06 pm
by stefanw
FYI @npolovenko and @ssax I sent you both the XI profile in PM since you each asked for it.

Thanks!
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited