CCM stuck on "Changes detected!" after Apply Configuration

[npolovenko]

@stefanw , Please run the following command as sudo to reset the config permissions.

/usr/local/nagiosxi/scripts/reset_config_perms.sh

Then please follow the following article, section on how to fix the sudoers entries.
https://support.nagios.com/kb/article/n ... es-34.html


Finally, run the following commands:

service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start

If the instrcutions above don't work run the following command and upload the support.txt file from the /tmp/ folder.

ls -lR /usr/local/nagios/ > /tmp/support.txt

[stefanw]

Thanks - results & comments below.

Summary: still no change - still seeing the same issue.

/usr/local/nagiosxi/scripts/reset_config_perms.sh

I've done that many times previously (also mentioned in OP), and doing it again didn't change anything.

Then please follow the following article, section on how to fix the sudoers entries.
https://support.nagios.com/kb/article/n ... es-34.html

Done - no change. FYI I had already mentioned looking in /var/log/secure for any sudo failures and there are none.

Here's a tip... after you download the XI install tarball and extract the nagiosxi.sudoers you can easily compare it to what's live in /etc/sudoers with this:

Code: Select all

# grep NAGIOS /tmp/nagiosxi.sudoers | sha256sum
02bbaec27094faad6d4aa9ed633500e2504ca4bc1a00c31b786c9c32e25dad08  -
# grep NAGIOS /etc/sudoers | sha256sum
02bbaec27094faad6d4aa9ed633500e2504ca4bc1a00c31b786c9c32e25dad08  -

That's safer than manipulating the live /etc/sudoers on a running production server.
Also your instructions should include running visudo -c at the end, to make sure there's no typos.

Finally, run the following commands:

Code: Select all

service crond stop
service npcd stop
service nagios stop
service ndo2db stop
pkill -9 -u nagios
for i in $(ipcs -q | grep nagios |awk '{print $2}'); do ipcrm -q $i; done
rm -rf /usr/local/nagiosxi/var/dbmaint.lock
rm -rf /usr/local/nagiosxi/var/event_handler.lock
rm -rf /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
service mysqld restart
service ndo2db start
service nagios start
service npcd start
service crond start

Done - no change. Before performing those steps I did see more than one message queue (there were two with Nagios still running) from ipcs -q but after removing it still no change in the problem condition.

Please note this MySQL is NOT local, it's on an AWS RDS, therefore I did not run the service mysqld restart as you listed. (You should probably add an instruction to those steps so that a novice admin doesn't accidentally start up MySQL on their Nagios server with potentially confusing if not security-related consequences.)

If the instrcutions above don't work run the following command and upload the support.txt file from the /tmp/ folder.

Code: Select all

ls -lR /usr/local/nagios/ > /tmp/support.txt

Since there's still no change I'll PM you that output.

[stefanw]

Also as I mentioned before, I'm willing to try running

Code: Select all

/usr/local/nagiosxi/scripts/restore_defaults.sh

if you can confirm for me that it ONLY sets Core / CCM to defaults. This is a pre-production server that has nothing important in Core configs, but I don't want to have to set up XI all over again.

Related to that, if there are database purges you want me to do that's also an option. There's no Core / monitoring / event history in the database that we need to keep - only the XI operation & config data.

[npolovenko]

@stefanw, Please do not run that command yet. Can you send me the database log file from the offloaded mysql server? Log in to the server and look for one of these logs(depending on if you're running mysql or mariadb).

/var/log/mariadb/mariadb.log
/var/log/mysqld.log

Let's compare time/timezones on your nagios server and the remote mysql server. Run these commands on the Nagios server and show me the output.

checking time / date on systems
grep "date.timezone" /etc/php.ini
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
date
echo "SELECT NOW();" | mysql -u root -pnagiosxi -h nagiosxi-prod-mysql-multiaz.cissnim4uwgq.us-east-1.rds.amazonaws.com
php -r "echo date('r').PHP_EOL;"
strings /etc/localtime | tail -1

[stefanw]

Let's compare time/timezones on your nagios server and the remote mysql server. Run these commands on the Nagios server and show me the output.

Code: Select all

## checking time / date on systems
grep "date.timezone" /etc/php.ini
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
date
echo "SELECT NOW();" | mysql -u root -pnagiosxi -h my-RDS-FQDN
php -r "echo date('r').PHP_EOL;"
strings /etc/localtime | tail -1

I see where you're going, and I see that the MySQL RDS is in UTC (details below), but I'm not aware of that having changed recently. IIRC the RDS has been unchanged since it was originally set up - and this issue with the Apply Configuration not "taking" just started very recently. However, I am going to RTFM about RDS timezone and (assuming it can be changed and I can find out how) I'll do that. Since this XI instance and the database are pre-prod we can make any needed changes immediately.

Also - If there are SQL calls to change timezone please let me know because I'm basically a MySQL novice.

I'll PM you the RDS logs. (FYI the RDS logs are different names than a conventional server MySQL. See attached screen shot for what's typical.)

Here's the time & timezone details:

Code: Select all

% grep "date.timezone" /etc/php.ini
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = US/Eastern
date.timezone = US/Eastern
% ls -l /etc/localtime
lrwxrwxrwx 1 root root 30 Jan 30  2018 /etc/localtime -> /usr/share/zoneinfo/US/Eastern
% php -r 'echo date("D M j G:i:s T Y")."\n";'
Fri Dec 14 10:33:00 EST 2018
% date
Fri Dec 14 10:33:06 EST 2018
% echo "SELECT NOW();" | mysql -u nagios -p -h XXXX.us-east-1.rds.amazonaws.com
Enter password:
NOW()
2018-12-14 15:33:23
% echo "SELECT NOW();" | mysql -u nagiosql -p -h XXXX.us-east-1.rds.amazonaws.com
Enter password: 
NOW()
2018-12-14 15:34:01
% php -r "echo date('r').PHP_EOL;"
Fri, 14 Dec 2018 10:34:07 -0500
% strings /etc/localtime | tail -1
EST5EDT,M3.2.0,M11.1.0

Screen Shot 2018-12-14 at 10.45.51 AM.png

[npolovenko]

@stefanw, Thanks for the log files. I am not familiar with RDS, but on CentOS/RHEL you can set mysql timezone in the /etc/my.cnf config file.

[mysqld]
default-time-zone = '-05:00'

Then restart mysql with:

service mysqld restart

Verify the timezone:

echo "SELECT @@global.time_zone;" | mysql -u root -pnagiosxi

Verify time:

echo "SELECT NOW();" | mysql -u root -pnagiosxi

[stefanw]

Holy @#&% setting the timezone in the RDS might have finally been the solution!!!

I have to do more testing, but now (for the first time in a week or more!) things that are edited are now appearing with "Status: Applied" and the "Changes detected!" is not showing up!

This is how things are now:

Code: Select all

mysql> SELECT @@global.time_zone;
+--------------------+
| @@global.time_zone |
+--------------------+
| US/Eastern         |
+--------------------+
1 row in set (0.00 sec)

mysql> SELECT NOW();
+---------------------+
| NOW()               |
+---------------------+
| 2018-12-14 15:00:30 |
+---------------------+
1 row in set (0.00 sec)

mysql> quit;
Bye
% date
Fri Dec 14 15:00:34 EST 2018

I have NO idea right now if the RDS time zone changed to UTC, or if this problem started because of something else. I'm not sure yet if the RDS Parameter Group has change logs or if they go back far enough. I'll try and find out.

In case it helps, this a a screen shot of the AWS Console where the timezone appears in the RDS Parameter Group configuration.

Screen Shot 2018-12-14 at 3.05.05 PM.png

In the meantime please keep this thread open until mid-week next week. By that time I'll have done thorough testing.

By the way while investigating the RDS, I discovered that AWS RDS only supports backups of InnoDB, not MyISAM.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#Overview.BackupDeviceRestrictions
If you have any docs on the table format - specifically why MyISAM not InnoDB - please let me know. I found the mention "This means that the MYSQL tables are in the wrong format" referring to InnoDB as "wrong" in:
https://assets.nagios.com/downloads/nag ... tabase.pdf
but if I can't ensure a proper RDS backup with MyISAM I may have no choice but to use the "wrong" table type!

I'll search the forum for more on this.

[npolovenko]

@stefanw, Glad that solution worked for you. Sure, we'll keep this open for a while.
As far as converting tables to InnoDB. The latest XI supports InnoDB engine for the nagiosxi and nagiosql databases. Nagios database needs to stay in My-ISAM. Perhaps instead of using AWS backup system, you can set up scheduled automatic backups in Nagios XI(DB snapshot is included in XI backups)?
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

[stefanw]

@stefanw, Glad that solution worked for you. Sure, we'll keep this open for a while.
As far as converting tables to InnoDB. The latest XI supports InnoDB engine for the nagiosxi and nagiosql databases. Nagios database needs to stay in My-ISAM. Perhaps instead of using AWS backup system, you can set up scheduled automatic backups in Nagios XI(DB snapshot is included in XI backups)?
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Thanks again for the help on this one. OK to close / lock this topic now. I haven't been able to get the issue to recur since changing the RDS Timezone.

(If I had more time I would try and figure out if the RDS Timezone changed to UTC, or if something in this XI instance suddenly changed to make the issue show up, but I don't have that opportunity now.)

I believe our design goal for our Nagios XI in AWS (with the MySQL on RDS) was to enable 100% recovery using AWS tools, but I'm not certain of that. I'll dig into that XI backup/restore doc with this new knowledge of the RDS backup restriction.

There may be a sweet spot somewhere in the middle between all-InnoDB with only AWS backup and just using the XI backup. TBD. (The multi-AZ replication of the RDS also has to be factored in.)
At least there are some documented options besides converting everything to InnoDB:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html#Overview.BackupDeviceRestrictions

Again, OK to lock this thread. I'll open a new one as needed for MySQL backup/restore concerns.

Thanks!

[npolovenko]

@stefanw,Sounds good. Yes, it might be worth researching some options in AWS to see if there's any workaround that would allow backing up of InnoDB.
Closing the thread.