Configuration files not updating

[drug]

Periodically, after making changes to services within Nagios XI (5.2.3) I am notified that the database has been updated, but the red notification about requiring "Apply Configuration" does not appear. If I do click on Apply Configuration it reports "Configuration applied successfully." with no errors. I have verified that the changes are not written to the respective services configuration file. The directory and file permissions also look sane to me but I ran "/usr/local/nagiosxi/scripts/reset_config_perms" just in case.

I am able to Write Configs To File without receiving any errors but despite this the configuration file is not updated. Is there anything else I can try?

Thanks.

[hsmith]

How often would you say that you are experiencing this issue?

Is the server under a very heavy load?

[drug]

It's not being used very often since we're still setting it up during our trial period. There are presently only about 8 hosts and 30 services configured. The server has been using very little resources.

[tgriep]

Can you run the following command as root on the XI server, click on Apply Config and post the output from the tail command?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

Also can you run this and post the output too?

Code: Select all

ps -ef

Thanks

[drug]

Please find the (sanitized) output attached.

[drug]

I should also note that the "Last Check" timestamp for one of the service checks in question is 6 years old. I have verified that the check was just recently executed. I've stopped nagios, removed the object cache files and retention.dat but the timestamp is still years behind (presently "2000-04-07 21:38:04").

[hsmith]

What is the output of the following commands?

Code: Select all

date
php -r "echo date('r').PHP_EOL;"
mysql -unagiosxi -pn@gweb -e "SELECT NOW();"

[drug]

Our database actually resides on a dedicated system but the results all return a year of 2016:

Wed Jan 6 13:29:31 EST 2016
Wed, 06 Jan 2016 18:29:40 +0000
2016-01-06 13:28:53

I should clarify that both systems are synced with NTP. The minute/second discrepancies appear because I executed these commands at different times in a different order.

[ssax]

Are you seeing any errors (crashed tables, etc) in your offloaded DBs msyql log? What about on your XI server in /var/lib/pgsql/data/pg_log/postgresql-DAY.log.

Are you able to see the changes after saving in the CCM?

Please post the output of these commands:

Code: Select all

sestatus
chage -l nagios
df -h
df -i
grep nag /etc/group
grep "User \|Group " /etc/httpd/conf/httpd.conf
ls -ld /usr/local/nagios/etc
ls -l /usr/local/nagios/etc
ls -l /usr/local/nagios/etc/your/hosts/or/service/config/file/that/isn't/updating

[drug]

I do see some Warnings for MySQL but they appear to only relate to replication (we have a read-only slave not used directly by our XI system):

Unsafe statement written to the binary log using statement format since BINLOG_FORMAT

I will investigate and remedy these.

I should also note that I was able to make the same change to the affected service today and the red warning appeared indicating I needed to Apply Configuration, which I did and the update was written to the file. Undoing the change did not result in the warning and an Apply Configuration did not write the "undone" change to the file. I.e. it seems if enough time passes the data will eventually be written to a file. Could there be a cron job that's purging or cleaning something?

Meanwhile, I've attached the output from those commands as well as the most recent errors from pg_log.