Page 1 of 2
Config change inconsistency - possible ghost object
Posted: Thu Oct 15, 2015 11:43 am
by Pres-Gas
I am currently running Nagios XI 5R1.0 (soon to update to the latest), but this has been happening since 2014R2.7.
I have not been able to have users complete removal of services/hosts on an installation recently. I go in as nagiosadmin and see that the CCM wants to apply a configuration and when I do I get the following failure:
NagiosXI5R1.0-CCM-apply-error.png
When I look in the file output from the terminal I don't see this config (it has been removed for a couple of days that I know of). Also, trying the service and host removal php scripts does not find the host/services:
Code: Select all
[root@esnagxitst01 scripts]# cd /usr/local/nagios/etc/hosts/
[root@esnagxitst01 hosts]# ls
estrain01.uits.iu.edu.cfg estrain03.uits.iu.edu.cfg estrain05.uits.iu.edu.cfg localhost.cfg
estrain02.uits.iu.edu.cfg estrain04.uits.iu.edu.cfg iu-uits-sacaz.ads.iu.edu.cfg
[root@esnagxitst01 hosts]# cd /usr/local/nagiosxi/scripts
[root@esnagxitst01 scripts]# ./nagiosql_delete_service.php --config=in-uits-sf2.ads.iu.edu
URL: https://localhost/nagiosxi/includes/components/ccm/
Unable find services in nagiosql database.
Usage: ./nagiosql_delete_service.php [--id=<service id>] [--config=<config_name>]
[root@esnagxitst01 scripts]# ./nagiosql_delete_host.php --host=in-uits-sf2.ads.iu.edu
URL: https://localhost/nagiosxi/includes/components/ccm/
Unable find host in nagiosql database
What can I do to troubleshoot this and clean up the configs or get them consistent?
Re: Config change inconsistency - possible ghost object
Posted: Thu Oct 15, 2015 12:01 pm
by Pres-Gas
I should also note that in "/usr/local/nagiosxi/scripts" is a lock file placed by the CCM interface, "reconfigure_nagios.lock". I tried removing this file in order to manually run "./reconfigure_nagios.sh" in my troubleshooting. This does not seem to impact this ghost object, but the CCM alert to apply configuration goes away.
Re: Config change inconsistency - possible ghost object
Posted: Thu Oct 15, 2015 1:55 pm
by tgriep
Can you run through this procedure to see if it resolves the issue?
Code: Select all
Go to the Core Config Manager
Under "Tools", click "Write Config Files"
Click the "Delete" button then click the "Write" button and then the "Verify" button
If you get any errors, resolve them and click on the "Write", "Verify" until all of the errors are resolved.
Click the Apply Configuration link and click the "Apply Configuration" button
Re: Config change inconsistency - possible ghost object
Posted: Thu Oct 15, 2015 2:44 pm
by Pres-Gas
I attempted those steps with the same failure as the screenshot:
Code: Select all
Error: Invalid max_check_attempts value for host 'in-uits-sf2.ads.iu.edu'
Error: Could not register host (config file '/usr/local/nagios/etc/hosts/in-uits-sf2.ads.iu.edu.cfg', starting on line 16)
I also don't have the host named in the screenshot in the files:
Code: Select all
[root@esnagxitst01 scripts]# ls /usr/local/nagios/etc/hosts
estrain01.uits.iu.edu.cfg estrain03.uits.iu.edu.cfg estrain05.uits.iu.edu.cfg localhost.cfg
estrain02.uits.iu.edu.cfg estrain04.uits.iu.edu.cfg iu-uits-sacaz.ads.iu.edu.cfg
So that did not fix.
Re: Config change inconsistency - possible ghost object
Posted: Thu Oct 15, 2015 5:17 pm
by tgriep
There could be a left over process causing it. Can you run the following in a shell as root?
Code: Select all
service nagios stop
killall -9 nagios
service ndo2db restart
service nagios start
Try and Apply the Config and see if there are any errors.
Re: Config change inconsistency - possible ghost object
Posted: Fri Oct 16, 2015 8:48 am
by Pres-Gas
The error is still showing up after this. Can this be something in the database part?
Note that the CCM does not show the need for applying the configuration if I remove the lock file in "/usr/local/nagiosxi/scripts" and run "./reconfigure_nagios.sh" manually. Nevertheless, the ghost object persists and appears when other configuration changes are made via the web interface.
Re: Config change inconsistency - possible ghost object
Posted: Fri Oct 16, 2015 11:19 am
by tgriep
Lets repair the database and see if that helps. In a shell as root, run the following.
Code: Select all
cd /usr/local/nagiosxi/scripts
./repair_databases.sh
service nagios stop
service ndo2db restart
service nagios start
Let us know if this helps.
Re: Config change inconsistency - possible ghost object
Posted: Fri Oct 16, 2015 11:37 am
by Pres-Gas
Code: Select all
Error: Invalid max_check_attempts value for host 'in-uits-sf2.ads.iu.edu'
Error: Could not register host (config file '/usr/local/nagios/etc/hosts/in-uits-sf2.ads.iu.edu.cfg', starting on line 16)
...still exists. I also got the following when I repaired:
Code: Select all
===============
REPAIR COMPLETE
===============
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
A reboot and attempt to repair again worked normally, but the ghost is still around. Should we attempt to find this in the config DB?
Re: Config change inconsistency - possible ghost object
Posted: Fri Oct 16, 2015 1:11 pm
by tgriep
Run the following in a shell as root and upload the /tmp/hosts.txt file.
Code: Select all
echo 'Select * FROM tbl_host;' |mysql -u root -pnagiosxi nagiosql >/tmp/hosts.txt
If you go in to Core Config Manager > Hosts, that host doesn't show up at all?
Re: Config change inconsistency - possible ghost object
Posted: Fri Oct 16, 2015 1:31 pm
by Pres-Gas
I will not upload the hosts, but I did grep for that host mentioned in the error:
Code: Select all
[root@esnagxitst01 scripts]# grep in-uits-sf2 /tmp/hosts.txt
[root@esnagxitst01 scripts]#
So it does not show up on CCM>Hosts nor does it show up in "/usr/local/nagios/etc/hosts/" nor does it show up in the query you had me run.
Thanks,
Robert