Config change inconsistency - possible ghost object

[Pres-Gas]

I am currently running Nagios XI 5R1.0 (soon to update to the latest), but this has been happening since 2014R2.7.

I have not been able to have users complete removal of services/hosts on an installation recently. I go in as nagiosadmin and see that the CCM wants to apply a configuration and when I do I get the following failure:

NagiosXI5R1.0-CCM-apply-error.png

When I look in the file output from the terminal I don't see this config (it has been removed for a couple of days that I know of). Also, trying the service and host removal php scripts does not find the host/services:

Code: Select all

[root@esnagxitst01 scripts]# cd /usr/local/nagios/etc/hosts/
[root@esnagxitst01 hosts]# ls
estrain01.uits.iu.edu.cfg  estrain03.uits.iu.edu.cfg  estrain05.uits.iu.edu.cfg     localhost.cfg
estrain02.uits.iu.edu.cfg  estrain04.uits.iu.edu.cfg  iu-uits-sacaz.ads.iu.edu.cfg
[root@esnagxitst01 hosts]# cd /usr/local/nagiosxi/scripts      
[root@esnagxitst01 scripts]# ./nagiosql_delete_service.php --config=in-uits-sf2.ads.iu.edu
URL: https://localhost/nagiosxi/includes/components/ccm/
Unable find services in nagiosql database.
Usage: ./nagiosql_delete_service.php [--id=<service id>] [--config=<config_name>]
[root@esnagxitst01 scripts]# ./nagiosql_delete_host.php --host=in-uits-sf2.ads.iu.edu
URL: https://localhost/nagiosxi/includes/components/ccm/
Unable find host in nagiosql database

What can I do to troubleshoot this and clean up the configs or get them consistent?

[Pres-Gas]

I should also note that in "/usr/local/nagiosxi/scripts" is a lock file placed by the CCM interface, "reconfigure_nagios.lock". I tried removing this file in order to manually run "./reconfigure_nagios.sh" in my troubleshooting. This does not seem to impact this ghost object, but the CCM alert to apply configuration goes away.

[tgriep]

Can you run through this procedure to see if it resolves the issue?

Code: Select all

Go to the Core Config Manager
Under "Tools", click "Write Config Files"
Click the "Delete" button then click the "Write" button and then the "Verify" button
If you get any errors, resolve them and click on the "Write", "Verify" until all of the errors are resolved.
Click the Apply Configuration link and click the "Apply Configuration" button

[Pres-Gas]

I attempted those steps with the same failure as the screenshot:

Code: Select all

Error: Invalid max_check_attempts value for host 'in-uits-sf2.ads.iu.edu'
Error: Could not register host (config file '/usr/local/nagios/etc/hosts/in-uits-sf2.ads.iu.edu.cfg', starting on line 16)

I also don't have the host named in the screenshot in the files:

Code: Select all

[root@esnagxitst01 scripts]# ls /usr/local/nagios/etc/hosts
estrain01.uits.iu.edu.cfg  estrain03.uits.iu.edu.cfg  estrain05.uits.iu.edu.cfg     localhost.cfg
estrain02.uits.iu.edu.cfg  estrain04.uits.iu.edu.cfg  iu-uits-sacaz.ads.iu.edu.cfg

So that did not fix.

[tgriep]

There could be a left over process causing it. Can you run the following in a shell as root?

Code: Select all

service nagios stop
killall -9 nagios
service ndo2db restart
service nagios start

Try and Apply the Config and see if there are any errors.

[Pres-Gas]

The error is still showing up after this. Can this be something in the database part?

Note that the CCM does not show the need for applying the configuration if I remove the lock file in "/usr/local/nagiosxi/scripts" and run "./reconfigure_nagios.sh" manually. Nevertheless, the ghost object persists and appears when other configuration changes are made via the web interface.

[tgriep]

Lets repair the database and see if that helps. In a shell as root, run the following.

Code: Select all

cd /usr/local/nagiosxi/scripts
./repair_databases.sh
service nagios stop
service ndo2db restart
service nagios start

Let us know if this helps.

[Pres-Gas]

Code: Select all

Error: Invalid max_check_attempts value for host 'in-uits-sf2.ads.iu.edu'
Error: Could not register host (config file '/usr/local/nagios/etc/hosts/in-uits-sf2.ads.iu.edu.cfg', starting on line 16)

...still exists. I also got the following when I repaired:

Code: Select all

===============
REPAIR COMPLETE
===============
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
Starting ndo2db: done.
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.

A reboot and attempt to repair again worked normally, but the ghost is still around. Should we attempt to find this in the config DB?

[tgriep]

Run the following in a shell as root and upload the /tmp/hosts.txt file.

Code: Select all

echo 'Select * FROM tbl_host;' |mysql -u root -pnagiosxi nagiosql >/tmp/hosts.txt

If you go in to Core Config Manager > Hosts, that host doesn't show up at all?

[Pres-Gas]

I will not upload the hosts, but I did grep for that host mentioned in the error:

Code: Select all

[root@esnagxitst01 scripts]# grep in-uits-sf2 /tmp/hosts.txt
[root@esnagxitst01 scripts]#

So it does not show up on CCM>Hosts nor does it show up in "/usr/local/nagios/etc/hosts/" nor does it show up in the query you had me run.

Thanks,
Robert