Phantom Data - *Queue Scary Music*

[CGraham]

This close to Halloween, we need a phantom data issue and I'm just the person bring it.

When you are in the CCM and click the "BANG" information icon for a configuration item it shows you all the things that item is connected to (listed at the bottom of the page). I have a number of items that have blank entries in the list.

It reads like this:
......
Relation to tbl_host, entry ATL02PRDAPP276 - deletion possible
Relation to tbl_host, entry ATL02UATAPP305 - deletion possible
Relation to tbl_host, entry - deletion possible <--- the host table has a phantom entry here
Relation to tbl_host, entry ATL02PRDAPP115 - deletion possible
Relation to tbl_host, entry ATL02PRDAPP151 - deletion possible
Relation to tbl_host, entry - deletion possible <--- the host table has a phantom entry here
Relation to tbl_host, entry ATL02PRDAPP07 - deletion possible
Relation to tbl_host, entry ATL02PRDAPP104 - deletion possible
Relation to tbl_host, entry ATL02PRDAPP88 - deletion possible
.......

We're under a big monitoring push so we've had as many as 3 people adding, changing, and screwing up our Nagios at the same time. Hints on where to start for finding and clearing the data?

[mguthrie]

You can get these particularly when importing configs. The way the importer works is that it will look for the related object defined in the config, if it doesn't find it, it will create a placeholder entry in the database. However, if that placeholder never gets populated in the config you'll get a ghost entry. You should see the blank entries on the hosts page in the CCM, and you can look at the Info (icon) output to see where it's relationships have been assigned.

[CGraham]

Just searched my hosts list, no ghosts there...

I didn't cross the streams either, FYI.

[mguthrie]

If there aren't actual ghost host entries, then the actual probably should be fairly minimal. The CCM has a whole bunch of DB tables like lnkHostToHost, which just link a parent-child hosts together. I'm not quite sure what would cause the scenario of there being empty connections in that table, maybe direct sql deletion of hosts?? Have you had any issues Applying Configuration?

[CGraham]

Yes, sometimes you have to apply the configuration more than once for it to work.

other times the configuration will appear to apply but changes won't be made.

We don't manipulate the sql directly.

[mguthrie]

sometimes you have to apply the configuration more than once for it to work.

Does it bail because of a configuration error, or does it "appear" to apply, but nothing happens?

Do you get any revealing information from the /var/log/httpd/error_log when you Apply Configuration?

Is your PHP timezone set to match your system time? (check the Admin->System Profile page to verify).

[CGraham]

Does it bail because of a configuration error, or does it "appear" to apply, but nothing happens?

Both. Sometimes it fails, and I re-apply and it succeeds. Then recently, I disabled some hosts in the config and applied successfully but the hosts weren't disabled. I ended up having to delete them.

Do you get any revealing information from the /var/log/httpd/error_log when you Apply Configuration?

Applied the config while tailing the logs. Nothing happened. Scrolled up a bit and saw this:
[Mon Oct 08 17:18:05 2012] [error] [client 10.1.211.57] PHP Notice: Undefined variable: services_event_handler_disabled in /usr/local/nagiosxi/html/includes/components/xicore/ajaxhelpers-tac.inc.php on line 723, referer: http://nagios-savvis/nagiosxi/includes/ ... re/tac.php

Not sure if that applies.

Is your PHP timezone set to match your system time? (check the Admin->System Profile page to verify).

Yup. Timezones look good now:

PHP Timezone: America/New_York
PHP Time: Wed, 10 Oct 2012 16:08:22 -0400
System Time: Wed, 10 Oct 2012 16:08:22 -0400

But I did have this issue previously...

[mguthrie]

There is a sporadic bug in the 2011 version of the CCM where hosts can be disabled from the CCM without having the file removed. Does your team utilize the disable/enable functionality for hosts in the CCM quite a bit?

We've corrected for this issue in the new CCM that exists in the 2012 release. I can probably produce the component zip for the latest version if you're wanting to try it with your 2011 release.

[CGraham]

We disable quite a bit since we are adding the hosts in Nagios prior to the servers and services being stood up. If you have something for me to try I will try it.

[mguthrie]

Here's a late beta version of the new CCM component. The current trunk is set up for the 2012 release so I'm not sure what it will do on a 2011 install right now. This can be installed through the Admin->Manage Components page. This should correct for the disabling objects bug that fails to delete the files, or at the very least it will tell when when it fails and keep the item in the DB.
http://assets.nagios.com/downloads/exch ... cm/ccm.zip

It also has more handy user feedback about DB relationships, and it shows you the DB entry id's for each listing if you need to trace something.