Ghost hosts

[awatch]

I have a problem with removed hosts from the nagios xi configuration still existing in the monitored host views. If you search for the host under the view it will find two hosts which I have deleted, however if you just browse the hosts normally they are not present. Nagios is still sending alerts on failed checks for these hosts, I cannot acknowledge or stop notifications on them because they no longer actually exist in the configuration.

Based upon other posts I have tried:

• Resetting nagios core credentials
  Verifying/resetting the /usr/local/nagios/etc/ permissions to 775 with owner as nagios.nagios
  Querying the database to verify that the host definition no longer exists within it. (It also is not within the core config manager)
  Removing all extraneous configuration files (left over from some static configs)
  Removing ALL configuration files (host configs, service configs, etc etc) and rewriting them from the database
  Verified that the host exists nowhere in the flat config files via

  Code: Select all

  grep -rl "hostname" /usr/local/nagios/etc/*

  Deleting my browser's cache.

I am still receiving alerts pertaining to this ghost host! How can I delete this once and for all? I have already missed a real issue due to the swath of alerts from these missing hosts and their respective services. (The services associated with the host no longer exist either.)

What can I do to resolve this?

Thanks,
awatch

[mmestnik]

you missed using the CCM's write config files tool.

http://support.nagios.com/wiki/index.ph ... ger_Issues

[awatch]

Oh, I'm sorry I failed to mention that, I thought it would be assumed I used "write config files" when I said I rewrote the config files from the DB after manually removing all of them.

How I typically update configurations is from the CCM: Write Monitoring Data, Write additional info, check configuration, then I proceed to apply configuration. This does in fact rewrite all of the configuration files, and the host which was removed keeps appearing in the host views. After writing -new- configuration files from the CCM I verify with the methods listed in my previous post. The host is still in the views but in no cfg files...

[mmestnik]

You had a long list of things you tried, so I just thought you had accidentally skipped over step one.

As a note, I don't consider "Deleting my browser's cache." to be equivalent to "holding shift and hitting reload" and I also don't believe that reload is always F5(as it's ctrl-r here). However that's not your problem here, so there is no need to repeat this step today.

"Querying the database", there are several DBs. NagiosQL and ndodb are in MySQL(IIRC) and NagiosXI stores data into Postgress. The ndodb is where all the service information displayed by NagiosXI comes from, were as Nagios Core has it's internal database. These can get out of sync.

Verify that the hosts no-longer exist in the Nagios Core(under the advanced tab) interface. This indelicate that Nagios Core is refusing to reload the config files, ultimately fix the issue it reports in that case. This would have been reported on the Write Configuration Page, let us know if that's not the case and help us discover why. We can't direct ppl to this tool as "the" solution if it doesn't always work.

As stated it's possible for Nagios Core to not inform ndodb of a change. I'm unsure how to debug an issue like that. One problem that had come up was an issue with case changes and adding previously deleted hosts. Any thing like that happen to you? What version of XI are you running? Did you make any changes?

[awatch]

The hosts no longer exist in the Core Config manager, however I can't seem to find the appropriate link to the true Nagios Core interface underneath it all. To which advanced tab are you referring, as there is not one in the basic headers, and I couldn't find one in any of the standard tabs which pointed to Core. I would imagine it wouldn't be as the hosts are not defined in the configuration files currently written.

We have not recently deleted/added a host with a slight name variation, or capitalization variation to my knowledge. We're running version Nagios XI 2009R1.1G. The only changes we've been trying to make are removing these hosts.

Thanks again for the help.

[mmestnik]

Please endeavor to persevere. The advanced tab is there and you will find the link to Nagios Core. Perhaps your Ghost's are different from other hosts. You know historical data for a host should _NOT_ be just discarded on a Delete.

[Box293]

Sometimes I've found a service config file left behind (after deleting a service), and this is referencing the host object that no longer exists.

To fix it I needed to delete the config file that has been left behind. They are located in:

• /usr/local/nagios/etc

Also, look at the failed Configuration Snapshot, the last item down the bottom should be complaining about the object that cannot be found.

This should help you find the problem config file and delete it. After that try apply configuration.


Hope this is of some help.

[mmestnik]

Thank you for bringing this to our attention. This is something we will look into.
http://go.nagios.com/tracker/73

[awatch]

Okay, I've completely scoured the interface... There is only one 'advanced' tab as a header in the left quicklink frame inside the core configuration manager. No where else is there an advanced tab or any reference to the Nagios Core interface.

Screenies20100803105544.jpg

Regardless, I gained access to the core interface by updating my security credentials and copying the nagios core backend password, then navigating to nagiosserver/nagios... And the ghost hosts do not appear in the core interface... Which makes sense because there are no configuration flat files which even dare to mention the hosts I removed. However, the hosts are still in the nagiosxi interface. The ghosts are aw4-26...1 and 2. 3 is still supposed to be there. Here are some screenshots...

Screenies20100803104627.jpg

Screenies20100803104443.jpg

To reiterate...
I have removed the files in /usr/local/nagios/etc which have anything at all to do with these ghost hosts...
I have removed the hosts from the core config manager database
I have rewritten the configuration flat files through the interface.
I have verified that the hosts do not exist in the rewritten config files.
I have reapplied the configuration changes.
The hosts do not initially appear in the host monitoring view. After about five minutes they return.
We are still getting alerts for down services on these removed hosts, I cannot acknowledge the problem as it cannot find the host/service to acknowledge and simply errors.

Now what can I do to fix this?

[Box293]

Thank you for bringing this to our attention. This is something we will look into.

I've noticed that it usually occurs when you disable a service (but not delete it). I've only noticed it occur when I disable a service that has been assigned to a host group.

The real issues may arrise later on when you try and delete or disable the hostgroup that the disabled service is "assigned to".