Cleanup - Getting rid of old hosts that won't go away

[atremblay]

Hey everyone,

So I've got a strange one that I've looked at off and on for a couple years now, and have just never had the time to really dig deep and find out why to clear out some old hosts which we no longer monitor.

So a couple years ago before the pandemic we essentially split our company into two companies. This split wasn't just on the accountants books, it was a true divorce with teams being made for each department on both sides. In doing so we got the bigger side of the infrastructure and they had only a couple hundred servers, network devices, etc... As part of the post cleanup job the old hosts which we no longer manage were to be cleaned up. So one of our team members went into our Nagios XI server, and just began deleting the Services and Hosts related to the devices we no longer manage. It worked, they were gone, no where to be found anymore. Job done.

It wasn't until months later when I was doing a completely unrelated investigation and discovered a bunch of connections to what were essentially IP Subnets that I knew to not be in our network anymore. It didn't take long as I could see all the connections were all SNMP (ICMP and NCPA seem gone) and they were coming from our Nagios XI server. I tried looking it into then, but after a bit I had to return to my other case.

I came back around to it several times later, usually because I'd again come across it while doing something unrelated. Anyways, this time I figured I'd reach out to the community to ask : How would a Nagios XI server continue to try and monitor SNMP data from machines that are not in their configs anymore?

I have gone through the Nagios configurations looking for traces of those hosts. The only places they seem still to be located are in Archived files (usr/local/nagios/var/archive/), and in some MRTG files (/usr/local/nagiosxi/tmp/). But I can't find any trace of them in host configs or any of the like.

find /usr/local/nagios/etc -name "*.cfg" -print | xargs grep "<IP Address/Hostname>"
find /usr/local/nagios/ -name "*" -print | xargs grep "<IP Address/Hostname>"
find /usr/local/nagiosxi/ -name "*" -print | xargs grep "<IP Address/Hostname>"

Any pointers to where these runts can be hidden. Thanks community.

A

[kfanselow]

Hi atremblay,

Try /etc/mrtg/conf.d/ :

Code: Select all

find /etc/mrtg/conf.d/  -name "*.cfg" -ls 

You should be able to safely remove the cfg files for the hosts no longer in your environment. They are sourced from the main cfg with wildcard: "Include: conf.d/*.cfg", so the removal of the cfg file should stop the queries. If this doesn't resolve your issue please generate a system profile and send it to me via a Private Message so we can take a look.

Login to the Nagios XI GUI using a web browser.
Click the "Admin" (Top) -> "System Profile" Menu (Left)
Click the "Download Profile" button

Thanks and Best Regards,
Keith