Page 1 of 2
ghost check
Posted: Mon Aug 22, 2016 6:58 am
by Frédéric GRANAT
Hi,
According to the firewall console, my colleague of the network team reports that Nagios is checking an host using snmp.
Looking in nagiosxi, I cannot see such a host or IP address.
Do you have any explanation for such a phenomenon ?
Regards,
Frederic
Re: ghost check
Posted: Mon Aug 22, 2016 11:13 am
by bwallace
Sometimes we see where a host has been deleted from XI and continues to run as usual, but in your case you can't see it at all in the UI?
Although this doc is for the first scenario I mentioned, I believe it still applies to your situation. Please run through the steps therein and see if that doesn't resolve the issue.
https://support.nagios.com/kb/article.php?id=27
If not, please obtain a screenshot of the FW console or simply the log entry of the Firewall where this check is recorded.
Re: ghost check
Posted: Wed Aug 24, 2016 2:27 am
by Frédéric GRANAT
Hi,
According to the doc
https://support.nagios.com/kb/article.php?id=27, I ran ps -ef | head -1 && ps -ef | grep bin/nagios
Code: Select all
[root@nagiosxi hosts]# ps -ef | head -1 && ps -ef | grep bin/nagios
UID PID PPID C STIME TTY TIME CMD
root 7762 552 0 09:36 pts/0 00:00:00 grep bin/nagios
nagios 22595 1 0 Jul26 ? 02:20:42 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 22601 22595 2 Jul26 ? 19:21:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 22602 22595 2 Jul26 ? 19:45:24 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 22603 22595 3 Jul26 ? 20:12:54 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 22605 22595 2 Jul26 ? 19:38:30 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 22606 22595 2 Jul26 ? 19:55:45 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 22607 22595 3 Jul26 ? 20:37:12 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios 22662 22595 0 Jul26 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
As you can see, no ghost process.
You will find attached the screen capture of the firewall console.
Rgds,
Frederic
Re: ghost check
Posted: Wed Aug 24, 2016 9:24 am
by lmiltchev
Can you find the host by running the following command from the CLI?
Code: Select all
grep -R 192.168.100.14 /usr/local/nagios/etc/hosts/
If you do, check the services on this host.
What is the output of the command below?
Code: Select all
grep -R -i snmp /usr/local/nagios/etc/services
Do you have only one Nagios XI server?
Re: ghost check
Posted: Wed Aug 24, 2016 10:13 am
by Frédéric GRANAT
Hi,
Can you find the host by running the following command from the CLI?
[root@nagiosxi hosts]# grep -R 192.168.100.14 /usr/local/nagios/etc/hosts/
=> No
What is the output of the command below?
=>
[root@nagiosxi hosts]# grep -R -i snmp /usr/local/nagios/etc/services
/usr/local/nagios/etc/services/W2K-AUTOCOM01.cg.ahp.cfg: check_command check_win_service!domcompta/svc_riverbed!dsisvc!Auto!0!0!Journaux et alertes de performance|Protection logicielle|HP AlertService|ROM|Citrix vDisk Update Service|D.*tection mat.*riel noyau|Fournisseur de clich.* instantan.* logiciel Microsoft|clr_optimization|Backup Exec Remote Agent for Windows Systems|Security Center|Operations Manager Audit Collection Service|Emulex HBA Management|Security Center|Security Center|Update Windows|01 service|OpsMgr Health Service|Service d|Service SNMP|Support Boot|Backup Exec Remote Agent for Windows Systems|Service Google Update|Acronis VSS Provider!!
/usr/local/nagios/etc/services/cpu.txt:nagios 9596 1.0 0.3 13568 9400 ? S 15:24 0:00 /usr/bin/perl -w? /usr/local/nagios/libexec/check_wmi_plus.pl -H 192.168.61.6 -u svc_riverbed -p dsisvc -m checkservice -a Auto -w 0 -c 0 -o Journaux et alertes de performance|Protection logicielle|HP AlertService|ROM|Citrix vDisk Update Service|D.*tection mat.*riel noyau|Fournisseur de clich.* instantan.* logiciel Microsoft|clr_optimization|Backup Exec Remote Agent for Windows Systems|Security Center|Operations Manager Audit Collection Service|Emulex HBA Management|Security Center|Security Center|Update Windows|01 service|OpsMgr Health Service|Service d|Service SNMP|Support Boot|Backup Exec Remote Agent for Windows Systems|Service Google Update|Registre .* distance|Programme d.*installation pour les modules Windows|HP Insight Event Notifier|LogMein*|UniVerse Resource Service|UniVerse Telnet Service|LMIGuardianSvc|Gestionnaire de disque logique|Mises .* jour automatiques|Audio Windows|Explorateur d.*ordinateurs|Services de cryptographie|Serveur|Station de travail|Ouverture de session secondaire|Client de suivi de lien distribu.*
/usr/local/nagios/etc/services/cpu.txt:root 3669 0.0 0.2 13836 6640 ? Ss Feb04 0:01 /usr/bin/perl /usr/local/sbin/snmptt --daemon
/usr/local/nagios/etc/services/cpu.txt:root 3670 0.0 0.2 13880 7048 ? Ss Feb04 0:01 /usr/bin/perl /usr/local/sbin/snmptt --daemon
/usr/local/nagios/etc/services/cpu.txt:root 3683 0.0 0.0 16032 2812 ? Ss Feb04 0:00 /usr/sbin/snmptrapd -Lsd -On -p /var/run/snmptrapd.pid
Re: ghost check
Posted: Wed Aug 24, 2016 10:17 am
by bwallace
Thanks, but still a mystery. Assuming the host in question is a switch or router, can you run this command on your Nagios XI server and post the output ? I want to see if there is a lingering config file there for 192.168.100.14 --
ls /etc/mrtg/conf.d/
Also, post a copy (in code wraps please) of the objects.cache file, found in:
/usr/local/nagios/var/objects.cache
Re: ghost check
Posted: Thu Aug 25, 2016 1:26 am
by Frédéric GRANAT
[root@nagiosxi hosts]# ls /etc/mrtg/conf.d/
172.31.0.16.cfg 172.31.2.4.cfg 192.168.32.14.cfg 192.168.37.14.cfg 192.168.42.14.cfg 192.168.51.14.cfg 192.168.63.14.cfg 192.168.80.14.cfg
172.31.0.2.cfg 192.168.100.14.cfg 192.168.34.14.cfg 192.168.38.14.cfg 192.168.43.14.cfg 192.168.60.14.cfg 192.168.64.14.cfg 192.168.82.14.cfg
172.31.0.3.cfg 192.168.105.14.cfg 192.168.35.14.cfg 192.168.39.14.cfg 192.168.45.14.cfg 192.168.61.14.cfg 192.168.66.14.cfg 192.168.97.14.cfg
172.31.0.9.cfg 192.168.30.14.cfg 192.168.36.14.cfg 192.168.41.14.cfg 192.168.46.14.cfg 192.168.62.14.cfg 192.168.70.14.cfg
I want to see if there is a lingering config file there for 192.168.100.14 --
=> It seems to be
Also, post a copy (in code wraps please) of the objects.cache file, found in:
=> I attached it to my reply
Re: ghost check
Posted: Thu Aug 25, 2016 9:35 am
by bwallace
Bingo!
ls /etc/mrtg/conf.d/
172.31.0.16.cfg 172.31.2.4.cfg 192.168.32.14.cfg 192.168.37.14.cfg 192.168.42.14.cfg 192.168.51.14.cfg 192.168.63.14.cfg 192.168.80.14.cfg
172.31.0.2.cfg
192.168.100.14.cfg 192.168.34.14.cfg....
Just go ahead and delete that file and the issue should be resolved. There is also a corresponding .rrd file you can delete to keep things clean, but this is safe to leave in place. It should be located:
/var/lib/mrtg/192.168.100.14.rrd
After deleting the .cfg file monitor your FW for awhile and let us know if this indeed resolves the issue.
While we're at it, take a moment to find out what mrtg version is installed on your XI machine by running the following command:
LANG=C LC_ALL=C /usr/bin/mrtg
The version in the output should be 2.17.4 or greater. If not, I'd recommend upgrading it. This doc will walk you through that process:
https://support.nagios.com/kb/article.php?id=511
Re: ghost check
Posted: Thu Aug 25, 2016 9:50 am
by Frédéric GRANAT
Hi,
Code: Select all
There is also a corresponding .rrd file you can delete to keep things clean, but this is safe to leave in place. It should be located:/var/lib/mrtg/192.168.100.14.rrd
There are 7 files, do I have to remove it ?
Code: Select all
After deleting the .cfg file monitor your FW for awhile and let us know if this indeed resolves the issue.
Do I have to restart nagios before checking ?
Code: Select all
[root@nagiosxi mrtg]# LANG=C LC_ALL=C /usr/bin/mrtg
Usage: mrtg <config-file>
mrtg-2.17.4 - Multi Router Traffic Grapher
It seems to be OK
Re: ghost check
Posted: Thu Aug 25, 2016 9:55 am
by bwallace
- The .rrd files are totally harmless but yeah, go ahead and delete all 192.168.100.14.rrd files.
- No restart required after deleting .cfg