ghost check

[Frédéric GRANAT]

Hi,
According to the firewall console, my colleague of the network team reports that Nagios is checking an host using snmp.
Looking in nagiosxi, I cannot see such a host or IP address.
Do you have any explanation for such a phenomenon ?

Regards,

Frederic

[bwallace]

Sometimes we see where a host has been deleted from XI and continues to run as usual, but in your case you can't see it at all in the UI?

Although this doc is for the first scenario I mentioned, I believe it still applies to your situation. Please run through the steps therein and see if that doesn't resolve the issue.
https://support.nagios.com/kb/article.php?id=27

If not, please obtain a screenshot of the FW console or simply the log entry of the Firewall where this check is recorded.

[Frédéric GRANAT]

Hi,
According to the doc https://support.nagios.com/kb/article.php?id=27, I ran ps -ef | head -1 && ps -ef | grep bin/nagios

Code: Select all

[root@nagiosxi hosts]# ps -ef | head -1 && ps -ef | grep bin/nagios
UID        PID  PPID  C STIME TTY          TIME CMD
root      7762   552  0 09:36 pts/0    00:00:00 grep bin/nagios
nagios   22595     1  0 Jul26 ?        02:20:42 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   22601 22595  2 Jul26 ?        19:21:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22602 22595  2 Jul26 ?        19:45:24 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22603 22595  3 Jul26 ?        20:12:54 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22605 22595  2 Jul26 ?        19:38:30 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22606 22595  2 Jul26 ?        19:55:45 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22607 22595  3 Jul26 ?        20:37:12 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   22662 22595  0 Jul26 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

As you can see, no ghost process.

You will find attached the screen capture of the firewall console.

Rgds,

Frederic

[lmiltchev]

Can you find the host by running the following command from the CLI?

Code: Select all

grep -R 192.168.100.14 /usr/local/nagios/etc/hosts/

If you do, check the services on this host.

What is the output of the command below?

Code: Select all

grep -R -i snmp /usr/local/nagios/etc/services

Do you have only one Nagios XI server?

[Frédéric GRANAT]

Hi,
Can you find the host by running the following command from the CLI?
[root@nagiosxi hosts]# grep -R 192.168.100.14 /usr/local/nagios/etc/hosts/
=> No
What is the output of the command below?
=>
[root@nagiosxi hosts]# grep -R -i snmp /usr/local/nagios/etc/services
/usr/local/nagios/etc/services/W2K-AUTOCOM01.cg.ahp.cfg: check_command check_win_service!domcompta/svc_riverbed!dsisvc!Auto!0!0!Journaux et alertes de performance|Protection logicielle|HP AlertService|ROM|Citrix vDisk Update Service|D.*tection mat.*riel noyau|Fournisseur de clich.* instantan.* logiciel Microsoft|clr_optimization|Backup Exec Remote Agent for Windows Systems|Security Center|Operations Manager Audit Collection Service|Emulex HBA Management|Security Center|Security Center|Update Windows|01 service|OpsMgr Health Service|Service d|Service SNMP|Support Boot|Backup Exec Remote Agent for Windows Systems|Service Google Update|Acronis VSS Provider!!
/usr/local/nagios/etc/services/cpu.txt:nagios 9596 1.0 0.3 13568 9400 ? S 15:24 0:00 /usr/bin/perl -w? /usr/local/nagios/libexec/check_wmi_plus.pl -H 192.168.61.6 -u svc_riverbed -p dsisvc -m checkservice -a Auto -w 0 -c 0 -o Journaux et alertes de performance|Protection logicielle|HP AlertService|ROM|Citrix vDisk Update Service|D.*tection mat.*riel noyau|Fournisseur de clich.* instantan.* logiciel Microsoft|clr_optimization|Backup Exec Remote Agent for Windows Systems|Security Center|Operations Manager Audit Collection Service|Emulex HBA Management|Security Center|Security Center|Update Windows|01 service|OpsMgr Health Service|Service d|Service SNMP|Support Boot|Backup Exec Remote Agent for Windows Systems|Service Google Update|Registre .* distance|Programme d.*installation pour les modules Windows|HP Insight Event Notifier|LogMein*|UniVerse Resource Service|UniVerse Telnet Service|LMIGuardianSvc|Gestionnaire de disque logique|Mises .* jour automatiques|Audio Windows|Explorateur d.*ordinateurs|Services de cryptographie|Serveur|Station de travail|Ouverture de session secondaire|Client de suivi de lien distribu.*
/usr/local/nagios/etc/services/cpu.txt:root 3669 0.0 0.2 13836 6640 ? Ss Feb04 0:01 /usr/bin/perl /usr/local/sbin/snmptt --daemon
/usr/local/nagios/etc/services/cpu.txt:root 3670 0.0 0.2 13880 7048 ? Ss Feb04 0:01 /usr/bin/perl /usr/local/sbin/snmptt --daemon
/usr/local/nagios/etc/services/cpu.txt:root 3683 0.0 0.0 16032 2812 ? Ss Feb04 0:00 /usr/sbin/snmptrapd -Lsd -On -p /var/run/snmptrapd.pid

[bwallace]

Thanks, but still a mystery. Assuming the host in question is a switch or router, can you run this command on your Nagios XI server and post the output ? I want to see if there is a lingering config file there for 192.168.100.14 --

ls /etc/mrtg/conf.d/


Also, post a copy (in code wraps please) of the objects.cache file, found in:
/usr/local/nagios/var/objects.cache

[Frédéric GRANAT]

[root@nagiosxi hosts]# ls /etc/mrtg/conf.d/
172.31.0.16.cfg 172.31.2.4.cfg 192.168.32.14.cfg 192.168.37.14.cfg 192.168.42.14.cfg 192.168.51.14.cfg 192.168.63.14.cfg 192.168.80.14.cfg
172.31.0.2.cfg 192.168.100.14.cfg 192.168.34.14.cfg 192.168.38.14.cfg 192.168.43.14.cfg 192.168.60.14.cfg 192.168.64.14.cfg 192.168.82.14.cfg
172.31.0.3.cfg 192.168.105.14.cfg 192.168.35.14.cfg 192.168.39.14.cfg 192.168.45.14.cfg 192.168.61.14.cfg 192.168.66.14.cfg 192.168.97.14.cfg
172.31.0.9.cfg 192.168.30.14.cfg 192.168.36.14.cfg 192.168.41.14.cfg 192.168.46.14.cfg 192.168.62.14.cfg 192.168.70.14.cfg

I want to see if there is a lingering config file there for 192.168.100.14 --
=> It seems to be

Also, post a copy (in code wraps please) of the objects.cache file, found in:
=> I attached it to my reply

[bwallace]

Bingo!
ls /etc/mrtg/conf.d/
172.31.0.16.cfg 172.31.2.4.cfg 192.168.32.14.cfg 192.168.37.14.cfg 192.168.42.14.cfg 192.168.51.14.cfg 192.168.63.14.cfg 192.168.80.14.cfg
172.31.0.2.cfg 192.168.100.14.cfg 192.168.34.14.cfg....

Just go ahead and delete that file and the issue should be resolved. There is also a corresponding .rrd file you can delete to keep things clean, but this is safe to leave in place. It should be located:
/var/lib/mrtg/192.168.100.14.rrd

After deleting the .cfg file monitor your FW for awhile and let us know if this indeed resolves the issue.

While we're at it, take a moment to find out what mrtg version is installed on your XI machine by running the following command:
LANG=C LC_ALL=C /usr/bin/mrtg
The version in the output should be 2.17.4 or greater. If not, I'd recommend upgrading it. This doc will walk you through that process:
https://support.nagios.com/kb/article.php?id=511

[Frédéric GRANAT]

Hi,

Code: Select all

There is also a corresponding .rrd file you can delete to keep things clean, but this is safe to leave in place. It should be located:/var/lib/mrtg/192.168.100.14.rrd

There are 7 files, do I have to remove it ?

Code: Select all

After deleting the .cfg file monitor your FW for awhile and let us know if this indeed resolves the issue.

Do I have to restart nagios before checking ?

Code: Select all

[root@nagiosxi mrtg]# LANG=C LC_ALL=C /usr/bin/mrtg
Usage: mrtg <config-file>

mrtg-2.17.4 - Multi Router Traffic Grapher

It seems to be OK

[bwallace]

- The .rrd files are totally harmless but yeah, go ahead and delete all 192.168.100.14.rrd files.
- No restart required after deleting .cfg