Page 1 of 1

Nagios ghost services

Posted: Mon Feb 04, 2019 9:08 am
by hak5
Hi Y'all,

After updating from 5.5.8 to 5.5.9, I noticed that several websites that we're monitoring suddenly started monitoring for CPU and Memory information. None of the services appeared in services under CCM and do not appear/usr/local/nagios/etc/services .cfg files.

#
# Services configuration file
#
# Created by: Nagios CCM 2.7.4
# Date: 2019-01-30 12:32:28
# Version: Nagios Core 4.x
#
# --- DO NOT EDIT THIS FILE BY HAND ---
# Nagios CCM will overwrite all manual settings during the next update if you
# would like to edit files manually, place them in the 'static' directory or
# import your configs into the CCM by placing them in the 'import' directory.
#
###############################################################################

define service {
host_name website
service_description DNS IP Match
use xiwizard_website_dnsip_service
check_command check_xi_service_dns!-a 192.168..x.x!!!!!!!
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
_xiwizard website
register 1
}

define service {
host_name website
service_description DNS Resolution
use xiwizard_website_dns_service
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
_xiwizard website
register 1
}

###############################################################################
#
# Services configuration file
#
# END OF FILE
#
###############################################################################
[root@nagios services]#

I restarted nagios and the ndo2db server hoping that it would resolve the issue. Using systemctl restart nagios and systemctl restart ndo2db which did nothing. After doing so research i found files were in the following directory /usr/local/nagios/share/perfdata which holds temporary files used for graphic performance data from services. After removing the services from that directory, i noted that the services were still showing up in the GUI. After restarting nagios, i then noticed that those same files that i removed from perfdata started to appear again.

[root@nagios perfdata]# ll website/
total 3032
-rw-rw-r--. 1 nagios nagios 384952 Feb 4 08:30 CPU_Usage.rrd
-rw-rw-r--. 1 nagios nagios 2208 Feb 4 08:30 CPU_Usage.xml
-rw-rw-r--. 1 nagios nagios 384952 Feb 4 08:32 DNS_IP_Match.rrd
-rw-rw-r--. 1 nagios nagios 2213 Feb 4 08:32 DNS_IP_Match.xml
-rw-rw-r--. 1 nagios nagios 384952 Feb 4 08:33 DNS_Resolution.rrd
-rw-rw-r--. 1 nagios nagios 2175 Feb 4 08:33 DNS_Resolution.xml
-rw-rw-r--. 1 nagios nagios 384952 Feb 4 08:32 Memory_Usage.rrd
-rw-rw-r--. 1 nagios nagios 2361 Feb 4 08:32 Memory_Usage.xml
-rw-rw-r--. 1 nagios nagios 768224 Dec 27 12:48 SSL_Certificate.rrd
-rw-rw-r--. 1 nagios nagios 2979 Dec 27 12:48 SSL_Certificate.xml
-rw-rw-r--. 1 nagios nagios 768224 Feb 4 08:31 _HOST_.rrd
-rw-rw-r--. 1 nagios nagios 2811 Feb 4 08:31 _HOST_.xml
[root@nagios perfdata]#

Now, here is where things get weird. We're having the same issue with one of the routers that we monitor. We only ping the routers because we use another monitoring tool for our routers. This host also has CPU and Memory monitoring but they do not show up under perfdata/router-colo/ directory. on top of that, there is nothing in the /usr/local/nagios/etc/services .cfg filethat says is supposed to monitor for those services.
[root@nagios router-colo]# ll
total 3012
-rw-rw-r--. 1 nagios nagios 1534768 Feb 4 08:34 Ping.rrd
-rw-rw-r--. 1 nagios nagios 4133 Feb 4 08:34 Ping.xml
-rw-rw-r--. 1 nagios nagios 1534768 Feb 4 08:36 _HOST_.rrd
-rw-rw-r--. 1 nagios nagios 4020 Feb 4 08:36 _HOST_.xml
[root@nagios router-colo]#


[root@nagios services]# cat router-colo.cfg
###############################################################################
#
# Services configuration file
#
# Created by: Nagios CCM 2.7.4
# Date: 2019-01-30 14:15:53
# Version: Nagios Core 4.x
#
# --- DO NOT EDIT THIS FILE BY HAND ---
# Nagios CCM will overwrite all manual settings during the next update if you
# would like to edit files manually, place them in the 'static' directory or
# import your configs into the CCM by placing them in the 'import' directory.
#
###############################################################################

define service {
host_name router-colo
service_description Ping
use xiwizard_genericnetdevice_ping_service
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts admin,nagiosadmin
_xiwizard genericnetdevice
register 1
}

###############################################################################
#
# Services configuration file
#
# END OF FILE
#
###############################################################################
[root@nagios services]#

Im at a complete loss here.

Re: Nagios ghost services

Posted: Mon Feb 04, 2019 9:11 am
by scottwilkerson
If I had to guess, these website hosts are members of a hostgroup.

Someone added this hostgroup to one of the "CPU Usage" and services being "Memory Usage" services being monitored

Re: Nagios ghost services

Posted: Mon Feb 04, 2019 9:32 am
by hak5
scottwilkerson wrote:If I had to guess, these website hosts are members of a hostgroup.

Someone added this hostgroup to one of the "CPU Usage" and services being "Memory Usage" services being monitored
Hi scottwilkerson,

I tried that and it didnt work. Now, the ghost services are no longer there.

Re: Nagios ghost services

Posted: Mon Feb 04, 2019 9:42 am
by scottwilkerson
hak5 wrote: I tried that and it didnt work. Now, the ghost services are no longer there.
It didn't work but the ghost services are no longer there?

Is there any remaining issue?

Re: Nagios ghost services

Posted: Mon Feb 04, 2019 9:51 am
by hak5
The ghost services just stopped showing up. I guess we can close this, ill just open up a new issue should it happen again.

Re: Nagios ghost services

Posted: Mon Feb 04, 2019 9:55 am
by scottwilkerson
hak5 wrote:The ghost services just stopped showing up. I guess we can close this, ill just open up a new issue should it happen again.
Sounds good

Locking thread