Ghost services - not in nagios config files or nagiosql db, but in nagios db

[btmikkelsen]

We have about 1500 services in pending with 'No check results for service yet...' I've deleted retenion.dat, objI dects.cache, and status.dat, applied configuration multiple times, but they keep coming back. I noticed they were in the nagios nagios_services, and I did a
MariaDB [nagios]> delete from nagios_services where check_command_args like '%No data%';
Query OK, 2537 rows affected (0.05 sec)
, applied configuration, and still they came back. I've also repaired all database files.

also - these appear in nagios core, also.

Any advice how to remove these?

[kg2857]

NXI runs on core by writing the config to the DB then writing core config files from the info in the DB. Thus, if NXI is saying you have services, they're coming from core, and config files have to exist.
I'm not sure manually deleting things from the DB is a great idea.
I'd probably look at one service and figure out why it isn't working, than try to extrapolate to others rather than look at things as a whole.

[btmikkelsen]

Thank you. That was my process - i focused in on one of the services - which is is duplicated 4x as these 'ghost' services. there is no record in the nagiosql db, nor any reference in the written .cfg files in /usr/local/nagios/etc/services. There are no dangling references in servicegroups, or any other gotchas.

[kg2857]

Once the config is applied in CCM, nagios runs off the config files so any services you see in nagios have to come from config files.
Could they be in the static dir?

[btmikkelsen]

That's exactly what I'd have thought. but I've removed all config files, rewritten, republished. the services are in the running nagios, but not in any config file. I've gone so far as to delete every .cfg, truncate tables in the mysql naigos database, then rewrite everything, and the services still come back.

service npcd stop
service nagios stop
service crond stop
/usr/local/nagiosxi/scripts/repair_databases.sh
echo 'truncate nagios_hoststatus; truncate nagios_hosts; truncate nagios_services; truncate nagios_servicestatus;' |mysql -u root -pnagiosxi nagios
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/nagios.lock
rm -f /var/lib/mrtg/mrtg_l
rm -f /usr/local/nagiosxi/var/*.lock
rm -f /usr/local/nagiosxi/tmp/*.lock
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
service httpd restart
service nagios start
service npcd start