Deleted services remain

[seaward1983]

RHEL 6
Nagios XI (5.6.6)

I tried deleting a BUNCH of services, and applied.
Under CCM Monitoring Services - it shows correctly 24 services.
But on the home page - Services status shows 404 services.
I've "applied" the config a number of times, restarted Nagios XI
What's next?

[cdienger]

An attempt to rewrite the configs. Under Configure > Core Config Manager > Tools > Config File Management, click:

-Delete Files
-Write Configs
-Verfiy Files
-Restart Nagios Core

And then apply the configuration again and check the the home page

[seaward1983]

Nope... didn't work. Somewhere it's stuck or still being reported. When I told it to write the configs - it saw the number of services correctly, but when it verified - it still sees 404...

Checking objects...
Checked 404 services.
Checked 384 hosts.
Checked 21 host groups.
Checked 2 service groups.
Checked 20 contacts.
Checked 4 contact groups.
Checked 134 commands.
Checked 27 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 384 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 27 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

[cdienger]

Please PM me a profile from Admin > System Config > System Profile > Download Profile.

[seaward1983]

I think I sent it to you PM

[seaward1983]

For those keeping score - I've tried the following:

/usr/local/nagiosxi/scripts/repair_databases.sh
nagios database repair succeeded
#no change in the webgui - still see 404 services
#no change in the /services directory - the expected 4 cfg files are intact but no add'l ones

Stopped/restarted nagios

service nagios stop
service npcd stop
service ndo2db stop
service mysqld stop
service httpd stop
service crond stop
service crond start
service httpd start
service mysqld start
service ndo2db start
service npcd start
service nagios start

#no change in the webgui - still see 404 services
#no change in the /services directory - the expected 4 cfg files are intact but no add'l ones

grep -i query /var/log/messages

#I am seeing a number of errors

Error: mysql_query() failed for 'UPDATE nagios_conninfo SET disconnect_time=NOW(), last_checkin_time=NOW(), data_end_time=FROM_UNIXTIME(0), bytes_processed='0', lines_processed='0', entries_processed='0' WHERE conninfo_id='0''

Error: mysql_query() failed for 'INSERT INTO nagios_hoststatus SET instance_id='1', host_object_id='1610', status_update_time=FROM_UNIXTIME(1567523541), output='OK - 172\.21\.1\.26: rta 0\.430ms, lost 0%', long_output='', perfdata='rta=0\.430ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.572ms;;;; rtmin=0\.383ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1567523541), next_check=FROM_UNIXTIME(1567523841), check_type='0', last_state_change=FROM_UNIXTIME(1567171529), last_hard_state_change=FROM_UNIXTIME(1567171529), last_hard_state='0', last_time_up=FROM_UNIXTIME(1567523541), last_time_down=FROM_UNIXTIME(0), last_time_unreachable=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='0.000000', execution_time='0.003790', scheduled_downtime_depth='0', failure_prediction_enabled='0', process_performance_data='1', obsess_over_host='1', modified_host_attributes='0', event_handler='', check_command='check-host-alive!!!!!!!!', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='146' ON DUPLICATE KEY UPDATE instance_id='1', host_object_id='1610', status_update_time=FROM_UNIXTIME(1567523541), output='OK - 172\.21\.1\.26: rta 0\.430ms, lost 0%', long_output='', perfdata='rta=0\.430ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.572ms;;;; rtmin=0\.383ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1567523541), next_check=FROM_UNIXTIME(1567523841), check_type='0', last_state_change=FROM_

[seaward1983]

Based upon the error - I ran

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<hoststatus>
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

Since it's in services that I'm seeing the issue - I also ran the above for servicestatus.

No change in the services in the GUI - although in the servicestatus table - it's showing 404 services, versus the reduced number that should be there (I counted 24, within the 4 cfg files)

I ran for the following errors

nagios_hoststatus
nagios_servicestatus
nagios_conninfo
nagios_systemcommands

I even ran

mysqlcheck -f -r -u root -p --all-databases

No change, but here's the latest in the messages

Error: mysql_query() failed for 'INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='1779', status_update_time=FROM_UNIXTIME(1567527911), output='PING OK - Packet loss = 0%, RTA = 2\.83 ms', long_output='', perfdata='rta=2\.831000ms;100\.000000;500\.000000;0\.000000 pl=0%;20;60;0', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='4', last_check=FROM_UNIXTIME(1567527907), next_check=FROM_UNIXTIME(1567528207), check_type='0', last_state_change=FROM_UNIXTIME(1567171442), last_hard_state_change=FROM_UNIXTIME(1567171442), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1567527907), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='0.000050', execution_time='4.013750', scheduled_downtime_depth='0', failure_prediction_enabled='0', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_ping!100\.0,20%!500\.0,60%!!!!!!', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='146' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='1779', status_update_time=FROM_UNIXTIME(1567527911), output='PING OK - Packet loss = 0%, RTA = 2\.83 ms', long_output='', perfdata='rta=2\.831000ms;100\.000000;500\.000000;0\.000000 pl=0%;20;60;0', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='4', last_check=FROM_UNIXTIME(1567527907), next_check=FROM_UNIXTIME(1567528207), check_type='0', last_sta


I suspect I'm going to have to manually remove them from the servicestatus table? Or ???

[cdienger]

Edit the PING service found in the CCM and click Manage Hosts and/or the Manage Host Groups button to review the hosts and/or host groups this service applies to. Right now this single service accounts for 382 of the services seen on the Home page since it is being applied to multiple hosts. Removing hosts and/or host groups here will reduce the number seen on the home page.

You can also try disabling the PING service altogther since having a PING service is usually redundant - the host check will usually use PING to determine if a host is up or down.

[seaward1983]

I had already deleted the PING service I had set up, so there was no where to go, to remove all the host. That being said - I deleted the only host that lingered, and that removed it. Weird.

This can be closed.