Pending Checks on each restart

[kotterbein]

RHEL 5 host, Nagios XI 2012R1.6

I came in today and there was some instability in the system due to work done over the weekend.

I rebooted the server to get everything back in line, and on the server coming back up and encountered the following issues:

1. I was encountering instability, so I ran repairmysql.sh nagios

2. on reboot, old templates for nagiosxi_passive_host were applied to all the hosts I have passive checks running on, and I had to go in and manually remove all of them from the hosts, as it was preventing host-checks from occuring.

3. After this, I noticed service checks had hundreds pending. after looking through them, I have noticed two things:
-services show no history, and do not keep their state, so whenever I apply a configuration, or restart the nagios process, it shows most services to be

Code: Select all

Pending 	4892d 11h 9m 50s 	0/0 	1999-11-30 00:00:00 	No check results for service yet...

- only some of the pending checks are actually checks still in force. I noticed several services I had deleted from nodes were now present on hosts again

I am sure this is a database issue, and I am curious to run the database repair again, but do not want to have to go back around to replace all of the manually moved templates and configurations. I believe nagios should maintain state during a configuration apply, and not default back to the pending with last check 1999-11-30- this is behavior I have never seen prior to coming in this morning.

Anyone have experience with this?

[lmiltchev]

What kind of changes have you made on the system prior to this weekend? What do you mean by "instability in the system"? Do you have a good backup that you can revert to?

[kotterbein]

When I got in, the last check was shown to be 4/20 @13:20 EST.

We had some changes in our environment, but all to end-points that are not effected in this problem, as their checks are being completed well.

I have a nightly backup running that I could revert to, but am trying to avoid this if at all possible. Just did a simple remove of a service, and yet the services still remain as part of the pending checks, but are no longer part of the services configuration under the server-

[scottwilkerson]

did you apply configuration after removing the service?

[kotterbein]

Yes-

Seems I'm looking at close to the same issue I had before that we worked on with Services not showing up under the hosts. I think I'm going to arrange to rebuild this server, as it looks like we are getting into more and more issues as we go along. issuing a reboot command to try and get things squared away again.

[slansing]

Let us know if you are going to re-build and import or if you would rather continue on this path. We are more than happy to assist with either.

[kotterbein]

I'm going to have to do a rebuild and import, but this will have to be scheduled. mean-time I need to get this instance back up and running. looks like this is now the same exact problem I had with http://support.nagios.com/forum/viewtop ... =16&t=9868

Services are no longer tied to the hosts. several reboots and restarts later I'm in the same spot.

[kotterbein]

I am now doing a full backup to restore on a secondary node which I will be upgrading to 1.8. after that I will be able to upgrade the production instance. only issue is that I have a lot of NRDS checks assigned to hit the production node that I will have to pipe back to a secondary node- using puppet it's not a huge deal, but getting the change in place will be, so it will take a couple of days for approvals. man time I'm trying to get everything running smoothly as possible on the current prod instance.

[scottwilkerson]

Let us know the this comes back up after the rebuild