Nagios Support Forum

RHEL 5 host, Nagios XI 2012R1.6

I came in today and there was some instability in the system due to work done over the weekend.

I rebooted the server to get everything back in line, and on the server coming back up and encountered the following issues:

1. I was encountering instability, so I ran repairmysql.sh nagios

2. on reboot, old templates for nagiosxi_passive_host were applied to all the hosts I have passive checks running on, and I had to go in and manually remove all of them from the hosts, as it was preventing host-checks from occuring.

3. After this, I noticed service checks had hundreds pending. after looking through them, I have noticed two things:
-services show no history, and do not keep their state, so whenever I apply a configuration, or restart the nagios process, it shows most services to be - only some of the pending checks are actually checks still in force. I noticed several services I had deleted from nodes were now present on hosts again

I am sure this is a database issue, and I am curious to run the database repair again, but do not want to have to go back around to replace all of the manually moved templates and configurations. I believe nagios should maintain state during a configuration apply, and not default back to the pending with last check 1999-11-30- this is behavior I have never seen prior to coming in this morning.

Anyone have experience with this?

What kind of changes have you made on the system prior to this weekend? What do you mean by "instability in the system"? Do you have a good backup that you can revert to?

When I got in, the last check was shown to be 4/20 @13:20 EST.

We had some changes in our environment, but all to end-points that are not effected in this problem, as their checks are being completed well.

I have a nightly backup running that I could revert to, but am trying to avoid this if at all possible. Just did a simple remove of a service, and yet the services still remain as part of the pending checks, but are no longer part of the services configuration under the server-

did you apply configuration after removing the service?

Yes-

Seems I'm looking at close to the same issue I had before that we worked on with Services not showing up under the hosts. I think I'm going to arrange to rebuild this server, as it looks like we are getting into more and more issues as we go along. issuing a reboot command to try and get things squared away again.

Let us know if you are going to re-build and import or if you would rather continue on this path. We are more than happy to assist with either.

I'm going to have to do a rebuild and import, but this will have to be scheduled. mean-time I need to get this instance back up and running. looks like this is now the same exact problem I had with http://support.nagios.com/forum/viewtop ... =16&t=9868

Services are no longer tied to the hosts. several reboots and restarts later I'm in the same spot.

I am now doing a full backup to restore on a secondary node which I will be upgrading to 1.8. after that I will be able to upgrade the production instance. only issue is that I have a lot of NRDS checks assigned to hit the production node that I will have to pipe back to a secondary node- using puppet it's not a huge deal, but getting the change in place will be, so it will take a couple of days for approvals. man time I'm trying to get everything running smoothly as possible on the current prod instance.

Let us know the this comes back up after the rebuild