Can't apply config since upgrade to 2014.2.3

[krutaw]

After upgrading Nagios XI to latest version (last weekend) I am unable to apply configuration. After some investigation, I found that if I re-run the upgrade script it does something that causes Nagios XI to be able to apply configuration again without error.

For example, on my smallest nagios server I receive the following error:

Error: Contact 'etsi-all' is not defined anywhere!
Error: Could not add contact 'etsi-all' to host (config file '/usr/local/nagios/etc/hosts/<SERVERNAMEOMMITTED>.<DOMAINNAME>.com.cfg', starting on line 14)

When I look at the /usr/local/nagios/etc/contacts.cfg I see the following:

define contact {
contact_name etsi-all
alias etsi-all
host_notification_period etsi-all_notification_times
service_notification_period etsi-all_notification_times
host_notification_options d,u,r,f,s
service_notification_options w,u,c,r,f,s
email <EMAILADDRESS>@<DOMAINNAME>.com
use xi_contact_generic
}


So it's clearly defined but I'll play along. I then deleted all of the references to the contact on hosts and services using the bulk modification tool. Unfortunately, still unable to apply config. So I re-run the upgrade. Now I'm able to apply config (for a short time.) To hopefully settle things down I removed the Nagios XI user and the CCM contact for etsi-all (and verified that the user was removed from contacts.cfg.) Wait 20 minutes. Try applying configuration, same exact error:

Error: Contact 'etsi-all' is not defined anywhere!
Error: Could not add contact 'etsi-all' to host (config file '/usr/local/nagios/etc/hosts/<SERVERNAMEOMMITTED>.<DOMAINNAME>.com.cfg', starting on line 14)


So then I attempted to run a database repair (/usr/local/nagiosxi/scripts/repair_databases.sh) Again, the error persists. Then it dawned on me, I went to look in the database and when running a simple query like 'SELECT * FROM nagios.nagios_contacts;' MySQL Workbench completely crashes or (on subsequent attempts) hangs trying to run the query. I'm seriously starting to think its directly database related based on these results.

Any ideas how to fix this beyond starting over from scratch?

[abrist]

Go to CCM --> Write config files --> Delete --> Write --> Verify.
If it verifies ok, run apply config. If not, resolve the errors and rerun Write --> Verify. And repeat until it verifies ok.

[krutaw]

So, I did that and got the following back:

Successfully deleted all Host / Service Config Files


-------------
Host configuration files successfully written!
Service configuration files successfully written!

Configuration file: hostgroups.cfg successfully written!
Configuration file: servicegroups.cfg successfully written!
Configuration file: hosttemplates.cfg successfully written!
Configuration file: servicetemplates.cfg successfully written!
Configuration file: timeperiods.cfg successfully written!
Configuration file: commands.cfg successfully written!
Configuration file: contacts.cfg successfully written!
Configuration file: contactgroups.cfg successfully written!
Configuration file: contacttemplates.cfg successfully written!
Configuration file: servicedependencies.cfg successfully written!
Configuration file: hostdependencies.cfg successfully written!
Configuration file: serviceescalations.cfg successfully written!
Configuration file: hostescalations.cfg successfully written!
Configuration file: serviceextinfo.cfg successfully written!
Configuration file: hostextinfo.cfg successfully written!
----------------

Then verify says:
Total Warnings: 10
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

----------------

Then apply configuration gives the exact same error as in original post. Ideas?

[tmcdonald]

You may be running into a recent issue with the sudoers file. Please run the following as root and post the output:

Code: Select all

cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh

[krutaw]

So sorry, I forgot to mention that piece of the puzzle in my original post! I am able to apply configuration via the command line and any changes in the database are respected, I simply can't apply configuration via the WEB. Here's the output of the tail end of the command:


Checked 36 contacts.
Checked 12 contact groups.
Checked 117 commands.
Checked 36 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 42 hosts
Checked 422 service dependencies
Checked 0 host dependencies
Checked 36 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 10
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios: done.
Starting nagios: done.


-------------

In short, it does apply configuration. Does that change the color of this unicorn?

[tmcdonald]

Did the nagios user account expire?

Code: Select all

chage -l nagios

[krutaw]

Good thought, but sadly no. Here's the output:

Last password change : Feb 23, 2013
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7

[abrist]

Try this. Edit:

Code: Select all

/usr/local/nagiosxi/scripts/restart_nagios_with_export.sh

Change line # 44 from:

Code: Select all

sudo ./manage_services.sh restart nagios

To:

Code: Select all

./manage_services.sh restart nagios

Save out and try to apply config again. This is a bug that is fixed in the next version of XI.

[krutaw]

BOOM! Thank you, that worked!

[slansing]

Awesome! Locking this one up.