Nagios apply configuration issue

[mejokj]

Hi,

We are facing a isssue with the nagios apply configuration.

+++++++++++++++++
Error: Service description missing from list '' (config file '/usr/local/nagios/etc/servicegroups.cfg', starting at line 117)
Error: Could not expand members specified in servicegroup 'UPS - Temp' (config file '/usr/local/nagios/etc/servicegroups.cfg', starting at line 117)
++++++++++++++++++

After we activate the the service name "Internal Temp" of host example.com the issue will be fixed. But the again after some time the service will be disabled automatically and apply configuration failed. This service is parent service under the service there are 28 managed hosts . Why this service Internal Temp is automatically removed.

[benjaminsmith]

Hi,

After you activate the service again and it's working, are you making any new configuration changes and are using any type of automation to make those changes?

Try to run through the following steps in Configure > CCM > Tools > Config File Management to clear the error.

1. Delete Files
2. Write Files
3. Verify Files ( if you see any errors follow the message to correct and repeat steps 1-3 until it passes)
4. Restart Nagios Core

Benjamin

[mejokj]

Hi Benjaminsmith,

After you activate the service again and it's working, are you making any new configuration changes and are using any type of automation to make those changes?

Yes, we have tried removing a few hosts after activating the service and not using any automation to do any changes.

Note: The service gets failed even if we did not do any configuration changes.

----------------------------------------------------------------------------------------------------------------------------------------------------
Try to run through the following steps in Configure > CCM > Tools > Config File Management to clear the error.

1. Delete Files
2. Write Files
3. Verify Files ( if you see any errors follow the message to correct and repeat steps 1-3 until it passes)
4. Restart Nagios Core

Kindly confirm if we perform the above steps, will it remove any data/graph already existing in Nagiosxi?

Thanks

[benjaminsmith]

Hi,

Kindly confirm if we perform the above steps, will it remove any data/graph already existing in Nagiosxi?

No, this is a safe procedure, it won't affect existing or future performance graphs.

Let me know if you're able to resolve the issue.

Thanks,
Benjamin

[mejokj]

Hi Benjamin,

Thanks for the confirmation.

We have performed all steps provided by you. But the issue still not resolved, the service again gets inactive in a few mins.

Kindly help us to fix this issue. PM the system profile file.

Please find the verify output from Nagiosxi.

-----------------------------------------------------------------------------------------------------------------------
Checked 53 contacts.
Checked 2 contact groups.
Checked 147 commands.
Checked 46 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 351 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 46 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 53
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
--------------------------------------------------------------------------------------------------------------------

Thanks,

[benjaminsmith]

Hi,

So the the logs look ok in the profile but I noticed that the database is offloaded in this instance so that was not included. Can you retrieve this from the offload server (db)?

Code: Select all

/var/log/mariadb/mariadb.log
/var/log/mysqld.log

Also, I can test your configurations here with a database dump. If could post the following file, that would be very helpful. Thank you, Benjamin

Code: Select all

mysqldump -uroot -pnagiosxi nagiosxi > nagiosxi.sql

[mejokj]

Hi Benjaminsmith,

I have sent you the details in PM to you.

Please have a check

Thanks,

[benjaminsmith]

Hi,

I received your files. Thank you. Currently setting up a test environment will update shortly.

Thank you for your patience.

Benjamin

[benjaminsmith]

Hi,

I imported your configurations and have been able to apply configuration. I'm able to view the service group UPS - Temp (with 28 services).

However, I did notice some database issues.

Code: Select all

181118 11:45:13 [ERROR] mysqld: Table './nagios/nagios_notifications' is marked as crashed and last (automatic?) repair failed
181118 11:45:13 [ERROR] mysqld: Table './nagios/nagios_notifications' is marked as crashed and last (automatic?) repair failed
181118 11:45:13 [ERROR] mysqld: Table './nagios/nagios_notifications' is marked as crashed and last (automatic?) repair failed

Please log in as root and run the following repair script.

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh

Then try to repeat the steps in Config File management and let me know the results. If it's still failing try to correct the error based on the message. If that's unsuccessful, please dump the nagiosql database and send it over.

Code: Select all

 mysqldump -uroot -pnagiosxi nagiosql > nagiosql.sql

Thank you,
Benjamin

[mejokj]

Hi Benjaminsmith,

We ran the database repair script. Please find below the output below.

After that, we repeated the steps in Config File management. Please find below the verify output.

We have send you the nagiosql database dump as PM to you.

The issue still persists, the service getting failed very frequently. Please suggest

Config File management Output
---------------------------------------

Checked 53 contacts.
Checked 2 contact groups.
Checked 147 commands.
Checked 46 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 340 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 46 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 45
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check

--------------------------------------------------------------------------------------------------------------------------------
Repair script output
------------------------

nagios.nagios_acknowledgements OK
nagios.nagios_commands OK
nagios.nagios_commenthistory OK
nagios.nagios_comments OK
nagios.nagios_configfiles OK
nagios.nagios_configfilevariables OK
nagios.nagios_conninfo OK
nagios.nagios_contact_addresses OK
nagios.nagios_contact_notificationcommands OK
nagios.nagios_contactgroup_members OK
nagios.nagios_contactgroups OK
nagios.nagios_contactnotificationmethods OK
nagios.nagios_contactnotifications OK
nagios.nagios_contacts OK
nagios.nagios_contactstatus OK
nagios.nagios_customvariables OK
nagios.nagios_customvariablestatus OK
nagios.nagios_dbversion OK

nagios offloaded database repair succeeded
nagiosql offloaded database repair succeeded
nagiosxi offloaded database repair succeeded


Thanks