db maintenance will not run, Nagios crashes after ~1 hour

[mxthunder]

Hello,
Having some difficulties with one of my Nagios instances. Any attempt to run the database repair script simply results in more failures and errors, and ends up filling up /var on my server.

Databse Error
A database connection error has been detected, please follow the repair prompt below. If the issue persists, please contact Nagios support.
Run the following from the CLI as root to attempt to repair the DB:

/usr/local/nagiosxi/scripts/repair_databases.sh



- recovering (with sort) MyISAM-table 'tbl_user.MYI'
Data records: 2
- Fixing index 1
- Fixing index 2
/usr/bin/myisamchk: Can't change ownership of the file '/var/lib/mysql/nagiosql/tbl_user.MYD' (Errcode: 1)
myisamchk: error: 1 when fixing table
MyISAM-table 'tbl_user.MYI' is not fixed because of errors
Try fixing it by using the --safe-recover (-o), the --force (-f) option or by not using the --quick (-q) flag

---------

- recovering (with sort) MyISAM-table 'tbl_variabledefinition.MYI'
Data records: 1658
- Fixing index 1
/usr/bin/myisamchk: Can't change ownership of the file '/var/lib/mysql/nagiosql/tbl_variabledefinition.MYD' (Errcode: 1)
myisamchk: error: 1 when fixing table
MyISAM-table 'tbl_variabledefinition.MYI' is not fixed because of errors
Try fixing it by using the --safe-recover (-o), the --force (-f) option or by not using the --quick (-q) flag
Starting mysqld: [ OK ]
/usr/local/nagiosxi/scripts



this has been happening for the last 2 days. A reboot will bring up nagios for a short while, but then it crashes again. I have attempted to manually stop/start all services with the same result.

Please advise....

[benjaminsmith]

Hello @mxthunder,

this has been happening for the last 2 days. A reboot will bring up nagios for a short while, but then it crashes again.

Did you make any changes to the Nagios instance before experiencing the database issues?

Please try to run the following repair command as root:

Code: Select all

mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm

If that does not resolve the issue, please PM your system profile for us to review.

To send us your system profile.
Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and share in a private message or upload it to the post/ticket.

[mxthunder]

That did not resolve my issue, the command immediately failed. See output below.
There is no way for me to upload a system profile - as the web gui is currently down....
There were no changes to the environment when this issue started to occur.

Thanks,

-bash-4.1$ mysqlcheck -r -f -uroot -pnagiosxi --all-databases --use_frm
mysql.columns_priv OK
mysql.db
warning : Number of rows changed from 0 to 4
status : OK
mysql.event OK
mysql.func OK
mysql.general_log OK
mysql.help_category
warning : Number of rows changed from 0 to 40
status : OK
mysql.help_keyword
warning : Number of rows changed from 0 to 453
status : OK
mysql.help_relation
warning : Number of rows changed from 0 to 1009
status : OK
mysql.help_topic
warning : Number of rows changed from 0 to 510
status : OK
mysql.host OK
mysql.ndb_binlog_index OK
mysql.plugin OK
mysql.proc OK
mysql.procs_priv OK
mysql.servers OK
mysql.slow_log OK
mysql.tables_priv OK
mysql.time_zone OK
mysql.time_zone_leap_second OK
mysql.time_zone_name OK
mysql.time_zone_transition OK
mysql.time_zone_transition_type OK
mysql.user
warning : Number of rows changed from 0 to 9
status : OK
nagios.nagios_acknowledgements
error : Can't create new tempfile: './nagios/nagios_acknowledgements.TMD'
status : Operation failed
nagios.nagios_commands
error : Can't create new tempfile: './nagios/nagios_commands.TMD'
status : Operation failed
nagios.nagios_commenthistory
error : Can't create new tempfile: './nagios/nagios_commenthistory.TMD'
status : Operation failed
nagios.nagios_comments
error : Can't create new tempfile: './nagios/nagios_comments.TMD'
status : Operation failed
nagios.nagios_configfiles
error : Can't create new tempfile: './nagios/nagios_configfiles.TMD'
status : Operation failed
nagios.nagios_configfilevariables
error : Can't create new tempfile: './nagios/nagios_configfilevariables.TMD'
status : Operation failed

[benjaminsmith]

Hello @maxthunder,

There is no way for me to upload a system profile - as the web gui is currently down....
There were no changes to the environment when this issue started to occur.

If you're not able to access the server, I would recommend opening a support ticket as we have the option of providing a remote session if needed.

You can open a support ticket at:
https://support.nagios.com/tickets/

You can try to re-start the services, and if you're able to login please provide a system profile as soon as possible. Thanks.

To re-start Nagios:

Code: Select all

service nagios stop
service ndo2db stop
service mysqld stop
service httpd stop
pkill -9 nagios
service nagios restart
service ndo2db restart
service mysqld restart
service httpd restart

[benjaminsmith]

Hi @maxthunder,

I see you opened a ticket for this issue, so I'll close this thread. Please post any other replies to the ticket.

Thanks