Nagios Server Crashed

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
RIDS_I2MP
Posts: 751
Joined: Thu Mar 13, 2014 9:25 am

Nagios Server Crashed

Post by RIDS_I2MP »

Hi Team,
Our Nagios XI server crashed this morning, we are not able to access the server via putty and when we try to access the webpage it gives a message saying

Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Code: Select all

Run the following from the CLI as root to attempt to repair the DB


/usr/local/nagiosxi/scripts/repair_databases.sh
Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB


/usr/local/nagiosxi/scripts/repair_databases.sh
Message: A database connection error has been detected, we are attempting to repair the server, if the repair does not resolve the issue, please contact Nagios support.

Run the following from the CLI as root to attempt to repair the DB


/usr/local/nagiosxi/scripts/repair_databases.sh
Our admin did login to the server as root and we see that the nagios process is running. when we trid to restart the service we got a message saying

"Running configuration check... CONFIG ERROR! Restart aborted. Check your Nagios configuration."

we checked both Postgres and MySQLd and it returned a message saying " postmaster dead but pid file exists" similar message for mysqld.

I have tried to restart the ndo2db process and it gave a error message saying
"service ndo2db start
Starting ndo2db:rm: cannot remove `/usr/local/nagiosxi/var/subsys/ndo2db': Read-only file system
rm: cannot remove `/usr/local/nagios/var/ndo.sock': Read-only file system
touch: cannot touch `/usr/local/nagios/var/ndo2db.lock': Read-only file system
chown: changing ownership of `/usr/local/nagios/var/ndo2db.lock': Read-only file system
Could not bind socket: Address already in use
touch: cannot touch `/usr/local/nagiosxi/var/subsys/ndo2db': Read-only file system
chown: changing ownership of `/usr/local/nagiosxi/var/subsys/ndo2db': Read-only file system
done."

we tried to run a backup

Code: Select all

Sthitha Pragna Madhiraju	As per the email we received from Aravindhan (Attached), we are asked to submit our goal setting and evaluation to our reviewer and appraiser in Performagic as per the new Org changes and provide the details of the reporting lead to the appraiser for consulting during the evaluation process. Please let me know we are indeed required to send the details to our reporting lead, as we will be completing our implementation next month and the reporting lead might change if we move to a new project.
	10:46
/backup_xi.sh
./backup_xi.sh: line 6: ./config.dat: Read-only file system
./backup_xi.sh: line 7: ./config.dat: No such file or directory
[root@eu2napu002 scripts]# pwd
/usr/local/nagiosxi/scripts
when we tried to repair the DB manually, we got the below error

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh
touch: cannot touch `/usr/local/nagiosxi/scripts/repair_databases.lock': Read-only file system
mysqld dead but pid file exists
rm: cannot remove `/var/lib/mysql/mysql.sock': Read-only file system
Starting mysqld:                                           [FAILED]
mysqld dead but pid file exists
rm: cannot remove `/var/lib/mysql/mysql.sock': Read-only file system
DATABASE: nagios
TABLE:
/var/lib/mysql/nagios ~
Stopping mysqld:                                           [FAILED]
myisamchk: error: 30 when opening MyISAM-table 'nagios_acknowledgements.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_commands.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_commenthistory.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_comments.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_configfiles.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_configfilevariables.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_conninfo.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contact_addresses.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contactgroup_members.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contactgroups.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contact_notificationcommands.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contactnotificationmethods.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contactnotifications.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contacts.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_contactstatus.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_customvariables.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_customvariablestatus.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_dbversion.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_downtimehistory.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_eventhandlers.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_externalcommands.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_flappinghistory.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hostchecks.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_host_contactgroups.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_host_contacts.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hostdependencies.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hostescalation_contactgroups.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hostescalation_contacts.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hostescalations.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hostgroup_members.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hostgroups.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_host_parenthosts.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hosts.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_hoststatus.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_instances.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_logentries.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_notifications.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_objects.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_processevents.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_programstatus.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_runtimevariables.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_scheduleddowntime.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_servicechecks.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_service_contactgroups.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_service_contacts.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_servicedependencies.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_serviceescalation_contactgroups.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_serviceescalation_contacts.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_serviceescalations.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_servicegroup_members.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_servicegroups.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_service_parentservices.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_services.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_servicestatus.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_statehistory.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_systemcommands.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_timedeventqueue.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_timedevents.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_timeperiods.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'nagios_timeperiod_timeranges.MYI'
Starting mysqld:                                           [FAILED]
~

===============
REPAIR COMPLETE
===============
mysqld dead but pid file exists
rm: cannot remove `/var/lib/mysql/mysql.sock': Read-only file system
DATABASE: nagiosql
TABLE:
/var/lib/mysql/nagiosql ~
Stopping mysqld:                                           [FAILED]
myisamchk: error: 30 when opening MyISAM-table 'tbl_command.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_contactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_contact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_contacttemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_domain.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_hostdependency.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_hostescalation.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_hostextinfo.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_hostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_host.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_hosttemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_info.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContactgroupToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContactgroupToContact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContacttemplateToCommandHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContacttemplateToCommandService.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContacttemplateToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContacttemplateToContacttemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContacttemplateToVariabledefinition.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContactToCommandHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContactToCommandService.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContactToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContactToContacttemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkContactToVariabledefinition.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostdependencyToHost_DH.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostdependencyToHostgroup_DH.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostdependencyToHostgroup_H.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostdependencyToHost_H.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostescalationToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostescalationToContact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostescalationToHostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostescalationToHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostgroupToHostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostgroupToHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHosttemplateToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHosttemplateToContact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHosttemplateToHostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHosttemplateToHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHosttemplateToHosttemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHosttemplateToVariabledefinition.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostToContact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostToHostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostToHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostToHosttemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkHostToVariabledefinition.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicedependencyToHost_DH.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicedependencyToHostgroup_DH.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicedependencyToHostgroup_H.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicedependencyToHost_H.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicedependencyToService_DS.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicedependencyToService_S.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceescalationToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceescalationToContact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceescalationToHostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceescalationToHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceescalationToService.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicegroupToServicegroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicegroupToService.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicetemplateToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicetemplateToContact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicetemplateToHostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicetemplateToHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicetemplateToServicegroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicetemplateToServicetemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServicetemplateToVariabledefinition.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceToContactgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceToContact.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceToHostgroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceToHost.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceToServicegroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceToServicetemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkServiceToVariabledefinition.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_lnkTimeperiodToTimeperiod.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_logbook.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_mainmenu.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_servicedependency.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_serviceescalation.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_serviceextinfo.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_servicegroup.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_service.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_servicetemplate.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_settings.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_submenu.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_timedefinition.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_timeperiod.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_user.MYI'

---------

myisamchk: error: 30 when opening MyISAM-table 'tbl_variabledefinition.MYI'
Starting mysqld:                                           [FAILED]
~

===============
REPAIR COMPLETE
===============
Stopping ndo2db: done.
rm: cannot remove `/usr/local/nagios/var/ndo2db.lock': Read-only file system
rm: cannot remove `/usr/local/nagiosxi/var/subsys/ndo2db': Read-only file system
Starting ndo2db:rm: cannot remove `/usr/local/nagiosxi/var/subsys/ndo2db': Read-only file system
rm: cannot remove `/usr/local/nagios/var/ndo.sock': Read-only file system
touch: cannot touch `/usr/local/nagios/var/ndo2db.lock': Read-only file system
chown: changing ownership of `/usr/local/nagios/var/ndo2db.lock': Read-only file system
Could not bind socket: Address already in use
touch: cannot touch `/usr/local/nagiosxi/var/subsys/ndo2db': Read-only file system
chown: changing ownership of `/usr/local/nagiosxi/var/subsys/ndo2db': Read-only file system
done.
Running configuration check... CONFIG ERROR!  Restart aborted.  Check your Nagios configuration.
Thanks & Regards,
I2MP Team.
Top
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Nagios Server Crashed

Post by lmiltchev »

Run the following:

Code: Select all

service mysqld stop
rm -f /var/lib/mysql/mysql.sock
service mysqld start
Then try to run the repair script again:

Code: Select all

cd /usr/local/nagiosxi/scripts
./repair_databases.sh
BTW, how much HDD space do you have on the system?

Code: Select all

df -h
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
RIDS_I2MP
Posts: 751
Joined: Thu Mar 13, 2014 9:25 am

Re: Nagios Server Crashed

Post by RIDS_I2MP »

Hi TEam,

Thanks for the reply...This issue has been resolved now.
Thanks & Regards,
I2MP Team.
Top
Locked

Return to “Nagios XI”