Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Nagios XI 2014 2.5 mysql DB errors...

https://support.nagios.com/forum/viewtopic.php?t=34535

Page 1 of 1

Nagios XI 2014 2.5 mysql DB errors...

Posted: Fri Sep 04, 2015 7:57 am
by JakeHatMacys
We've been getting spammed with these errors for a good few weeks now. We'll run the DB repair but it doesn't seem to resolve the issue. Any ideas where to start on fixing something like this?
log: /var/log/messages

Code: Select all

Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 nagios: wproc: Core Worker 12000: job 126427 (pid=9426): Dormant child reaped
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26619                                               7', logentry_type='1', logentry_data='wproc:   host=esu2v812; service=Check for read only File system on linu                                               x;', realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26623                                               3', logentry_type='1', logentry_data='wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;',                                                realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26628                                               0', logentry_type='2', logentry_data='Warning: Check of service \'Check for read only File system on linux\'                                                on host \'esu2v812\' timed out after 60\.005s!', realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26654                                               7', logentry_type='262144', logentry_data='wproc: Core Worker 12000: job 126427 \(pid=9426\): Dormant child r                                               eaped', realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'

Re: Nagios XI 2014 2.5 mysql DB errors...

Posted: Fri Sep 04, 2015 9:07 am
by tgriep
Try this procedure to see if it will repair the MYSQL database.

Code: Select all

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f *
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

Re: Nagios XI 2014 2.5 mysql DB errors...

Posted: Tue Sep 08, 2015 1:10 pm
by JakeHatMacys
From the first part FYI: ( couldn't get back to the very top)

Code: Select all

- recovering (with sort) MyISAM-table 'nagios_servicedependencies.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_serviceescalation_contactgroups.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_serviceescalation_contactgroups.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contactgroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_serviceescalation_contacts.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_serviceescalation_contacts.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contacts.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_serviceescalations.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_serviceescalations.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalations.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_servicegroup_members.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_servicegroup_members.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroup_members.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_servicegroups.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_servicegroups.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_service_parentservices.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_service_parentservices.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_service_parentservices.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_services.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_services.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_services.MYI'
Data records: 9876
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_servicestatus.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_servicestatus.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_servicestatus.MYI'
Data records: 9876
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
- Fixing index 7
- Fixing index 8
- Fixing index 9
- Fixing index 10
- Fixing index 11
- Fixing index 12
- Fixing index 13
- Fixing index 14
- Fixing index 15
- Fixing index 16
- Fixing index 17
- Fixing index 18
- Fixing index 19

---------

myisamchk: error: 'nagios_statehistory.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_statehistory.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_statehistory.MYI'
Data records: 1115452
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_systemcommands.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_systemcommands.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_systemcommands.MYI'
Data records: 140
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_timedeventqueue.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timedeventqueue.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timedeventqueue.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

myisamchk: error: 'nagios_timedevents.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timedevents.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timedevents.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

myisamchk: error: 'nagios_timeperiods.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timeperiods.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiods.MYI'
Data records: 17
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_timeperiod_timeranges.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timeperiod_timeranges.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiod_timeranges.MYI'
Data records: 96
- Fixing index 1
- Fixing index 2
Still running the php command. Is it supposed to take awhile? Just sitting here like this:

[root@****** nagios]# php /usr/local/nagiosxi/cron/dbmaint.php
CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1441130895)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1433959695)

Re: Nagios XI 2014 2.5 mysql DB errors...

Posted: Tue Sep 08, 2015 1:17 pm
by JakeHatMacys
Okay NM that finished:

Code: Select all

[root@******* nagios]# php /usr/local/nagiosxi/cron/dbmaint.php
CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1441130895)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'notifications'...
SQL: DELETE FROM nagios_notifications WHERE start_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'contactnotifications'...
SQL: DELETE FROM nagios_contactnotifications WHERE start_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'contactnotificationmethods'...
SQL: DELETE FROM nagios_contactnotificationmethods WHERE start_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'statehistory'...
SQL: DELETE FROM nagios_statehistory WHERE state_time < FROM_UNIXTIME(1378663695)
CLEANING ndoutils TABLE 'timedevents'...
SQL: DELETE FROM nagios_timedevents WHERE event_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'systemcommands'...
SQL: DELETE FROM nagios_systemcommands WHERE start_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'servicechecks'...
SQL: DELETE FROM nagios_servicechecks WHERE start_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'hostchecks'...
SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1441735395)
LASTOPT:  1441734901
INTERVAL: 60
NOW:      1441735695
OPTTIME:  1441738501
CLEANING nagiosxi TABLE 'commands'...
SQL: DELETE FROM xi_commands WHERE processing_time < 1441706895::abstime::timestamp without time zone
CLEANING nagiosxi TABLE 'events'...
SQL: DELETE FROM xi_events WHERE processing_time < 1441706895::abstime::timestamp without time zone
SQL1: SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL
SQL2: DELETE FROM xi_meta WHERE meta_id IN (SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL)
CLEANING nagiosxi TABLE 'auditlog'...
SQL: DELETE FROM xi_auditlog WHERE log_time < 1439143695::abstime::timestamp without time zone
CLEANING nagiosql TABLE 'logbook'...
SQL: DELETE FROM tbl_logbook WHERE time < FROM_UNIXTIME(1441706895)




Repair Complete: Removing Lock File

Re: Nagios XI 2014 2.5 mysql DB errors...

Posted: Tue Sep 08, 2015 3:31 pm
by ssax
Usually that error is caused by mysqld not shutting down gracefully, after doing the repair are you seeing it in the logs anymore?

How often does it occur?

How's your disk space?

Code: Select all

df -h
df -i
Also, is your DB offloaded (different server)?

Re: Nagios XI 2014 2.5 mysql DB errors...

Posted: Wed Sep 09, 2015 1:10 pm
by JakeHatMacys
ssax wrote:Usually that error is caused by mysqld not shutting down gracefully, after doing the repair are you seeing it in the logs anymore?

How often does it occur?

How's your disk space?

Code: Select all

df -h
df -i
Also, is your DB offloaded (different server)?
It looks as though the log isn't writeable atm, which I reached out to my unix team to address... DB is on the server. So once it's logging again we'll see what it says.

Re: Nagios XI 2014 2.5 mysql DB errors...

Posted: Wed Sep 09, 2015 3:55 pm
by tgriep
Let us know when the log file is resolved and we can continue on.
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited