Nagios XI 2014 2.5 mysql DB errors...

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Nagios XI 2014 2.5 mysql DB errors...

Post by JakeHatMacys »

We've been getting spammed with these errors for a good few weeks now. We'll run the DB repair but it doesn't seem to resolve the issue. Any ideas where to start on fixing something like this?
log: /var/log/messages

Code: Select all

Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 nagios: wproc: Core Worker 12000: job 126427 (pid=9426): Dormant child reaped
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26619                                               7', logentry_type='1', logentry_data='wproc:   host=esu2v812; service=Check for read only File system on linu                                               x;', realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26623                                               3', logentry_type='1', logentry_data='wproc:   early_timeout=1; exited_ok=0; wait_status=0; error_code=62;',                                                realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26628                                               0', logentry_type='2', logentry_data='Warning: Check of service \'Check for read only File system on linux\'                                                on host \'esu2v812\' timed out after 60\.005s!', realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Sep  4 08:51:42 esu1l268 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_                                               id='1', logentry_time=FROM_UNIXTIME(1441371102), entry_time=FROM_UNIXTIME(1441371102), entry_time_usec='26654                                               7', logentry_type='262144', logentry_data='wproc: Core Worker 12000: job 126427 \(pid=9426\): Dormant child r                                               eaped', realtime_data='1', inferred_data_extracted='1''
Sep  4 08:51:42 esu1l268 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and la                                               st (automatic?) repair failed'
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios XI 2014 2.5 mysql DB errors...

Post by tgriep »

Try this procedure to see if it will repair the MYSQL database.

Code: Select all

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f *
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI 2014 2.5 mysql DB errors...

Post by JakeHatMacys »

From the first part FYI: ( couldn't get back to the very top)

Code: Select all

- recovering (with sort) MyISAM-table 'nagios_servicedependencies.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_serviceescalation_contactgroups.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_serviceescalation_contactgroups.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contactgroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_serviceescalation_contacts.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_serviceescalation_contacts.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalation_contacts.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_serviceescalations.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_serviceescalations.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_serviceescalations.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_servicegroup_members.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_servicegroup_members.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroup_members.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_servicegroups.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_servicegroups.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_servicegroups.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_service_parentservices.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_service_parentservices.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_service_parentservices.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_services.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_services.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_services.MYI'
Data records: 9876
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_servicestatus.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_servicestatus.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_servicestatus.MYI'
Data records: 9876
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6
- Fixing index 7
- Fixing index 8
- Fixing index 9
- Fixing index 10
- Fixing index 11
- Fixing index 12
- Fixing index 13
- Fixing index 14
- Fixing index 15
- Fixing index 16
- Fixing index 17
- Fixing index 18
- Fixing index 19

---------

myisamchk: error: 'nagios_statehistory.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_statehistory.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_statehistory.MYI'
Data records: 1115452
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_systemcommands.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_systemcommands.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_systemcommands.MYI'
Data records: 140
- Fixing index 1
- Fixing index 2
- Fixing index 3

---------

myisamchk: error: 'nagios_timedeventqueue.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timedeventqueue.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timedeventqueue.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

myisamchk: error: 'nagios_timedevents.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timedevents.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timedevents.MYI'
Data records: 0
- Fixing index 1
- Fixing index 2
- Fixing index 3
- Fixing index 4
- Fixing index 5
- Fixing index 6

---------

myisamchk: error: 'nagios_timeperiods.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timeperiods.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiods.MYI'
Data records: 17
- Fixing index 1
- Fixing index 2

---------

myisamchk: error: 'nagios_timeperiod_timeranges.frm' is not a MyISAM-table

---------

myisamchk: error: 'nagios_timeperiod_timeranges.MYD' is not a MyISAM-table

---------

- recovering (with sort) MyISAM-table 'nagios_timeperiod_timeranges.MYI'
Data records: 96
- Fixing index 1
- Fixing index 2
Still running the php command. Is it supposed to take awhile? Just sitting here like this:

[root@****** nagios]# php /usr/local/nagiosxi/cron/dbmaint.php
CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1441130895)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1433959695)
Top
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI 2014 2.5 mysql DB errors...

Post by JakeHatMacys »

Okay NM that finished:

Code: Select all

[root@******* nagios]# php /usr/local/nagiosxi/cron/dbmaint.php
CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1410199695)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1441130895)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'notifications'...
SQL: DELETE FROM nagios_notifications WHERE start_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'contactnotifications'...
SQL: DELETE FROM nagios_contactnotifications WHERE start_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'contactnotificationmethods'...
SQL: DELETE FROM nagios_contactnotificationmethods WHERE start_time < FROM_UNIXTIME(1433959695)
CLEANING ndoutils TABLE 'statehistory'...
SQL: DELETE FROM nagios_statehistory WHERE state_time < FROM_UNIXTIME(1378663695)
CLEANING ndoutils TABLE 'timedevents'...
SQL: DELETE FROM nagios_timedevents WHERE event_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'systemcommands'...
SQL: DELETE FROM nagios_systemcommands WHERE start_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'servicechecks'...
SQL: DELETE FROM nagios_servicechecks WHERE start_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'hostchecks'...
SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1441735395)
CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1441735395)
LASTOPT:  1441734901
INTERVAL: 60
NOW:      1441735695
OPTTIME:  1441738501
CLEANING nagiosxi TABLE 'commands'...
SQL: DELETE FROM xi_commands WHERE processing_time < 1441706895::abstime::timestamp without time zone
CLEANING nagiosxi TABLE 'events'...
SQL: DELETE FROM xi_events WHERE processing_time < 1441706895::abstime::timestamp without time zone
SQL1: SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL
SQL2: DELETE FROM xi_meta WHERE meta_id IN (SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL)
CLEANING nagiosxi TABLE 'auditlog'...
SQL: DELETE FROM xi_auditlog WHERE log_time < 1439143695::abstime::timestamp without time zone
CLEANING nagiosql TABLE 'logbook'...
SQL: DELETE FROM tbl_logbook WHERE time < FROM_UNIXTIME(1441706895)




Repair Complete: Removing Lock File
Top
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Nagios XI 2014 2.5 mysql DB errors...

Post by ssax »

Usually that error is caused by mysqld not shutting down gracefully, after doing the repair are you seeing it in the logs anymore?

How often does it occur?

How's your disk space?

Code: Select all

df -h
df -i
Also, is your DB offloaded (different server)?
Top
JakeHatMacys
Posts: 281
Joined: Thu Sep 25, 2014 3:21 pm

Re: Nagios XI 2014 2.5 mysql DB errors...

Post by JakeHatMacys »

ssax wrote:Usually that error is caused by mysqld not shutting down gracefully, after doing the repair are you seeing it in the logs anymore?

How often does it occur?

How's your disk space?

Code: Select all

df -h
df -i
Also, is your DB offloaded (different server)?
It looks as though the log isn't writeable atm, which I reached out to my unix team to address... DB is on the server. So once it's logging again we'll see what it says.
Top
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: Nagios XI 2014 2.5 mysql DB errors...

Post by tgriep »

Let us know when the log file is resolved and we can continue on.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Top
Locked

Return to “Nagios XI”