MySQL Problem

[kotterbein]

began seeing this error again:

2012-08-14T10:27:41.313565-04:00 id-nag01 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_notifications SET instance_id='1', notification_ty
pe='1', notification_reason='0', start_time=FROM_UNIXTIME(1344953061), start_time_usec='148075', end_time=FROM_UNIXTIME(0), end_time_usec='0', object_id
='6056', state='2', output='Connection refused by host', long_output='Connection refused by host', escalated='0', contacts_notified='0' ON DUPLICATE KEY
UPDATE instance_id='1', notification_type='1', notification_reason='0', start_time=FROM_UNIXTIME(1344953061), start_time_usec='148075', end_time=FROM_U
NIXTIME(0), end_time_usec='0', object_id='6056', state='2', output='Connection refused by host', long_output='Connection refused by host', escalated='0'
, contacts_notified='0''
2012-08-14T10:27:41.313584-04:00 id-nag01 ndo2db: mysql_error: 'Server shutdown in progress'
2012-08-14T10:27:41.313764-04:00 id-nag01 ndo2db: Error: mysql_query() failed for 'DELETE FROM nagios_timedevents WHERE instance_id='1' AND scheduled_ti
me<FROM_UNIXTIME(1344868061)'
2012-08-14T10:27:41.313772-04:00 id-nag01 ndo2db: mysql_error: 'Lost connection to MySQL server during query'
2012-08-14T10:27:41.313778-04:00 id-nag01 ndo2db: Error: Connection to MySQL database has been lost!

running again now after restarting all services.

[scottwilkerson]

These logs are only showing information during s mysql shutdown. The logs you posted only span a time period of 0.0002 seconds.

[kotterbein]

the last refresh time in the xi interface was approximately 10:04AM. I see nothing prior in /var/log/messages until 10:27 at which point we see the mysql errors start to flow in.

is there another error log I should be looking for?

[kotterbein]

looking now there is a 10 minute gap in /var/log/messages, starting with an queue send error for ndo2db:

2012-08-14T10:17:09.318123-04:00 id-nag01 ndo2db: Error: queue send error, retrying...

2012-08-14T10:17:09.320375-04:00 id-nag01 nagios: SERVICE NOTIFICATION: nagiosadmin;id-ufm01b;SSHD Process;CRITICAL;xi_service_notification_handler;Conn
ection refused by host
2012-08-14T10:17:09.457583-04:00 id-nag01 nagios: SERVICE NOTIFICATION: nagiosadmin;pd-cpm02;Memory Usage;WARNING;xi_service_notification_handler;WARNING - 356 / 24151 MB (1%) Free Memory, Used: 23795 MB, Shared: 0 MB, Buffers: 553 MB, Cached: 15049 MB
2012-08-14T10:27:14.155966-04:00 id-nag01 nscd: nss_ldap: reconnected to LDAP server ldap://11.4.8.81
2012-08-14T10:27:41.313565-04:00 id-nag01 ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_notifications SET instance_id='1', notification_ty
pe='1', notification_reason='0', start_time=FROM_UNIXTIME(1344953061), start_time_usec='148075', end_time=FROM_UNIXTIME(0), end_time_usec='0', object_id
='6056', state='2', output='Connection refused by host', long_output='Connection refused by host', escalated='0', contacts_notified='0' ON DUPLICATE KEY
UPDATE instance_id='1', notification_type='1', notification_reason='0', start_time=FROM_UNIXTIME(1344953061), start_time_usec='148075', end_time=FROM_U
NIXTIME(0), end_time_usec='0', object_id='6056', state='2', output='Connection refused by host', long_output='Connection refused by host', escalated='0'
, contacts_notified='0''
2012-08-14T10:27:41.313584-04:00 id-nag01 ndo2db: mysql_error: 'Server shutdown in progress'
2012-08-14T10:27:41.313764-04:00 id-nag01 ndo2db: Error: mysql_query() failed for 'DELETE FROM nagios_timedevents WHERE instance_id='1' AND scheduled_ti
me<FROM_UNIXTIME(1344868061)'

[scottwilkerson]

The error you are seeing here

Code: Select all

2012-08-14T10:27:41.313565-04:00 id-nag01 ndo2db: Error: mysql_query()

Is only happening because mysql had already initiated the shutdown which was logged at

Code: Select all

2012-08-14T10:27:41.313584-04:00 id-nag01 ndo2db: mysql_error: 'Server shutdown in progress'

To have an occasional line like the following is normal

Code: Select all

2012-08-14T10:17:09.318123-04:00 id-nag01 ndo2db: Error: queue send error, retrying...

If you have a stream of them or a message saying queue items are deleted, that would be a different story.