Page 1 of 1

ndo2db errors - many duplicate key errors

Posted: Sun Dec 22, 2013 6:59 pm
by btmikkelsen
Nagios XI Version : 2012R2.7
xx 2.6.18-348.18.1.el5 x86_64
CentOS release 5.9 (Final)
Gnome is not installed
Apache Information

PHP Version: 5.1.6
Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Server Port: 80
Date/Time

PHP Timezone: America/Los_Angeles
PHP Time: Sun, 22 Dec 2013 15:55:51 -0800
System Time: Sun, 22 Dec 2013 15:55:51 -0800




having major loging output with ndo2db in the log files. the ndo2db is logging duplicate messages for all services ever time it runs. I'm not sure it's production impacting as the count(*) of nagios_servicestatus is the correct number. I've checked for duplicate nagios processes. Unable to fix this. ndoutils is at 1.5.2. See some references to this on internet searches, but no real solution. Any help?

ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='32204', status_update_time=FROM_UNIXTIME(1387756227), output='OK - 10\.111\.193\.152: rta 0\.361ms, lost 0%', long_output='', perfdata='rta=0\.361ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.617ms;;;; rtmin=0\.198ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1387756222), next_check=FROM_UNIXTIME(1387756522), check_type='0', last_state_change=FROM_UNIXTIME(1387342693), last_hard_state_change=FROM_UNIXTIME(1387342693), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1387756222), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='2.619000', execution_time='0.007560', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_xi_service_ping!3000\.0!80%!5000\.0!100%', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='32204', status_update_time=FROM_UNIXTIME(1387756227), output='OK - 10\.111\.193\.152: rta 0\.361ms, lost 0%', long_output='', perfdata='rta=0\.361ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.617ms;;;; rtmin=0\.198ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1387756222), next_check=FROM_UNIXTIME(1387756522), check_type='0', last_state_change=FROM_UNIXTIME(1387342693), last_hard_state_change=FROM_UNIXTIME(1387342693), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1387756222), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='2.619000', execution_time='0.007560', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_xi_service_ping!3000\.0!80%!5000\.0!100%', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2''
Dec 22 15:50:41 nanm21 ndo2db: mysql_error: 'Duplicate entry '32204' for key 'PRIMARY''

Re: ndo2db errors - many duplicate key errors

Posted: Mon Dec 23, 2013 9:11 am
by scottwilkerson
This is odd, almost like there are duplicate ndo2db instances running or duplicate nagios instances running..

Let run

Code: Select all

service nagios stop
service ndo2db stop
killall -9 ndo2db
killall -9 nagios
service ndo2db start
service nagios start

Re: ndo2db errors - many duplicate key errors

Posted: Mon Dec 23, 2013 11:06 am
by btmikkelsen
I agree with you, Scott. I've tried to do this a few times. I did just restart the server to be 100% sure - same results.

Re: ndo2db errors - many duplicate key errors

Posted: Mon Dec 23, 2013 11:53 am
by lmiltchev
Do you have any errors in the mysqld.log?

Code: Select all

tail -50 /var/log/mysqld.log
Have you tried running the database repair script just to rule out corrupted tables?

Code: Select all

cd /usr/local/nagiosxi/scripts
./repairmysql.sh nagios
./repairmysql.sh nagiosql

Re: ndo2db errors - many duplicate key errors

Posted: Mon Dec 23, 2013 3:35 pm
by btmikkelsen
zero errors in the mysqld.log.

Zero errors on tables.
I've even exported the 'nagios' database, dropped it, and re-imported it.
I've also truncated the nagios_servicestate table.
I've also ran a mysqlcheck on the db, and it comes up clean.

I've thought of re-initializing the database, but I don't want to lose the historical data.

Re: ndo2db errors - many duplicate key errors

Posted: Mon Dec 23, 2013 4:10 pm
by abrist
It would be curious if the db backup imported back into a reinitialized db cleanly. Though it should not be attempted without a snapshot/db backup first.

Re: ndo2db errors - many duplicate key errors

Posted: Mon Dec 23, 2013 4:53 pm
by btmikkelsen
I did verify all the auto increments were reset, so it did import cleanly. I can try to create a new database and see what happens. Although I suspect it will still happen. I'll try that out and let you know. it will be a day or three before I can get to it.

Re: ndo2db errors - many duplicate key errors

Posted: Thu Dec 26, 2013 8:17 am
by scottwilkerson
Sounds Good, let us know if you run into other problems.

Re: ndo2db errors - many duplicate key errors

Posted: Tue Dec 31, 2013 1:38 pm
by btmikkelsen
resetting the ndoutils db seems to have fixed it. I suppose we can live without the history. our logentries table was getting a bit to huge, anyway. Thank you for your help.
-b