ndo2db errors - many duplicate key errors

[btmikkelsen]

Nagios XI Version : 2012R2.7
xx 2.6.18-348.18.1.el5 x86_64
CentOS release 5.9 (Final)
Gnome is not installed
Apache Information

PHP Version: 5.1.6
Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Server Port: 80
Date/Time

PHP Timezone: America/Los_Angeles
PHP Time: Sun, 22 Dec 2013 15:55:51 -0800
System Time: Sun, 22 Dec 2013 15:55:51 -0800



having major loging output with ndo2db in the log files. the ndo2db is logging duplicate messages for all services ever time it runs. I'm not sure it's production impacting as the count(*) of nagios_servicestatus is the correct number. I've checked for duplicate nagios processes. Unable to fix this. ndoutils is at 1.5.2. See some references to this on internet searches, but no real solution. Any help?

ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='32204', status_update_time=FROM_UNIXTIME(1387756227), output='OK - 10\.111\.193\.152: rta 0\.361ms, lost 0%', long_output='', perfdata='rta=0\.361ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.617ms;;;; rtmin=0\.198ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1387756222), next_check=FROM_UNIXTIME(1387756522), check_type='0', last_state_change=FROM_UNIXTIME(1387342693), last_hard_state_change=FROM_UNIXTIME(1387342693), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1387756222), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='2.619000', execution_time='0.007560', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_xi_service_ping!3000\.0!80%!5000\.0!100%', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='32204', status_update_time=FROM_UNIXTIME(1387756227), output='OK - 10\.111\.193\.152: rta 0\.361ms, lost 0%', long_output='', perfdata='rta=0\.361ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.617ms;;;; rtmin=0\.198ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1387756222), next_check=FROM_UNIXTIME(1387756522), check_type='0', last_state_change=FROM_UNIXTIME(1387342693), last_hard_state_change=FROM_UNIXTIME(1387342693), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1387756222), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='2.619000', execution_time='0.007560', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_xi_service_ping!3000\.0!80%!5000\.0!100%', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2''
Dec 22 15:50:41 nanm21 ndo2db: mysql_error: 'Duplicate entry '32204' for key 'PRIMARY''

[scottwilkerson]

This is odd, almost like there are duplicate ndo2db instances running or duplicate nagios instances running..

Let run

Code: Select all

service nagios stop
service ndo2db stop
killall -9 ndo2db
killall -9 nagios
service ndo2db start
service nagios start

[btmikkelsen]

I agree with you, Scott. I've tried to do this a few times. I did just restart the server to be 100% sure - same results.

[lmiltchev]

Do you have any errors in the mysqld.log?

Code: Select all

tail -50 /var/log/mysqld.log

Have you tried running the database repair script just to rule out corrupted tables?

Code: Select all

cd /usr/local/nagiosxi/scripts
./repairmysql.sh nagios
./repairmysql.sh nagiosql

[btmikkelsen]

zero errors in the mysqld.log.

Zero errors on tables.
I've even exported the 'nagios' database, dropped it, and re-imported it.
I've also truncated the nagios_servicestate table.
I've also ran a mysqlcheck on the db, and it comes up clean.

I've thought of re-initializing the database, but I don't want to lose the historical data.

[abrist]

It would be curious if the db backup imported back into a reinitialized db cleanly. Though it should not be attempted without a snapshot/db backup first.

[btmikkelsen]

I did verify all the auto increments were reset, so it did import cleanly. I can try to create a new database and see what happens. Although I suspect it will still happen. I'll try that out and let you know. it will be a day or three before I can get to it.

[scottwilkerson]

Sounds Good, let us know if you run into other problems.

[btmikkelsen]

resetting the ndoutils db seems to have fixed it. I suppose we can live without the history. our logentries table was getting a bit to huge, anyway. Thank you for your help.
-b