Nagios XI Version : 2012R2.7
xx 2.6.18-348.18.1.el5 x86_64
CentOS release 5.9 (Final)
Gnome is not installed
Apache Information
PHP Version: 5.1.6
Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
Server Port: 80
Date/Time
PHP Timezone: America/Los_Angeles
PHP Time: Sun, 22 Dec 2013 15:55:51 -0800
System Time: Sun, 22 Dec 2013 15:55:51 -0800
having major loging output with ndo2db in the log files. the ndo2db is logging duplicate messages for all services ever time it runs. I'm not sure it's production impacting as the count(*) of nagios_servicestatus is the correct number. I've checked for duplicate nagios processes. Unable to fix this. ndoutils is at 1.5.2. See some references to this on internet searches, but no real solution. Any help?
ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_servicestatus SET instance_id='1', service_object_id='32204', status_update_time=FROM_UNIXTIME(1387756227), output='OK - 10\.111\.193\.152: rta 0\.361ms, lost 0%', long_output='', perfdata='rta=0\.361ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.617ms;;;; rtmin=0\.198ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1387756222), next_check=FROM_UNIXTIME(1387756522), check_type='0', last_state_change=FROM_UNIXTIME(1387342693), last_hard_state_change=FROM_UNIXTIME(1387342693), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1387756222), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='2.619000', execution_time='0.007560', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_xi_service_ping!3000\.0!80%!5000\.0!100%', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2' ON DUPLICATE KEY UPDATE instance_id='1', service_object_id='32204', status_update_time=FROM_UNIXTIME(1387756227), output='OK - 10\.111\.193\.152: rta 0\.361ms, lost 0%', long_output='', perfdata='rta=0\.361ms;3000\.000;5000\.000;0; pl=0%;80;100;; rtmax=0\.617ms;;;; rtmin=0\.198ms;;;;', current_state='0', has_been_checked='1', should_be_scheduled='1', current_check_attempt='1', max_check_attempts='5', last_check=FROM_UNIXTIME(1387756222), next_check=FROM_UNIXTIME(1387756522), check_type='0', last_state_change=FROM_UNIXTIME(1387342693), last_hard_state_change=FROM_UNIXTIME(1387342693), last_hard_state='0', last_time_ok=FROM_UNIXTIME(1387756222), last_time_warning=FROM_UNIXTIME(0), last_time_unknown=FROM_UNIXTIME(0), last_time_critical=FROM_UNIXTIME(0), state_type='1', last_notification=FROM_UNIXTIME(0), next_notification=FROM_UNIXTIME(0), no_more_notifications='0', notifications_enabled='1', problem_has_been_acknowledged='0', acknowledgement_type='0', current_notification_number='0', passive_checks_enabled='1', active_checks_enabled='1', event_handler_enabled='1', flap_detection_enabled='1', is_flapping='0', percent_state_change='0.000000', latency='2.619000', execution_time='0.007560', scheduled_downtime_depth='0', failure_prediction_enabled='1', process_performance_data='1', obsess_over_service='1', modified_service_attributes='0', event_handler='', check_command='check_xi_service_ping!3000\.0!80%!5000\.0!100%', normal_check_interval='5.000000', retry_check_interval='1.000000', check_timeperiod_object_id='2''
Dec 22 15:50:41 nanm21 ndo2db: mysql_error: 'Duplicate entry '32204' for key 'PRIMARY''
ndo2db errors - many duplicate key errors
-
- Posts: 38
- Joined: Wed Feb 23, 2011 10:29 am
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: ndo2db errors - many duplicate key errors
This is odd, almost like there are duplicate ndo2db instances running or duplicate nagios instances running..
Let run
Let run
Code: Select all
service nagios stop
service ndo2db stop
killall -9 ndo2db
killall -9 nagios
service ndo2db start
service nagios start
-
- Posts: 38
- Joined: Wed Feb 23, 2011 10:29 am
Re: ndo2db errors - many duplicate key errors
I agree with you, Scott. I've tried to do this a few times. I did just restart the server to be 100% sure - same results.
Re: ndo2db errors - many duplicate key errors
Do you have any errors in the mysqld.log?
Have you tried running the database repair script just to rule out corrupted tables?
Code: Select all
tail -50 /var/log/mysqld.log
Code: Select all
cd /usr/local/nagiosxi/scripts
./repairmysql.sh nagios
./repairmysql.sh nagiosql
Be sure to check out our Knowledgebase for helpful articles and solutions!
-
- Posts: 38
- Joined: Wed Feb 23, 2011 10:29 am
Re: ndo2db errors - many duplicate key errors
zero errors in the mysqld.log.
Zero errors on tables.
I've even exported the 'nagios' database, dropped it, and re-imported it.
I've also truncated the nagios_servicestate table.
I've also ran a mysqlcheck on the db, and it comes up clean.
I've thought of re-initializing the database, but I don't want to lose the historical data.
Zero errors on tables.
I've even exported the 'nagios' database, dropped it, and re-imported it.
I've also truncated the nagios_servicestate table.
I've also ran a mysqlcheck on the db, and it comes up clean.
I've thought of re-initializing the database, but I don't want to lose the historical data.
Re: ndo2db errors - many duplicate key errors
It would be curious if the db backup imported back into a reinitialized db cleanly. Though it should not be attempted without a snapshot/db backup first.
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
-
- Posts: 38
- Joined: Wed Feb 23, 2011 10:29 am
Re: ndo2db errors - many duplicate key errors
I did verify all the auto increments were reset, so it did import cleanly. I can try to create a new database and see what happens. Although I suspect it will still happen. I'll try that out and let you know. it will be a day or three before I can get to it.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: ndo2db errors - many duplicate key errors
Sounds Good, let us know if you run into other problems.
-
- Posts: 38
- Joined: Wed Feb 23, 2011 10:29 am
Re: ndo2db errors - many duplicate key errors
resetting the ndoutils db seems to have fixed it. I suppose we can live without the history. our logentries table was getting a bit to huge, anyway. Thank you for your help.
-b
-b