Page 2 of 2
Re: mysql hang when hitting "Notifications" tab
Posted: Wed Oct 03, 2012 12:48 pm
by mguthrie
Would you guys be willing to test a fix for this?
Make a backup and replace the following file with the attached file that is zipped.
/usr/local/nagiosxi/html/includes/utils-xmlreports.inc.php
Re: mysql hang when hitting "Notifications" tab
Posted: Fri Oct 05, 2012 3:32 am
by Brian_M
PHP timezone set successfully, same results. MySQL 5.1.61-4.e16, php-mysql 5.3.3-14.e16_3 on my install. Will try the fix in your next post this afternoon and post the results.
Thanks,
Brian
mguthrie wrote:This could be completely unrelated, but can you set your PHP timezone just to rule that out as a possibility.
FAQ is for an unrelated issue, but it has the instructions for setting the timezone.
http://support.nagios.com/wiki/index.ph ... e.22_Error
What versions of mysql are you guys running?
Re: mysql hang when hitting "Notifications" tab
Posted: Fri Oct 05, 2012 3:53 am
by Brian_M
Gave it a shot, query is still hanging. I also tried a database repair with the same results.
-Brian
mguthrie wrote:Would you guys be willing to test a fix for this?
Make a backup and replace the following file with the attached file that is zipped.
/usr/local/nagiosxi/html/includes/utils-xmlreports.inc.php
Re: mysql hang when hitting "Notifications" tab
Posted: Fri Oct 05, 2012 2:10 pm
by mguthrie
Can you post the output you get from running the following script:
Code: Select all
php /usr/local/nagiosxi/cron/dbmaint.php
Re: mysql hang when hitting "Notifications" tab
Posted: Mon Oct 08, 2012 1:26 am
by Brian_M
Code: Select all
[root@ceeprgpmon01 ~]# php /usr/local/nagiosxi/cron/dbmaint.php
CREATING: /usr/local/nagiosxi/var/dbmaint.lock
CLEANING ndoutils TABLE 'commenthistory'...
SQL: DELETE FROM nagios_commenthistory WHERE entry_time < FROM_UNIXTIME(1318141372)
CLEANING ndoutils TABLE 'processevents'...
SQL: DELETE FROM nagios_processevents WHERE event_time < FROM_UNIXTIME(1318141372)
CLEANING ndoutils TABLE 'externalcommands'...
SQL: DELETE FROM nagios_externalcommands WHERE entry_time < FROM_UNIXTIME(1349072572)
CLEANING ndoutils TABLE 'logentries'...
SQL: DELETE FROM nagios_logentries WHERE logentry_time < FROM_UNIXTIME(1341901372)
CLEANING ndoutils TABLE 'notifications'...
SQL: DELETE FROM nagios_notifications WHERE start_time < FROM_UNIXTIME(1341901372)
CLEANING ndoutils TABLE 'contactnotifications'...
SQL: DELETE FROM nagios_contactnotifications WHERE start_time < FROM_UNIXTIME(1341901372)
CLEANING ndoutils TABLE 'contactnotificationmethods'...
SQL: DELETE FROM nagios_contactnotificationmethods WHERE start_time < FROM_UNIXTIME(1341901372)
CLEANING ndoutils TABLE 'statehistory'...
SQL: DELETE FROM nagios_statehistory WHERE state_time < FROM_UNIXTIME(1286605372)
CLEANING ndoutils TABLE 'timedevents'...
SQL: DELETE FROM nagios_timedevents WHERE event_time < FROM_UNIXTIME(1349677072)
CLEANING ndoutils TABLE 'systemcommands'...
SQL: DELETE FROM nagios_systemcommands WHERE start_time < FROM_UNIXTIME(1349677072)
CLEANING ndoutils TABLE 'servicechecks'...
SQL: DELETE FROM nagios_servicechecks WHERE start_time < FROM_UNIXTIME(1349677072)
CLEANING ndoutils TABLE 'hostchecks'...
SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1349677072)
CLEANING ndoutils TABLE 'eventhandlers'...
SQL: DELETE FROM nagios_eventhandlers WHERE start_time < FROM_UNIXTIME(1349677072)
LASTOPT: 1349675401
INTERVAL: 60
NOW: 1349677372
OPTTIME: 1349679001
CLEANING nagiosxi TABLE 'commands'...
SQL: DELETE FROM xi_commands WHERE processing_time < 1349648572::abstime::timestamp without time zone
CLEANING nagiosxi TABLE 'events'...
SQL: DELETE FROM xi_events WHERE processing_time < 1349648572::abstime::timestamp without time zone
SQL1: SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL
SQL2: DELETE FROM xi_meta WHERE meta_id IN (SELECT xi_meta.meta_id FROM xi_meta LEFT JOIN xi_events ON xi_meta.metaobj_id=xi_events.event_id WHERE metatype_id='1' AND event_id IS NULL)
CLEANING nagiosxi TABLE 'auditlog'...
SQL: DELETE FROM xi_auditlog WHERE log_time < 1347085372::abstime::timestamp without time zone
CLEANING nagiosql TABLE 'logbook'...
SQL: DELETE FROM tbl_logbook WHERE time < FROM_UNIXTIME(1349648572)
Repair Complete: Removing Lock File
mguthrie wrote:Can you post the output you get from running the following script:
Code: Select all
php /usr/local/nagiosxi/cron/dbmaint.php
Re: mysql hang when hitting "Notifications" tab
Posted: Mon Oct 08, 2012 9:49 am
by mguthrie
In your Admin->Performance Settings->Database(Tab), how long are you keeping notifications?
So far we haven't been able to recreate this. Can you run the following from your terminal and let us know how many notifications are stored in the backend?
Code: Select all
mysql -pnagiosxi nagios
select count(*) from nagios_notifications;
exit
Re: mysql hang when hitting "Notifications" tab
Posted: Mon Oct 08, 2012 10:15 am
by Brian_M
90 days notification retention.
79,141 count. 0.00 sec
I'm also noticing a few other strange things that have popped up lately. Might need to start up a support ticket to have one of you guys poke around in the system remotely. I've never noticed this particular issue until I read about it in this thread, and have had users tell me of other odd things that used to work properly....don't know when everything started going south.
-Brian
mguthrie wrote:In your Admin->Performance Settings->Database(Tab), how long are you keeping notifications?
So far we haven't been able to recreate this. Can you run the following from your terminal and let us know how many notifications are stored in the backend?
Code: Select all
mysql -pnagiosxi nagios
select count(*) from nagios_notifications;
exit
Re: mysql hang when hitting "Notifications" tab
Posted: Mon Oct 08, 2012 11:34 am
by mguthrie
Ok, that's a substantial amount of notifications to sift through since that query does multiple joins and groupings. I'm betting that's the issue there, so we may need to do some optimization on that query. I might also recommend tuning down how long ndoutils keeps those notifications, to maybe 30 days instead of 90. A much better suggestion would be to look at at what is causing that many notifications, if they're just getting sent to nowhere or getting ignored, I would really recommend re-examining what is done with those. With almost 900 notifications going out per day, that's a lot of extra work for a Nagios server to do if those alerts are going unread. However, if those are legitimate alerts and people are reading and responding to them, then I would just recommend tuning the backend so it doesn't store them as long.
Re: mysql hang when hitting "Notifications" tab
Posted: Thu Oct 11, 2012 2:05 am
by Brian_M
Thanks for the info. I was a bit horrified at the number myself - it makes sense as this is a relatively new install and we're in the process of migrating datacenters (hence a LOT of notifications). We have to keep 90 days worth of logs in the system at all times due to our regional security policy, but the volume of notifications will go down significantly once the migration to the new DC's is complete. I'll change the retention to just a few days (until production is moved to this server) and truncate the logs to limp along in the meantime.
Thanks as always,
Brian
mguthrie wrote:Ok, that's a substantial amount of notifications to sift through since that query does multiple joins and groupings. I'm betting that's the issue there, so we may need to do some optimization on that query. I might also recommend tuning down how long ndoutils keeps those notifications, to maybe 30 days instead of 90. A much better suggestion would be to look at at what is causing that many notifications, if they're just getting sent to nowhere or getting ignored, I would really recommend re-examining what is done with those. With almost 900 notifications going out per day, that's a lot of extra work for a Nagios server to do if those alerts are going unread. However, if those are legitimate alerts and people are reading and responding to them, then I would just recommend tuning the backend so it doesn't store them as long.
Re: mysql hang when hitting "Notifications" tab
Posted: Thu Oct 11, 2012 8:57 am
by mguthrie
OK, if the issue shows up after the DB trim definitely let us know. In the meantime I'll make a note to take a look at that query and see if we can tune it so that it doesn't tank with a large number of notifications.