Page 1 of 4
XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 3:37 pm
by arnab.roy
Hi Guys,
We are having major problems with one of our XI boxes where in whenevr you try to retreive more than couple of days notifications it sends mysql mad and ultimately requires a mysqld restart to bring back to life...its consistently sits at 100% cpu and finally kills the whole system...we have another XI server which runs double the load and half the number of nodes and works happily.
The platform is Centos 6.3
Your MySQL connection id is 184
Server version: 5.1.61 Source distribution
Please help!!!!!!! this system is monitoring a hospital network on a customer site not ideal!
Many Thanks
Arnab
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 3:49 pm
by mguthrie
Is this when you access the Notifications report from the UI?
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 3:52 pm
by arnab.roy
Yes. also its talking a long time to generate Availability reports etc...which points to the any sort of large mysql fetches...
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 3:58 pm
by scottwilkerson
Actually, Availability reports aren't generated with MySQL at all, they are generated strictly through log files...
Can you check to see if you have any errors showing up in
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 4:02 pm
by arnab.roy
Hi Scott,
Nope no errors, some how something is happening to the system..it looks like when we run the reports avail.cgi sits and 100% for a while and finally sorts itself out....
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 4:04 pm
by arnab.roy
Should I attempt a db repair ?
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 4:09 pm
by scottwilkerson
arnab.roy wrote:it looks like when we run the reports avail.cgi sits and 100% for a while and finally sorts itself out....
This is normal for the availability report, especially on a larger system, or over large timeperiods.
How loaded is the system?
Also, for the MySQL problem, could we be hitting a mysql process limit?
Code: Select all
echo "show processlist;"|mysql -pnagiosxi
cat /etc/my.cnf|grep max_connections
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 4:12 pm
by arnab.roy
The laod avg is sitting at 0.10 when we dont try to do anything on the gui i.e run reports etc...it only goes wrong when running the report...
Code: Select all
Id User Host db Command Time State Info
2 ndoutils localhost nagios Sleep 2 NULL
3 ndoutils localhost nagios Sleep 23 NULL
4 nagiosql localhost nagiosql Sleep 2 NULL
5 nagiosql localhost nagiosql Sleep 23 NULL
7 ndoutils localhost nagios Sleep 15 NULL
8 nagiosql localhost nagiosql Sleep 15 NULL
9 ndoutils localhost nagios Sleep 45 NULL
10 nagiosql localhost nagiosql Sleep 45 NULL
11 ndoutils localhost nagios Sleep 8 NULL
12 nagiosql localhost nagiosql Sleep 8 NULL
101 ndoutils localhost nagios Sleep 23 NULL
102 nagiosql localhost nagiosql Sleep 23 NULL
103 ndoutils localhost nagios Sleep 8 NULL
104 nagiosql localhost nagiosql Sleep 8 NULL
105 ndoutils localhost nagios Sleep 15 NULL
106 nagiosql localhost nagiosql Sleep 15 NULL
125 ndoutils localhost nagios Sleep 15 NULL
126 nagiosql localhost nagiosql Sleep 15 NULL
139 ndoutils localhost nagios Sleep 38 NULL
140 nagiosql localhost nagiosql Sleep 38 NULL
141 ndoutils localhost nagios Sleep 45 NULL
142 nagiosql localhost nagiosql Sleep 45 NULL
157 ndoutils localhost nagios Sleep 18 NULL
158 nagiosql localhost nagiosql Sleep 18 NULL
213 ndoutils localhost nagios Sleep 1548 NULL
214 ndoutils localhost nagios Sleep 0 NULL
215 ndoutils localhost nagios Sleep 15 NULL
216 nagiosql localhost nagiosql Sleep 15 NULL
219 ndoutils localhost nagios Sleep 8 NULL
220 nagiosql localhost nagiosql Sleep 8 NULL
221 ndoutils localhost nagios Sleep 8 NULL
222 nagiosql localhost nagiosql Sleep 8 NULL
223 ndoutils localhost nagios Sleep 23 NULL
224 nagiosql localhost nagiosql Sleep 23 NULL
1014 ndoutils localhost nagios Sleep 18 NULL
1016 nagiosql localhost nagiosql Sleep 18 NULL
1018 ndoutils localhost nagios Sleep 18 NULL
1019 nagiosql localhost nagiosql Sleep 18 NULL
1020 ndoutils localhost nagios Sleep 18 NULL
1021 nagiosql localhost nagiosql Sleep 18 NULL
1022 ndoutils localhost nagios Sleep 2 NULL
1023 nagiosql localhost nagiosql Sleep 18 NULL
1026 ndoutils localhost nagios Sleep 8 NULL
1027 nagiosql localhost nagiosql Sleep 8 NULL
1032 root localhost NULL Query 0 NULL show processlist
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 4:14 pm
by arnab.roy
Their is no cat /etc/my.cnf| grep max_connections max_connections set...
Re: XI 2011R3.2 Mysql Crash
Posted: Fri Oct 19, 2012 4:30 pm
by arnab.roy
Ok i did raw sql query's on the nagios_notifications tables and it doesnt have any affect on the system ...so must be something within XI which is causing a db lockup.......