XI 2011R3.2 Mysql Crash

[arnab.roy]

Hi Guys,

We are having major problems with one of our XI boxes where in whenevr you try to retreive more than couple of days notifications it sends mysql mad and ultimately requires a mysqld restart to bring back to life...its consistently sits at 100% cpu and finally kills the whole system...we have another XI server which runs double the load and half the number of nodes and works happily.

The platform is Centos 6.3

Your MySQL connection id is 184
Server version: 5.1.61 Source distribution

Please help!!!!!!! this system is monitoring a hospital network on a customer site not ideal!

Many Thanks
Arnab

[mguthrie]

Is this when you access the Notifications report from the UI?

[arnab.roy]

Yes. also its talking a long time to generate Availability reports etc...which points to the any sort of large mysql fetches...

[scottwilkerson]

Actually, Availability reports aren't generated with MySQL at all, they are generated strictly through log files...

Can you check to see if you have any errors showing up in

Code: Select all

tail -f /var/log/mysql.log

[arnab.roy]

Hi Scott,

Nope no errors, some how something is happening to the system..it looks like when we run the reports avail.cgi sits and 100% for a while and finally sorts itself out....

[arnab.roy]

Should I attempt a db repair ?

[scottwilkerson]

it looks like when we run the reports avail.cgi sits and 100% for a while and finally sorts itself out....

This is normal for the availability report, especially on a larger system, or over large timeperiods.

How loaded is the system?

Also, for the MySQL problem, could we be hitting a mysql process limit?

Code: Select all

echo "show processlist;"|mysql -pnagiosxi
cat /etc/my.cnf|grep max_connections

[arnab.roy]

The laod avg is sitting at 0.10 when we dont try to do anything on the gui i.e run reports etc...it only goes wrong when running the report...

Code: Select all

Id	User	Host	db	Command	Time	State	Info
2	ndoutils	localhost	nagios	Sleep	2		NULL
3	ndoutils	localhost	nagios	Sleep	23		NULL
4	nagiosql	localhost	nagiosql	Sleep	2		NULL
5	nagiosql	localhost	nagiosql	Sleep	23		NULL
7	ndoutils	localhost	nagios	Sleep	15		NULL
8	nagiosql	localhost	nagiosql	Sleep	15		NULL
9	ndoutils	localhost	nagios	Sleep	45		NULL
10	nagiosql	localhost	nagiosql	Sleep	45		NULL
11	ndoutils	localhost	nagios	Sleep	8		NULL
12	nagiosql	localhost	nagiosql	Sleep	8		NULL
101	ndoutils	localhost	nagios	Sleep	23		NULL
102	nagiosql	localhost	nagiosql	Sleep	23		NULL
103	ndoutils	localhost	nagios	Sleep	8		NULL
104	nagiosql	localhost	nagiosql	Sleep	8		NULL
105	ndoutils	localhost	nagios	Sleep	15		NULL
106	nagiosql	localhost	nagiosql	Sleep	15		NULL
125	ndoutils	localhost	nagios	Sleep	15		NULL
126	nagiosql	localhost	nagiosql	Sleep	15		NULL
139	ndoutils	localhost	nagios	Sleep	38		NULL
140	nagiosql	localhost	nagiosql	Sleep	38		NULL
141	ndoutils	localhost	nagios	Sleep	45		NULL
142	nagiosql	localhost	nagiosql	Sleep	45		NULL
157	ndoutils	localhost	nagios	Sleep	18		NULL
158	nagiosql	localhost	nagiosql	Sleep	18		NULL
213	ndoutils	localhost	nagios	Sleep	1548		NULL
214	ndoutils	localhost	nagios	Sleep	0		NULL
215	ndoutils	localhost	nagios	Sleep	15		NULL
216	nagiosql	localhost	nagiosql	Sleep	15		NULL
219	ndoutils	localhost	nagios	Sleep	8		NULL
220	nagiosql	localhost	nagiosql	Sleep	8		NULL
221	ndoutils	localhost	nagios	Sleep	8		NULL
222	nagiosql	localhost	nagiosql	Sleep	8		NULL
223	ndoutils	localhost	nagios	Sleep	23		NULL
224	nagiosql	localhost	nagiosql	Sleep	23		NULL
1014	ndoutils	localhost	nagios	Sleep	18		NULL
1016	nagiosql	localhost	nagiosql	Sleep	18		NULL
1018	ndoutils	localhost	nagios	Sleep	18		NULL
1019	nagiosql	localhost	nagiosql	Sleep	18		NULL
1020	ndoutils	localhost	nagios	Sleep	18		NULL
1021	nagiosql	localhost	nagiosql	Sleep	18		NULL
1022	ndoutils	localhost	nagios	Sleep	2		NULL
1023	nagiosql	localhost	nagiosql	Sleep	18		NULL
1026	ndoutils	localhost	nagios	Sleep	8		NULL
1027	nagiosql	localhost	nagiosql	Sleep	8		NULL
1032	root	localhost	NULL	Query	0	NULL	show processlist

[arnab.roy]

Their is no cat /etc/my.cnf| grep max_connections max_connections set...

[arnab.roy]

Ok i did raw sql query's on the nagios_notifications tables and it doesnt have any affect on the system ...so must be something within XI which is causing a db lockup.......