mysql hang when hitting "Notifications" tab

[kotterbein]

I have attempted several times to run "notifications" under the reports tab, and each time, it ramps up MYSQL to 100+% (8 core node), and hangs the nagios interface, causing me to have to restart the mysqld service. Is this a known bug? using Nagios XI 2011R3.2.

following error was the only output I recieved the one time I was able to get to the Notifications page:

Code: Select all

SQL: SQL Error [ndoutils] : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') GROUP BY notification_id,contact_object_id,contactnotification_id,contactnotif' at line 20 SQL: SQL Error [ndoutils] : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') GROUP BY notification_id,contact_object_id,contactnotification_id,contactnotif' at line 20

[scottwilkerson]

Are we specifying dates or using the preset periods?

[kotterbein]

I can't even get that far- it dies before even returning an output and the page.

[mguthrie]

It's possible there's DB corruption somewhere causing the problem. We haven't seen this reported in 3.2 or 3.3. I'd try our DB repair procedure and see if it resolves the issue.
http://assets.nagios.com/downloads/nagi ... tabase.pdf

[kotterbein]

Hey Mike-

I ran the db fix, and no luck- still hangs when I try to see the notifications page. I'm assuming there must be either a page bug, or a query issue, so far I cannot isolate it.

[scottwilkerson]

Is this just on the notification Report and no others?

What kind of LOAD is the server under and from click to error approximately how much time passes?

[kotterbein]

server is under a light load, low latency on checks across the board and hardly any cpu utilization.

the ramp up is immediately upon clicking the tab, and is isolated to the Notifications tab.

[scottwilkerson]

We are trying to replicate this but have been unsuccessful in recreating it. You don't by chance have the use of special characters enabled in the nagios.cfg do you?

[Brian_M]

Hello gentlemen,

I can confirm the same behavior on our primary datacenter Xi install. Only way to recover is to restart the mysqld service. I have performed the database repair with no success, but truncating the log entries and notifications then running the repair worked (for a few days). Now I'm back to the notifications query hanging.

System load is very low (averaging around 4% utilization), I/O wait is less than 1%. 24 core server, 12GB RAM, 2x 15k SAS RAID0


===Nagios XI Installation Profile===


===System:====

Nagios XI Version : 2011R3.3
ceeprgpmon01.cee.ema.ad.pwcinternal.com 2.6.32-279.9.1.el6.i686 i686
CentOS release 6.3 (Final)
Gnome Installed

===Apache Information====

PHP Version: 5.3.3
Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0.1
Server Name: 10.43.174.4
Server Address: 10.43.174.4
Server Port: 443

===Date/Time====

PHP Timezone: Not set
PHP Time: Wed, 03 Oct 2012 09:36:16 +0200
System Time: Wed, 03 Oct 2012 09:36:16 +0200

===Nagios XI Data====

nagios (pid 30564) is running...
NPCD running (pid 18589).
ndo2db (pid 1149) is running...
CPU Load 15: 2.15
Total Hosts: 523
Total Services: 4543

===Ping Test localhost====

Running:

/bin/ping -c 3 localhost 2>&1

PING ceeprgpmon01 (127.0.0.1) 56(84) bytes of data.
64 bytes from ceeprgpmon01 (127.0.0.1): icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from ceeprgpmon01 (127.0.0.1): icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from ceeprgpmon01 (127.0.0.1): icmp_seq=3 ttl=64 time=0.032 ms

--- ceeprgpmon01 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.029/0.032/0.035/0.002 ms

===Test wget To locahost====

WGET From URL: http://localhost/nagiosql/index.php
Running:

/usr/bin/wget http://localhost/nagiosql/index.php

--2012-10-03 09:36:18-- http://localhost/nagiosql/index.php
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5259 (5.1K) [text/html]
Saving to: `/tmp/nagiosql_index.tmp'

0K ..... 100% 555M=0s

2012-10-03 09:36:18 (555 MB/s) - `/tmp/nagiosql_index.tmp' saved [5259/5259]

[mguthrie]

This could be completely unrelated, but can you set your PHP timezone just to rule that out as a possibility.

FAQ is for an unrelated issue, but it has the instructions for setting the timezone.
http://support.nagios.com/wiki/index.ph ... e.22_Error

What versions of mysql are you guys running?

Code: Select all

yum list installed | grep mysql