Page 2 of 2
Re: Delayed Alerts
Posted: Thu Sep 29, 2016 4:34 pm
by highness
rkennedy wrote:Looks like your DB is offloaded, and I believe this issue could be related to timestamps. Could you run the following commands on both the XI machine, and the offloaded DB machine? Replace the SQL credentials as needed for the SQL machine. Some of these may fail, and that's fine. -
Code: Select all
grep "date.timezone" /etc/php.ini
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
date
mysql -uroot-pn@gweb -e "SELECT NOW();"
The database is offloaded, but my previous posting of that information took that into account. Here is the info you asked for: (note that PHP is not installed on the MySQL server)
Code: Select all
NAGIOS SERVER
[email protected] (Linux) $ ./check.sh
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = America/Los_Angeles
-rw-r--r--. 1 root root 2819 May 13 2014 /etc/localtime
Thu Sep 29 14:27:20 PDT 2016
Thu Sep 29 14:27:20 PDT 2016
+---------------------+
| NOW() |
+---------------------+
| 2016-09-29 14:27:20 |
+---------------------+
Code: Select all
MySQL SERVER
[email protected] (Linux) $ ./check.sh
grep: /etc/php.ini: No such file or directory
-rw-r--r--. 1 root root 2819 Feb 14 2013 /etc/localtime
./check.sh: line 3: php: command not found
Thu Sep 29 14:27:20 PDT 2016
+---------------------+
| NOW() |
+---------------------+
| 2016-09-29 14:27:20 |
+---------------------+
Re: Delayed Alerts
Posted: Fri Sep 30, 2016 10:32 am
by rkennedy
I tried to look at the screenshots attached in your profile, but the resolution on them is very small and makes it hard to read any of them. Could you resend these over, so that they can be displayed properly?
At this point, it sounds like the timing we're discussing isn't anything automatic, but rather from the NOC screen vs what Nagios showed under the 'duration' field. Which page in XI are you using that does not update as soon as the host / service does down? I'd like to do some testing on my end. Also, which browser?
Re: Delayed Alerts
Posted: Tue Oct 04, 2016 9:23 am
by highness
After doing some more digging on the box, we're seeing what appears to be an ndo2db issue:
I see this in the logs:
Code: Select all
Oct 4 07:18:59 fe1 ndo2db: Warning: queue send error, retrying...
Oct 4 07:19:00 fe1 ndo2db: Message sent to queue.
Oct 4 07:19:00 fe1 ndo2db: Warning: queue send error, retrying...
Oct 4 07:19:01 fe1 ndo2db: Message sent to queue.
Oct 4 07:19:01 fe1 ndo2db: Warning: queue send error, retrying...
Oct 4 07:19:02 fe1 ndo2db: Message sent to queue.
Oct 4 07:19:02 fe1 ndo2db: Warning: queue send error, retrying...
Oct 4 07:19:03 fe1 ndo2db: Message sent to queue.
Oct 4 07:19:03 fe1 ndo2db: Warning: queue send error, retrying...
When I check message queues I see this:
Code: Select all
------ Message Queues --------
key msqid owner perms used-bytes messages
0x8d000002 884736 nagios 600 262004736 255864
and when I look at the Monitoring Engine Status I see this:
Screen Shot 2016-10-04 at 7.21.41 AM.png
Can we escalate this issue, as we're treading water right now.
Re: Delayed Alerts
Posted: Tue Oct 04, 2016 11:03 am
by tmcdonald
If your kernel message queue is filling up, you are probably running an older version of NDO:
/usr/local/nagios/bin/ndo2db --version
We can attempt an upgrade, but that would best be done in a remote session. Please email
[email protected] with a link back to this thread, and we will see about updating to a newer version of NDO.
Re: Delayed Alerts
Posted: Tue Oct 04, 2016 11:10 am
by highness
tmcdonald wrote:If your kernel message queue is filling up, you are probably running an older version of NDO:
/usr/local/nagios/bin/ndo2db --version
Code: Select all
[email protected] (Linux) $ /usr/local/nagios/bin/ndo2db --version
NDO2DB 2.0.0
Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Copyright (c) 2005-2008 Ethan Galstad
Last Modified: 02-28-2014
License: GPL v2
EDIT: Email sent to customer support
Re: Delayed Alerts
Posted: Tue Oct 04, 2016 1:42 pm
by ssax
Ticket received, locking this topic and we will continue it in the ticket.
Thank you