Delayed Alerts

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
highness
Posts: 192
Joined: Thu May 01, 2014 4:25 pm

Re: Delayed Alerts

Post by highness »

rkennedy wrote:Looks like your DB is offloaded, and I believe this issue could be related to timestamps. Could you run the following commands on both the XI machine, and the offloaded DB machine? Replace the SQL credentials as needed for the SQL machine. Some of these may fail, and that's fine. -

Code: Select all

grep "date.timezone" /etc/php.ini
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
date
mysql -uroot-pn@gweb -e "SELECT NOW();"
The database is offloaded, but my previous posting of that information took that into account. Here is the info you asked for: (note that PHP is not installed on the MySQL server)

Code: Select all

NAGIOS SERVER
root@fe1.nagios (Linux) $ ./check.sh
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = America/Los_Angeles
-rw-r--r--. 1 root root 2819 May 13  2014 /etc/localtime
Thu Sep 29 14:27:20 PDT 2016
Thu Sep 29 14:27:20 PDT 2016
+---------------------+
| NOW()               |
+---------------------+
| 2016-09-29 14:27:20 |
+---------------------+

Code: Select all

MySQL SERVER
root@db1.nagios (Linux) $ ./check.sh
grep: /etc/php.ini: No such file or directory
-rw-r--r--. 1 root root 2819 Feb 14  2013 /etc/localtime
./check.sh: line 3: php: command not found
Thu Sep 29 14:27:20 PDT 2016
+---------------------+
| NOW()               |
+---------------------+
| 2016-09-29 14:27:20 |
+---------------------+
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: Delayed Alerts

Post by rkennedy »

I tried to look at the screenshots attached in your profile, but the resolution on them is very small and makes it hard to read any of them. Could you resend these over, so that they can be displayed properly?

At this point, it sounds like the timing we're discussing isn't anything automatic, but rather from the NOC screen vs what Nagios showed under the 'duration' field. Which page in XI are you using that does not update as soon as the host / service does down? I'd like to do some testing on my end. Also, which browser?
Former Nagios Employee
highness
Posts: 192
Joined: Thu May 01, 2014 4:25 pm

Re: Delayed Alerts

Post by highness »

After doing some more digging on the box, we're seeing what appears to be an ndo2db issue:

I see this in the logs:

Code: Select all

Oct  4 07:18:59 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:00 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:00 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:01 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:01 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:02 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:02 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:03 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:03 fe1 ndo2db: Warning: queue send error, retrying...
When I check message queues I see this:

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x8d000002 884736     nagios     600        262004736    255864
and when I look at the Monitoring Engine Status I see this:
Screen Shot 2016-10-04 at 7.21.41 AM.png
Can we escalate this issue, as we're treading water right now.
You do not have the required permissions to view the files attached to this post.
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Delayed Alerts

Post by tmcdonald »

If your kernel message queue is filling up, you are probably running an older version of NDO:

/usr/local/nagios/bin/ndo2db --version

We can attempt an upgrade, but that would best be done in a remote session. Please email customersupport@nagios.com with a link back to this thread, and we will see about updating to a newer version of NDO.
Former Nagios employee
highness
Posts: 192
Joined: Thu May 01, 2014 4:25 pm

Re: Delayed Alerts

Post by highness »

tmcdonald wrote:If your kernel message queue is filling up, you are probably running an older version of NDO:

/usr/local/nagios/bin/ndo2db --version

Code: Select all

root@fe1.nagios (Linux) $ /usr/local/nagios/bin/ndo2db --version

NDO2DB 2.0.0
Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Copyright (c) 2005-2008 Ethan Galstad
Last Modified: 02-28-2014
License: GPL v2
EDIT: Email sent to customer support
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Delayed Alerts

Post by ssax »

Ticket received, locking this topic and we will continue it in the ticket.

Thank you
Locked