Delayed Alerts

[highness]

Looks like your DB is offloaded, and I believe this issue could be related to timestamps. Could you run the following commands on both the XI machine, and the offloaded DB machine? Replace the SQL credentials as needed for the SQL machine. Some of these may fail, and that's fine. -

Code: Select all

grep "date.timezone" /etc/php.ini
ls -l /etc/localtime
php -r 'echo date("D M j G:i:s T Y")."\n";'
date
mysql -uroot-pn@gweb -e "SELECT NOW();"

The database is offloaded, but my previous posting of that information took that into account. Here is the info you asked for: (note that PHP is not installed on the MySQL server)

Code: Select all

NAGIOS SERVER
root@fe1.nagios (Linux) $ ./check.sh
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.timezone
date.timezone = America/Los_Angeles
-rw-r--r--. 1 root root 2819 May 13  2014 /etc/localtime
Thu Sep 29 14:27:20 PDT 2016
Thu Sep 29 14:27:20 PDT 2016
+---------------------+
| NOW()               |
+---------------------+
| 2016-09-29 14:27:20 |
+---------------------+

Code: Select all

MySQL SERVER
root@db1.nagios (Linux) $ ./check.sh
grep: /etc/php.ini: No such file or directory
-rw-r--r--. 1 root root 2819 Feb 14  2013 /etc/localtime
./check.sh: line 3: php: command not found
Thu Sep 29 14:27:20 PDT 2016
+---------------------+
| NOW()               |
+---------------------+
| 2016-09-29 14:27:20 |
+---------------------+

[rkennedy]

I tried to look at the screenshots attached in your profile, but the resolution on them is very small and makes it hard to read any of them. Could you resend these over, so that they can be displayed properly?

At this point, it sounds like the timing we're discussing isn't anything automatic, but rather from the NOC screen vs what Nagios showed under the 'duration' field. Which page in XI are you using that does not update as soon as the host / service does down? I'd like to do some testing on my end. Also, which browser?

[highness]

After doing some more digging on the box, we're seeing what appears to be an ndo2db issue:

I see this in the logs:

Code: Select all

Oct  4 07:18:59 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:00 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:00 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:01 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:01 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:02 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:02 fe1 ndo2db: Warning: queue send error, retrying...
Oct  4 07:19:03 fe1 ndo2db: Message sent to queue.
Oct  4 07:19:03 fe1 ndo2db: Warning: queue send error, retrying...

When I check message queues I see this:

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x8d000002 884736     nagios     600        262004736    255864

and when I look at the Monitoring Engine Status I see this:

Screen Shot 2016-10-04 at 7.21.41 AM.png

Can we escalate this issue, as we're treading water right now.

[tmcdonald]

If your kernel message queue is filling up, you are probably running an older version of NDO:

/usr/local/nagios/bin/ndo2db --version

We can attempt an upgrade, but that would best be done in a remote session. Please email customersupport@nagios.com with a link back to this thread, and we will see about updating to a newer version of NDO.

[highness]

If your kernel message queue is filling up, you are probably running an older version of NDO:

/usr/local/nagios/bin/ndo2db --version

Code: Select all

root@fe1.nagios (Linux) $ /usr/local/nagios/bin/ndo2db --version

NDO2DB 2.0.0
Copyright (c) 2009 Nagios Core Development Team and Community Contributors
Copyright (c) 2005-2008 Ethan Galstad
Last Modified: 02-28-2014
License: GPL v2

EDIT: Email sent to customer support

[ssax]

Ticket received, locking this topic and we will continue it in the ticket.

Thank you