Page 1 of 1
"Warning" when escalated, shows ZERO for "state duration"
Posted: Wed Sep 07, 2016 12:01 am
by Junaid Shahid
Hello all,
I have a service definition that is configured to escalate an issue on 5th notification. What we are observing is that when an issue is escalated, the state duration is reset to zero. State duration should be around 5 hours (if notification interval is set to 1 hour). Below is the actual escalated service description. We are running version 4.1.1.
Has anybody seen a similar situation?
=======================================
Notification: PROBLEM [WARNING]
Service: Check /root
Server: wboimprlv712601
Address: 10.12.13.32
Date/Time: 08-08-2016 10:03:52
More Info:
https://wdc-nagios.xxxYYY.com/nagios/
Additional Info: DISK WARNING - free space: / 11741 MB /25% inode=98%/:
State Duration: 0 mins.
Service ExecTime: 0.297
Actions:
=======================================
Re: "Warning" when escalated, shows ZERO for "state duration
Posted: Wed Sep 07, 2016 1:13 pm
by lmiltchev
Can you show us the notification command that you are using and the text in the "Service Alert Message" box under "Notification Messages"?
Run the following commands, and show the output:
Code: Select all
uname -a
cat /etc/*release
date
locale
Re: "Warning" when escalated, shows ZERO for "state duration
Posted: Wed Sep 07, 2016 7:03 pm
by Junaid Shahid
Hello,
Well on other hosts I see correct notification command (we are using the default/ vanilla command), screenshot attached. But the affected host shows a red line saying "No notifications have been recorded for this host in the current log file".
Output for the commands is below:
Code: Select all
-bash-4.1$ uname -a
Linux wdcnmonlvm781 3.8.13-118.4.2.el6uek.x86_64 #2 SMP Tue Mar 22 20:47:10 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
-bash-4.1$ cat /etc/*release
Oracle Linux Server release 6.7
NAME="Oracle Linux Server"
VERSION="6.7"
ID="ol"
VERSION_ID="6.7"
PRETTY_NAME="Oracle Linux Server 6.7"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:6:7:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"
ORACLE_BUGZILLA_PRODUCT="Oracle Linux 6"
ORACLE_BUGZILLA_PRODUCT_VERSION=6.7
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=6.7
Red Hat Enterprise Linux Server release 6.7 (Santiago)
Oracle Linux Server release 6.7
-bash-4.1$
-bash-4.1$ date
Wed Sep 7 17:02:45 PDT 2016
-bash-4.1$
-bash-4.1$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
-bash-4.1$
Re: "Warning" when escalated, shows ZERO for "state duration
Posted: Thu Sep 08, 2016 1:50 pm
by lmiltchev
Can you show us how the "notify-service-by-email" command is defined?
Note: you will find it in "/usr/local/nagios/etc/commands.cfg".
Re: "Warning" when escalated, shows ZERO for "state duration
Posted: Fri Sep 09, 2016 9:36 pm
by Junaid Shahid
Here is how "notify-service-by-email" defined:
Code: Select all
define command{
command_name notify-service-by-email
command_line /usr/local/nagios/libexec/nagios_service_mail "$NOTIFICATIONTYPE$" "$HOSTNAME$" "$HOSTALIAS$" "$HOSTSTATE$" "$HOSTADDRESS$" "$SERVICEOUTPUT$" "$SHORTDATETIME$" "$SERVICEDESC$" "$SERVICESTATE$" "$CONTACTEMAIL$" "$SERVICEDURATIONSEC$" "$SERVICEEXECUTIONTIME$" "$TOTALSERVICESWARNING$" "$TOTALSERVICESCRITICAL$" "$TOTALSERVICESUNKNOWN$" "$LASTSERVICEOK$" "$LASTSERVICEWARNING$" "$SERVICENOTIFICATIONNUMBER$"
}
I would also mention here that there are other hosts and service definitions that are using the same notify by email command definition, and they don't exhibit this issue.
Re: "Warning" when escalated, shows ZERO for "state duration
Posted: Mon Sep 12, 2016 1:42 pm
by lmiltchev
As you are using a "non-default" notification handler, we will need to also see the "nagios_service_mail" script.