"Warning" when escalated, shows ZERO for "state duration"

[Junaid Shahid]

Hello all,

I have a service definition that is configured to escalate an issue on 5th notification. What we are observing is that when an issue is escalated, the state duration is reset to zero. State duration should be around 5 hours (if notification interval is set to 1 hour). Below is the actual escalated service description. We are running version 4.1.1.

Has anybody seen a similar situation?

=======================================
Notification: PROBLEM [WARNING]
Service: Check /root
Server: wboimprlv712601
Address: 10.12.13.32
Date/Time: 08-08-2016 10:03:52
More Info: https://wdc-nagios.xxxYYY.com/nagios/
Additional Info: DISK WARNING - free space: / 11741 MB /25% inode=98%/:
State Duration: 0 mins.
Service ExecTime: 0.297
Actions:
=======================================

[lmiltchev]

Can you show us the notification command that you are using and the text in the "Service Alert Message" box under "Notification Messages"?

Run the following commands, and show the output:

Code: Select all

uname -a
cat /etc/*release
date
locale

[Junaid Shahid]

Hello,

Well on other hosts I see correct notification command (we are using the default/ vanilla command), screenshot attached. But the affected host shows a red line saying "No notifications have been recorded for this host in the current log file".

Output for the commands is below:

Code: Select all

-bash-4.1$ uname -a
Linux wdcnmonlvm781 3.8.13-118.4.2.el6uek.x86_64 #2 SMP Tue Mar 22 20:47:10 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
-bash-4.1$ cat /etc/*release
Oracle Linux Server release 6.7
NAME="Oracle Linux Server" 
VERSION="6.7" 
ID="ol" 
VERSION_ID="6.7" 
PRETTY_NAME="Oracle Linux Server 6.7" 
ANSI_COLOR="0;31" 
CPE_NAME="cpe:/o:oracle:linux:6:7:server"
HOME_URL="https://linux.oracle.com/" 
BUG_REPORT_URL="https://bugzilla.oracle.com/" 

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 6" 
ORACLE_BUGZILLA_PRODUCT_VERSION=6.7 
ORACLE_SUPPORT_PRODUCT="Oracle Linux" 
ORACLE_SUPPORT_PRODUCT_VERSION=6.7
Red Hat Enterprise Linux Server release 6.7 (Santiago)
Oracle Linux Server release 6.7
-bash-4.1$ 
-bash-4.1$ date
Wed Sep  7 17:02:45 PDT 2016
-bash-4.1$ 
-bash-4.1$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
-bash-4.1$ 

[lmiltchev]

Can you show us how the "notify-service-by-email" command is defined?
Note: you will find it in "/usr/local/nagios/etc/commands.cfg".

[Junaid Shahid]

Here is how "notify-service-by-email" defined:

Code: Select all

define command{
command_name notify-service-by-email
command_line /usr/local/nagios/libexec/nagios_service_mail "$NOTIFICATIONTYPE$" "$HOSTNAME$" "$HOSTALIAS$" "$HOSTSTATE$" "$HOSTADDRESS$" "$SERVICEOUTPUT$" "$SHORTDATETIME$" "$SERVICEDESC$" "$SERVICESTATE$" "$CONTACTEMAIL$" "$SERVICEDURATIONSEC$" "$SERVICEEXECUTIONTIME$" "$TOTALSERVICESWARNING$" "$TOTALSERVICESCRITICAL$" "$TOTALSERVICESUNKNOWN$" "$LASTSERVICEOK$" "$LASTSERVICEWARNING$" "$SERVICENOTIFICATIONNUMBER$"
}

I would also mention here that there are other hosts and service definitions that are using the same notify by email command definition, and they don't exhibit this issue.

[lmiltchev]

As you are using a "non-default" notification handler, we will need to also see the "nagios_service_mail" script.