Page 1 of 3

Email alert: message not appearing

Posted: Mon Dec 09, 2019 12:07 pm
by mbeebe
Hello,

I'm running into an issue with emailed alerts: the "message" is not appearing in the email.

Template:

Code: Select all

<p>%alertname% returned with a <b>%state%</b> state at <b>%time%</b></p>

<p>The alert was processed with the following thresholds:<br>
<ul>
    <li>Lookback period: %lookback%</li>
    <li>Warning: %warning%</li>
    <li>Critical: %critical%</li>
</ul>
</p>

<p>
Here is the full alert output:
<div style="padding: 10px; background-color: #F9F9F9;">%output%</div>
</p>


<p>
Output from log: %last10alertlogs%.</p>

<p>See the last %lookback% in the <a href="%url%">Nagios Log Server dashboard</a>.</p>

<p>Nagios Log Server</p>

Output in email:

Code: Select all

Alertname Alert returned with a CRITICAL state at Mon, 09 Dec 2019 07:03:11 -0800
The alert was processed with the following thresholds:
•	Lookback period: 5m
•	Warning: 0
•	Critical: 0
Here is the full alert output: 
CRITICAL: 1 matching entries found |logs=1;0;0
Output from log: 
@timestamp	2019-12-09T15:02:50.766Z
message	
@version	1
tags	
host	<IP>
port	50818
type	productname_multiline
alert_names	
.
See the last 5m in the Nagios Log Server dashboard.
Nagios Log Server
Actual message body:

Code: Select all

<133>Dec  9 07:04:05 servername productname_multiline: [2019/12/09-07:04:02.134] [FT-ScenarioThread-o3Jlx6ZR-2642304] [INFO] [dip.scenarios.mail.abstract] scenario IS.VALIDATEDWSOURCETABLES#2019-12-09-07-03-37-894 - Expanded message body text to
<133>Dec  9 07:04:05 servername productname_multiline: ****Operations Support Alert****
<133>Dec  9 07:04:05 servername productname_multiline: Tier 1 RED Event from InterSystems ETL process:
<133>Dec  9 07:04:05 servername productname_multiline: Check of XXX Product Tables returned an error. This would typically be caused by issues with XXX Product tables. 
<133>Dec  9 07:04:05 servername productname_multiline: Here are details of the issue(s) found (connection indicates the SQL database and table the SQL table name): 
<133>Dec  9 07:04:05 servername productname_multiline:  Dataset Check Results
<133>Dec  9 07:04:05 servername productname_multiline:  ----------------------------
<133>Dec  9 07:04:05 servername productname_multiline:  Dataset: XXXXX
<133>Dec  9 07:04:05 servername productname_multiline:  > ERROR returned by check 'Stale Data Check' on metric 'Seconds Since Last Updated Record:SQL Data Refresh Check' (32033 (> 3600.0))
<133>Dec  9 07:04:05 servername productname_multiline: Note that an error may be returned for other reasons. See productname Error Details below for information on other types of errors.
<133>Dec  9 07:04:05 servername productname_multiline: Response Time Expectations: Immediate response needed between 6:00AM and 6:00PM, Monday through Friday
<133>Dec  9 07:04:05 servername productname_multiline: Tier 1 Action:
<133>Dec  9 07:04:05 servername productname_multiline: Correct XXX Product issue(s) described above. (No need to re-run process; it will re-run automatically.)
<133>Dec  9 07:04:05 servername productname_multiline: Tier 2 Contact:
<133>Dec  9 07:04:05 servername productname_multiline: xxxx@xxxx
<133>Dec  9 07:04:05 servername productname_multiline: Tier 2 Action:
<133>Dec  9 07:04:05 servername productname_multiline: N/A
<133>Dec  9 07:04:05 servername productname_multiline: --------------------------------------------------------------
<133>Dec  9 07:04:05 servername productname_multiline: productname Error Details:
<133>Dec  9 07:04:05 servername productname_multiline: FAILED	24.0s	Scenario Validate DW Source Tables in IS run as XXXXX
<133>Dec  9 07:04:05 servername productname_multiline: FAILED	3.0s		Step DW Source Tables in scenario Validate DW Source Tables in IS
<133>Dec  9 07:04:05 servername productname_multiline: FAILED	0.0s			Check dataset XXXXX
<133>Dec  9 07:04:05 servername productname_multiline: XXXXX
<133>Dec  9 07:04:05 servername productname_multiline: --------------------------------------------------------------
<133>Dec  9 07:04:05 servername productname_multiline: ****End Operations Support Alert****
NLS version: 2.1.0


Any ideas why we aren't seeing the message in email?

Thanks,

-- Mike Beebe

Re: Email alert: message not appearing

Posted: Mon Dec 09, 2019 1:01 pm
by mbellerue
So if I'm looking at this correctly, the "message" field in the Output Email isn't populated. That, along with the rest of the fields in that code block are populated by the %output% macro from the template.

Can you show a screenshot of the log entry that triggered this alert? I'm wondering if it's just not pulling in the right field.

Re: Email alert: message not appearing

Posted: Mon Dec 09, 2019 1:07 pm
by mbeebe
mbellerue wrote:So if I'm looking at this correctly, the "message" field in the Output Email isn't populated. That, along with the rest of the fields in that code block are populated by the %output% macro from the template.

Can you show a screenshot of the log entry that triggered this alert? I'm wondering if it's just not pulling in the right field.
Is it ok to send you that screen-shot directly?

Thanks,

-- Mike Beebe

Re: Email alert: message not appearing

Posted: Mon Dec 09, 2019 2:19 pm
by mbellerue
Okay, thank you for those! First thing to try would be to go to your template and put your %output% macro in quotes. I'm wondering if the greater than/less than signs are throwing it off.

Re: Email alert: message not appearing

Posted: Tue Dec 10, 2019 12:35 pm
by mbeebe
Hello,

Before I make that change, you could please explain the expected output of these two macros?

%lastalertlog%
The last log from the alert query.

%last10alertlogs%
The last 10 logs from the alert query.


What exactly is meant by "log(s)"?

Thanks,

-- Mikje

Re: Email alert: message not appearing

Posted: Tue Dec 10, 2019 3:41 pm
by mbellerue
By "log" they mean the log entry that is coming in to Log Server. In the Email Templates page, there is a View Macros button which has good descriptions for each of the macros.

Re: Email alert: message not appearing

Posted: Tue Dec 10, 2019 4:32 pm
by mbeebe
mbellerue wrote:By "log" they mean the log entry that is coming in to Log Server. In the Email Templates page, there is a View Macros button which has good descriptions for each of the macros.
Give that explanation, should I use %lastalertlog% as opposed to %last10alertlogs%?

-- Mike Beebe

Re: Email alert: message not appearing

Posted: Tue Dec 10, 2019 4:42 pm
by mbellerue
You can definitely give it a try. Also make sure you only have one of those. Apparently they can't both be used in the same email template.

Re: Email alert: message not appearing

Posted: Fri Dec 13, 2019 10:44 am
by mbeebe
mbellerue wrote:You can definitely give it a try. Also make sure you only have one of those. Apparently they can't both be used in the same email template.
Hello,

I tried using the "lastalert" macro, but had no success.

Is there anything else I should try? Having the message in the email is critical to our monitoring.

Thank you,

-- Mike Beebe

Re: Email alert: message not appearing

Posted: Fri Dec 13, 2019 2:47 pm
by mbellerue
Can you show me a screenshot of one of the logs that is triggering the email, as well as a screenshot of your alert configuration? We should also grab a system profile. You can PM those to me if you want.