Email alert: message not appearing

[mbeebe]

Hello,

I'm running into an issue with emailed alerts: the "message" is not appearing in the email.

Template:

Code: Select all

<p>%alertname% returned with a <b>%state%</b> state at <b>%time%</b></p>

<p>The alert was processed with the following thresholds:<br>
<ul>
    <li>Lookback period: %lookback%</li>
    <li>Warning: %warning%</li>
    <li>Critical: %critical%</li>
</ul>
</p>

<p>
Here is the full alert output:
<div style="padding: 10px; background-color: #F9F9F9;">%output%</div>
</p>


<p>
Output from log: %last10alertlogs%.</p>

<p>See the last %lookback% in the <a href="%url%">Nagios Log Server dashboard</a>.</p>

<p>Nagios Log Server</p>

Output in email:

Code: Select all

Alertname Alert returned with a CRITICAL state at Mon, 09 Dec 2019 07:03:11 -0800
The alert was processed with the following thresholds:
•	Lookback period: 5m
•	Warning: 0
•	Critical: 0
Here is the full alert output: 
CRITICAL: 1 matching entries found |logs=1;0;0
Output from log: 
@timestamp	2019-12-09T15:02:50.766Z
message	
@version	1
tags	
host	<IP>
port	50818
type	productname_multiline
alert_names	
.
See the last 5m in the Nagios Log Server dashboard.
Nagios Log Server

Actual message body:

Code: Select all

<133>Dec  9 07:04:05 servername productname_multiline: [2019/12/09-07:04:02.134] [FT-ScenarioThread-o3Jlx6ZR-2642304] [INFO] [dip.scenarios.mail.abstract] scenario IS.VALIDATEDWSOURCETABLES#2019-12-09-07-03-37-894 - Expanded message body text to
<133>Dec  9 07:04:05 servername productname_multiline: ****Operations Support Alert****
<133>Dec  9 07:04:05 servername productname_multiline: Tier 1 RED Event from InterSystems ETL process:
<133>Dec  9 07:04:05 servername productname_multiline: Check of XXX Product Tables returned an error. This would typically be caused by issues with XXX Product tables. 
<133>Dec  9 07:04:05 servername productname_multiline: Here are details of the issue(s) found (connection indicates the SQL database and table the SQL table name): 
<133>Dec  9 07:04:05 servername productname_multiline:  Dataset Check Results
<133>Dec  9 07:04:05 servername productname_multiline:  ----------------------------
<133>Dec  9 07:04:05 servername productname_multiline:  Dataset: XXXXX
<133>Dec  9 07:04:05 servername productname_multiline:  > ERROR returned by check 'Stale Data Check' on metric 'Seconds Since Last Updated Record:SQL Data Refresh Check' (32033 (> 3600.0))
<133>Dec  9 07:04:05 servername productname_multiline: Note that an error may be returned for other reasons. See productname Error Details below for information on other types of errors.
<133>Dec  9 07:04:05 servername productname_multiline: Response Time Expectations: Immediate response needed between 6:00AM and 6:00PM, Monday through Friday
<133>Dec  9 07:04:05 servername productname_multiline: Tier 1 Action:
<133>Dec  9 07:04:05 servername productname_multiline: Correct XXX Product issue(s) described above. (No need to re-run process; it will re-run automatically.)
<133>Dec  9 07:04:05 servername productname_multiline: Tier 2 Contact:
<133>Dec  9 07:04:05 servername productname_multiline: xxxx@xxxx
<133>Dec  9 07:04:05 servername productname_multiline: Tier 2 Action:
<133>Dec  9 07:04:05 servername productname_multiline: N/A
<133>Dec  9 07:04:05 servername productname_multiline: --------------------------------------------------------------
<133>Dec  9 07:04:05 servername productname_multiline: productname Error Details:
<133>Dec  9 07:04:05 servername productname_multiline: FAILED	24.0s	Scenario Validate DW Source Tables in IS run as XXXXX
<133>Dec  9 07:04:05 servername productname_multiline: FAILED	3.0s		Step DW Source Tables in scenario Validate DW Source Tables in IS
<133>Dec  9 07:04:05 servername productname_multiline: FAILED	0.0s			Check dataset XXXXX
<133>Dec  9 07:04:05 servername productname_multiline: XXXXX
<133>Dec  9 07:04:05 servername productname_multiline: --------------------------------------------------------------
<133>Dec  9 07:04:05 servername productname_multiline: ****End Operations Support Alert****

NLS version: 2.1.0


Any ideas why we aren't seeing the message in email?

Thanks,

-- Mike Beebe

[mbellerue]

So if I'm looking at this correctly, the "message" field in the Output Email isn't populated. That, along with the rest of the fields in that code block are populated by the %output% macro from the template.

Can you show a screenshot of the log entry that triggered this alert? I'm wondering if it's just not pulling in the right field.

[mbeebe]

So if I'm looking at this correctly, the "message" field in the Output Email isn't populated. That, along with the rest of the fields in that code block are populated by the %output% macro from the template.

Can you show a screenshot of the log entry that triggered this alert? I'm wondering if it's just not pulling in the right field.

Is it ok to send you that screen-shot directly?

Thanks,

-- Mike Beebe

[mbellerue]

Okay, thank you for those! First thing to try would be to go to your template and put your %output% macro in quotes. I'm wondering if the greater than/less than signs are throwing it off.

[mbeebe]

Hello,

Before I make that change, you could please explain the expected output of these two macros?

%lastalertlog%
The last log from the alert query.

%last10alertlogs%
The last 10 logs from the alert query.


What exactly is meant by "log(s)"?

Thanks,

-- Mikje

[mbellerue]

By "log" they mean the log entry that is coming in to Log Server. In the Email Templates page, there is a View Macros button which has good descriptions for each of the macros.

[mbeebe]

By "log" they mean the log entry that is coming in to Log Server. In the Email Templates page, there is a View Macros button which has good descriptions for each of the macros.

Give that explanation, should I use %lastalertlog% as opposed to %last10alertlogs%?

-- Mike Beebe

[mbellerue]

You can definitely give it a try. Also make sure you only have one of those. Apparently they can't both be used in the same email template.

[mbeebe]

You can definitely give it a try. Also make sure you only have one of those. Apparently they can't both be used in the same email template.

Hello,

I tried using the "lastalert" macro, but had no success.

Is there anything else I should try? Having the message in the email is critical to our monitoring.

Thank you,

-- Mike Beebe

[mbellerue]

Can you show me a screenshot of one of the logs that is triggering the email, as well as a screenshot of your alert configuration? We should also grab a system profile. You can PM those to me if you want.