Alerting setup and not receiving emails

[jworksinc]

Requirement:
Check every minute for an event (new log message), if one occurs send me an email. I want to do this for only [type] = "DELL". This seems like a simple task.

Configuration:
Check Interval = 60s
Loopback Period = 60s
Thresholds = 0 of 1 Events (I've tried 0 of 0 events, 1 of 1 events also)
SMTP config is using an external authenticated smtp server to avoid local config issues / spam / etc.

What's happening?
Shortly after this was setup, I received a few alerts and thought everything was good to go. Suddenly, the alerts stopped and I have yet to receive any for a good 8 hours now, even though about 50 events meeting the criteria set in my query have occurred during this perios.

Is something wrong with configuration?
Is there somewhere I can check to see if an alert was triggered over time, like a log of alerts being fired?

[mcapra]

Is there somewhere I can check to see if an alert was triggered over time, like a log of alerts being fired?

Yup:

2017_05_11_14_03_26_Reports_Nagios_Log_Server.png

You might also check the JOBS type to make sure the run_alerts job is being run regularly. Please share the output of these commands executed from the CLI of your Nagios Log Server machine:

Code: Select all

curl -XGET 'http://localhost:9200/nagioslogserver_log/ALERT/_search?size=50'
curl -XGET 'http://localhost:9200/nagioslogserver_log/JOBS/_search?size=50'
ps aux | grep php

[jworksinc]

Thanks for all the help. The response above provided invaluable tools to figure out why I was not getting alerts. It turned out that the query driving the alerts was changed and thus invalidated.

Long story short, we are parsing data using the logstash CSV input filter. The data itself had a space after each comma and we were using comma as the delimiter. Each field after the first column had a leading space in front of it when it was in elasticsearch. The query was looking for a value with this leading space (created clicking filter icons). Once we fixed the CSV input filter to use comma + space as the delimiter we fixed how the data was being stored, but broke the queries driving the alerts.

Now, I am receiving too many emails. I will start a new thread for this issue.