Alerts not working

[joshv]

I have an Alert defined from a global query as follows:

Screen Shot 2017-07-12 at 6.12.05 PM.png

The Global Query is generating timely events, here is one of hundreds that the query returned and should have been alerted on:

Screen Shot 2017-07-12 at 6.04.46 PM.png

However, even the Audit Log report agrees Nagios Log failed to recognize the event and send an alert:

Screen Shot 2017-07-12 at 6.04.03 PM.png

I don't believe any of my Alerts are sending anything. They used to work - for a couple of weeks alerts were sporadic and incomplete, now nothing has alerted for the last 24 hours.

The test SMTP mail does work.

How do I fix this? Where do I begin to look for the problem? Nothing useful in /var/log/messages.

TIA!
-Josh

[cdienger]

Hi Josh,

Please paste the query as seen under the Advanced section of the alert's settings.

I wonder if there maybe an issue with the check interval and lookback period. Try setting it to 5m and and 2h respectfully, verify that there was an alert within the last hour by clicking the "Show alert in Dashboard" button, and then run the alert with the "Run the alert now" button.

[joshv]

Here is the query:

{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1499198133609,"to":1499284533610}}},{"fquery":{"query":{"query_string":{"query":"severity_label: (alert error critical) OR Severity: (alert error critical)"}},"_cache":true}}],"must_not":[{"fquery":{"query":{"query_string":{"query":"EventType:(\"AUDIT_FAILURE\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (5858) AND SourceName: (Microsoft-Windows-WMI-Activity)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"logsource (\"SSCVG1ESX2.safestorz.com\" \"SSSDF1ESX2.safestorz.com\" \"SSCVG1ESX1.safestorz.com\" \"SSSDF1ESX1.safestorz.com\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (57405) AND SourceName: (Microsoft-Windows-Application Server-Applications)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (101) AND SourceName: (Microsoft-Windows-TaskScheduler) AND message: (Adobe Acrobat Update Task)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"facility(10) AND message: (\"pam_fprintd.so\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"facility: (3) AND message: (\"cli_rpc_pipe_open_spnego_ntlmssp\" \"ads_kinit_password\" \"Client not found in Kerberos database\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"facility: (0) AND message: (\"Host SMBus controller not enabled\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"Hostname: (BlackBeth.GRAVITY.local) AND SourceName: (Microsoft-Windows-Backup)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"SourceName: (\"Microsoft-SharePoint Products-SharePoint Foundation\") AND Hostname: (ABCPEDSERV)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (5722) AND SourceName: (NETLOGON) AND message: (Access is denied)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"facility: (21) AND message: (\"No such file OR directory\" \"No such file or directory\" \"Connection refused\" \"/vz/cp/panel/plesk/frameset\" \"Operation timed out\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (7022) AND SourceName: (Service Control Manager) AND message: (Windows Search service)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"Hostname: (ABCPEDSERV) and message: (\"BACKUP failed to complete the command BACKUP DATABASE MBSData\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"SourceName:(\"Microsoft-Windows-CEIP\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"SourceName: (\"VDS Basic Provider\") AND message: (\"Unexpected failure\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (7030) AND message: (\"system is configured to not allow interactive services\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (1000) and message: (\"jdekrnl.dll\" \"jdenet_k.exe\" \"Faulting application name: wmiprvse.exe\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (36888) and message: (TLS protocol defined fatal error code)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"Hostname: (ABCPEDSERV) AND SourceName: (Mastertech MBS RecalcService)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (10) AND SourceName: (Microsoft-Windows-TZSync)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"SourceName:(\"Microsoft-Windows-TerminalServices-Printers\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (7000) AND SourceName:(Service Control Manager) AND message: (UAC File Virtualization service failed to start)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (8193) and message: (Access is denied)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"SourceName:(\"Microsoft-Windows-CodeIntegrity\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (50 56) AND SourceName: (TermDD)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (30804) AND SourceName: (Microsoft-Windows-SMBClient)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"SourceName: (Microsoft-Windows-AppReadiness)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (1003) AND SourceName: (HlpCtntMgr)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (1500) AND SourceName: (SNMP)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (10009) AND SourceName: (VSS)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (7024) AND SourceName:(Service Control Manager) AND message: (SM Agent Update service terminated)"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (1002) AND SourceName: (Application Hang) AND message: (\"Explorer.EXE\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (36) and message: (\"latch register was identical to the interrupt enable and the receive registers\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (7009) and message: (\"waiting for the WMI Performance Adapter service to connect\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (100) and message(\"length 4 too short\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"program: (winbindd) AND message: (\"rpccli_netlogon_set_trust_password\" \"credentials chain check failed\" \"Ticket is ineligible for postdating\" \" cli_rpc_pipe_bind failed with error NT_STATUS_ACCESS_DENIED\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"program: (sshd) AND message: (\"fatal: Read from socket failed\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"logsource: (\"sscvg1cph1\") AND message: (\"IP TEMP BANNED\" \"TOO MANY FAILURES\" \"credentials chain check failed\" \"Cpanel::MailAuth: Failed to lookup domain owner of application.global\" \"Cpanel::MailAuth: Failed to getpwnam\" \"Cpanel::MailAuth: cphulk blocked login for user\" \"auth: Error: Cpanel::MailAuth: Unknown user\" \"PAM service(sshd) ignoring max retries\" \"Brute force detection active: 550 LOGIN DENIED\" \"Auth fail [preauth]\" \"Connection reset by peer [preauth]\" \"Failed to start The PHP FastCGI Process Manager\" \"Failed to lookup domain owner\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"Hostname:(\"BECKETTSERVER01\") AND message: (\"Microsoft Exchange Active Directory Topology service failed\" \"host name not found: nist1.symmetricom.com\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (8194) AND message: (\"Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"ErrorCode: (730054) AND message: (\"om_tcp send failed\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"ErrorCode: (730061) AND message: (\"couldn't connect to tcp socket \")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"ErrorCode: (730060) AND message: (\"couldn't connect to tcp socket\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"SourceName: (\"PowerShell Server\") AND message: (\"SSH connection attempt request denied; maximum connection count exceeded\" \"Error Processing SFTP data\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"facility: (5) AND message:(\"Connection refused (61)\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"facility_label: (kernel) AND message: (\"Assuming drive cache: write through\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"logsource: (\"sscvg1nas1\" \"sscvg2nas1\" \"sssdf1nas1\") AND message:(\"freenasOS.Configuration\" \"freenasOS.Manifest\" \"freenasOS.Update\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"Hostname: (CFI-CFIHQVWS01) and message: (\"could not be registered on the interface with IP address 192.168.1.30\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"EventID: (33) and message: (\"Microsoft Visual Studio\")"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"ErrorCode: (720005) and message: (\"apr_stat failed on file\")"}},"_cache":true}}]}}}}}

[joshv]

Yesterday I deleted my alert and recreated it - NL is sending emails now on some events and skipping others. This is the same behavior I was seeing before the alerts stopped entirely.

Maybe I am misunderstanding something here. My assumption is that, if the record appears in the alert query, it should generate an alert email. NL just seems to be skipping records that meet the query criteria when it is time to send the alerts, even if the very same record appears in the "Show Alerts in Dashboard" test. Is there some minimal criteria that a record has to meet to get sent as an alert - besides just meeting the query criteria?

Regarding a 2 hour lookback test:
Without even setting it any differently, using the "Show Alert In Dashboard" button, I obtain a list of records that should have been alerted on and weren't. My past attempt results are the same, it doesn't matter how many times I have the alert look-back at these records - there are some that just won't alert and some that it alerts on every time.

For example, here is when the alert ran:

Screen Shot 2017-07-14 at 10.30.26 AM.png

And here is the same "Show Alert In Dashboard" query:

Screen Shot 2017-07-14 at 10.27.54 AM.png

Why is NL ignoring all 3 of these records?

[tmcdonald]

I see you have also opened an email ticket for this, so we will be locking this thread and continuing in the email ticket.