Page 1 of 1
Alerts thresholds queries not updating?
Posted: Thu Mar 24, 2016 1:32 pm
by Jklre
We are seeing some issues of alerts and thresholds not updating properly.
We will take our existing alert and change a threshold from 0 to 1 or whatever and save it through the web interface. This is effecting one of our nodes which happens to be the one with the most alerts on it. (Around 2000 or so)
We also notice this on another node but when editing the query in the alert via the web interface.
We started with this query:
{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1458843407609,"to":1458843707611}}},{"fquery":{"query":{"query_string":{"query":"
servicename
\"ENTERPRISE-DOCSTORENAS-SERVICE\")"}},"_cache":true}}]}}}}}
We changed it withing the alert to:
{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1458843407609,"to":1458843707611}}},{"fquery":{"query":{"query_string":{"query":"
message\"ENTERPRISE-DOCSTORENAS-SERVICE\")"}},"_cache":true}}]}}}}}
After clicking on the view alert button on the view alerts page it displays the incorrect filter on the dashboard. Viewing the alert after updating shows the proper query its just viewing the alert dashboard that does not display the proper filters. Do we know if the alert is running with the proper query or is the view alert not functioning properly?
Thanks.
Re: Alerts thresholds queries not updating?
Posted: Thu Mar 24, 2016 4:33 pm
by jolson
I was capable of modifying my query and the alert responded accordingly.
Old query:
Code: Select all
{ "query": { "filtered": { "query": { "bool": { "should": [ { "query_string": { "query": "*" } } ] } }, "filter": { "bool": { "must": [ { "range": { "@timestamp": { "from": 1412797557683, "to": 1412883957683 } } }, { "fquery": { "query": { "query_string": { "query": "severity_label: (alert error critical) OR Severity: (alert error critical)" } }, "_cache": true } } ] } } } } }
New query:
Code: Select all
{ "query": { "filtered": { "query": { "bool": { "should": [ { "query_string": { "query": "*" } } ] } }, "filter": { "bool": { "must": [ { "range": { "@timestamp": { "from": 1412797557683, "to": 1412883957683 } } }, { "fquery": { "query": { "query_string": { "query": "severity_label: (yrdyd) OR Severity: (yrdyd)" } }, "_cache": true } } ] } } } } }
After the new query was put in place, the alert returned 0 results and did not email me. This tells me that the alert system is definitely responding to query changes appropriately. When I click through to the dashboard however, the old query is displayed, meaning that the click-through button on the alerts page doesn't respond to manual query edits. I will make a bug report that includes my findings.
Regarding your first question:
We will take our existing alert and change a threshold from 0 to 1 or whatever and save it through the web interface. This is effecting one of our nodes which happens to be the one with the most alerts on it. (Around 2000 or so)
Could you please elaborate? I'm not quite sure how to replicate this bug.
Re: Alerts thresholds queries not updating?
Posted: Thu Mar 24, 2016 4:47 pm
by Jklre
Could you please elaborate? I'm not quite sure how to replicate this bug.
I just went through and played around with it some more. There are some alerts that let us modify them and others that do not. Seems pretty random with the ones that are effected.
One that is not working consistently is as follows. We even deleted the alert and re-created a new one with the same properties and the behavior returned. This one does not allow us to change anything in it. not the name /threshold / query etc.. If we need to make a change we currently need to delete it and re create it.
ID: AVOGYoS3PlBiE1Llqurx
Alertname: 106929_FATAL_PARTIALLOSS.STANDARD_ASSIGNMENT_DELIVERY
Check Interval: 60m
Lookback Period: 60m
Thresholds: 10 10
Alert Method: Email
{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1458247617213,"to":1458248517215}}},{"fquery":{"query":{"query_string":{"query":"Mitch_ErrorNum:(\"106929\")"}},"_cache":true}}],"must_not":[{"fquery":{"query":{"query_string":{"query":"Mitch_Message:(\"%%A1%%%%\" OR \"%%AC%%%%\" OR \"%%DT%%%%\" OR \"%%EM%%%%\" OR \"%%HI%%%%\" OR \"%%MD%%%%\" \"%%MU%%%%\" OR \"%%MY%%%%\" OR \"%%NO%%%%\" OR \"%%OA%%%%\" OR \"%%OK%%%%\" OR \"%%PL%%%%\" OR \"%%PN%%%%\" OR \"%%RB%%%%\" OR \"%%SF%%%%\" OR \"%%SR%%%%\" OR \"%%WN%%%%\")"}},"_cache":true}}]}}}}}
Is there any other information I can provide?
Re: Alerts thresholds queries not updating?
Posted: Fri Mar 25, 2016 9:56 am
by hsmith
Do you have a query saved for each alert? If so, what happens if you modify the saved query?
Re: Alerts thresholds queries not updating?
Posted: Fri Mar 25, 2016 11:14 am
by Jklre
hsmith wrote:Do you have a query saved for each alert? If so, what happens if you modify the saved query?
When modifying the query and saving nothing happens. it just reverts to what it was before. I can delete the alert how ever.
Re: Alerts thresholds queries not updating?
Posted: Fri Mar 25, 2016 12:59 pm
by hsmith
You know the remote drill. I would like to see this one live. Can you send in a ticket, and then we can get something scheduled?
Re: Alerts thresholds queries not updating?
Posted: Mon Mar 28, 2016 10:58 am
by Jklre
hsmith wrote:You know the remote drill. I would like to see this one live. Can you send in a ticket, and then we can get something scheduled?
Will do Ill open up a case and we can schedule some time. thank you.
Re: Alerts thresholds queries not updating?
Posted: Mon Mar 28, 2016 11:27 am
by hsmith
Sounds good. Thanks!