NagiosXI -> NagiosLog specific type of query broken

[gsl_ops_practice]

That is correct, the output I am getting in the NLS GUI and via the command you supplied are what I am after. I also have a feeling it's a bug somewhere, I hope you would be able to duplicate this behavior in-house.

Can you please advise what are the next steps to get this resolve? This monitoring functionality is now a production requirement for our customer-facing application.

[mcapra]

We were able to replicate this behavior in house. Unfortunately, despite my best efforts, I was not able to find an immediate solution for this particular query that didn't involve revising it. You might try altering the query to say "response AND 200" instead of "response = 200" and see if that produces the same results by querying the Elasticsearch API directly:

Code: Select all

curl -XGET 127.0.0.1:9200/_search?pretty -d '{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1470754104641,"to":1470754404641}}},{"fquery":{"query":{"query_string":{"query":"*apache_access*"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"response AND 200"}},"_cache":true}}],"must_not":[{"fquery":{"query":{"query_string":{"query":"*staging*"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"*10.30*"}},"_cache":true}}]}}}}}'

I filed an internal bug report for this issue (ID 9294).

[gsl_ops_practice]

Thank you for raising an internal bug for this.

Using Response AND 200 unfortunately doesn't work. "Response" is a field that is generated by the apache global filter and the equal sign is the only way to filter out for the response codes that are required for monitoring.

Can you please advise the approximate timeframe for getting this bug fixed?

Thanks,
Alex

[tmcdonald]

Unfortunately bug fixes are in the hands of the developers, and they have their own system for prioritizing fixes. Fortunately, whenever a release is made they make sure to put in the bug or feature request ID in the changelog:

https://assets.nagios.com/downloads/nag ... NGES-5.TXT

So if you keep an eye out for 9294 you should see it, possibly in the next release.

[mcapra]

We don't typically offer timeframes on bugfixes. If it was a simple hot-patch it would be one thing, but this particular issue is pretty nitty-gritty into the elasticsearch parsing scheme.

However, I think we're getting to the bottom of this now. If you want to check the value of a specific field in elasticsearch, the syntax is different than that of a generic query. If you wanted to check the response field via the elasticsearch (or NLS) API, you would be better off using the "term" identifier rather than the "query_string" identifier. For example, this will return all entries with the response field equal to 200:

Code: Select all

{  
   "query":{  
      "constant_score":{  
         "filter":{  
            "term":{  
               "response":200
            }
         }
      }
   }
}

Or from the CLI:

Code: Select all

curl -XGET 127.0.0.1:9200/_search?pretty -d '{"query":{"constant_score":{"filter":{"term":{"response":200}}}}}'

So, replacing the old response = 200 logic, give this a try and see if it produces the results you're looking for. Be mindful of the timestamp in this one since it may be invalid now:

Code: Select all

curl -XGET 127.0.0.1:9200/_search?pretty -d '{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1470754104641,"to":1470754404641}}},{"fquery":{"query":{"query_string":{"query":"*apache_access*"}},"_cache":true}},{"query":{"constant_score":{"filter":{"term":{"response":200}}}}}],"must_not":[{"fquery":{"query":{"query_string":{"query":"*staging*"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"*10.30*"}},"_cache":true}}]}}}}}'

[gsl_ops_practice]

Hello,

The query you posted does now produce a valid reply, please see sample output below. I tried to adapt my NagiosXI query to match the syntax with

Code: Select all

 {"query_string":{"query":"response":200"}}

but that did not produce a valid result in the search. Any other suggestions on how to modify the nagiosxi check?

Code: Select all

      "_index" : "logstash-2016.08.09",
      "_type" : "apache_access",
      "_id" : "NNNNNNNNNNNNNNNNNNNNNNNN",
      "_score" : 1.0,
      "_source":{"message":"111.111.111.111 - - [09/Aug/2016:14:49:50 +0000] \"POST /webservice/webservice HTTP/1.1\" 200 2737 \"-\" \"Java/1.5.0\" inbytes=1939 outbytes=3051\n","@version":"1","@timestamp":"2016-08-09T14:49:50.000Z","type":"apache_access","host":"111.111.111.111","priority":133,"timestamp":["Aug  9 14:49:53","09/Aug/2016:14:49:50 +0000"],"logsource":"webserver","program":"apache_access","severity":5,"facility":16,"facility_label":"local0","severity_label":"Notice","clientip":"111.111.111.111","ident":"-","auth":"-","verb":"POST","request":"/webservice/webservice","httpversion":"1.1","response":200,"bytes":2737,"referrer":"\"-\"","agent":"\"Java/1.5.0\"","inbytes":1939,"outbytes":3051}

[mcapra]

Again, using the query_string identifier isn't going to work here. That logic should be replaced with the term identifier I provided in my last post.

Looking at your first post, the drop-in replacement should look something like this:

Code: Select all

check_xi_service_nagioslogserver!--url='http://1.1.1.1/nagioslogserver/' --apikey='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' --minutes='2' --warn='500' --crit='1000' --query='{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1470754104641,"to":1470754404641}}},{"fquery":{"query":{"query_string":{"query":"*apache_access*"}},"_cache":true}},{"query":{"constant_score":{"filter":{"term":{"response":200}}}}}],"must_not":[{"fquery":{"query":{"query_string":{"query":"*staging*"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"*10.30*"}},"_cache":true}}]}}}}}'

Again being mindful of the timestamps in use here.

[gsl_ops_practice]

Fantastic, this now works. Now for the most important part - how would you structure the query to show transactions where Response NOT = 200?

I tried below but it was throwing the same "Could not get data from NagiosLog server"

Code: Select all

check_xi_service_nagioslogserver!--url='http://1.1.1.1/nagioslogserver/' --apikey='NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN' --minutes='2' --warn='500' --crit='1000' --query='{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1470754104641,"to":1470754404641}}},{"fquery":{"query":{"query_string":{"query":"*apache_access*"}},"_cache":true}},"must_not":[{"query":{"constant_score":{"filter":{"term":{"response":200}}}}}],"must_not":[{"fquery":{"query":{"query_string":{"query":"*staging*"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"*10.30*"}},"_cache":true}}]}}}}}'

We have a more complex query in NagiosLog where we say "Response NOT = 200", "Response NOT = 302", "Response NOT = 304", trying to figure out how we can string that together if at the moment we can't depend on setting up a NagiosLog check with the wizard.

[mcapra]

You would basically apply that same term logic, except instead of putting it in the must block you would put it in the must_not block and alter it for the proper response codes. If you wanted to pull all entries where the response code is not 200, 302, or 304, the query might look like this:

Code: Select all

{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1470754104641,"to":1470754404641}}},{"fquery":{"query":{"query_string":{"query":"*apache_access*"}},"_cache":true}}],"must_not":[{"fquery":{"query":{"query_string":{"query":"*staging*"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"*10.30*"}},"_cache":true}},{"query":{"constant_score":{"filter":{"term":{"response":200}}}}},{"query":{"constant_score":{"filter":{"term":{"response":302}}}}},{"query":{"constant_score":{"filter":{"term":{"response":304}}}}}]}}}}}