Nagios - Elasticserach

[amitw]

Hi,
We are using ElasticSearch to manage our logs.
We would like run a query on the elasticsearch DB on a specific host and if the query will show s specific parameter, then we would like Nagios to alert.

How can we implement that kind of configuration ?

Thanks

[mcapra]

There's quite a plethora of Elasticsearch based Nagios plugins available in this repository:
https://github.com/HariSekhon/nagios-plugins

I don't think any of them are able to run queries, but they could serve as a skeleton for building out a plugin that does that. Here are the development guidelines if you wanted to build your own Nagios plugin:
https://nagios-plugins.org/doc/guidelines.html

The official Nagios Log Server configuration wizard for Nagios XI can run a query and check the returned record count, but it does this through the Nagios Log Server API and not through the native Elasticsearch API (though they are very similar). However, it could probably be modified to fit this use case:
https://support.nagios.com/kb/article.php?id=75

A link to download this wizard, which will include the plugin for doing this:
https://assets.nagios.com/downloads/nag ... server.zip

And if you're not sure how to use custom plugins with Nagios Core, this guide is a good starting point:
https://www.howtoforge.com/tutorial/wri ... ck-plugin/

Let us know if you have additional questions. This is definitely possible.

[tacolover101]

going with what @mcapra mentioned, a script like this should work for your needs - https://github.com/misiupajor/check_elasticsearch

a suggestion is to look at using NLS for this, as it will make querying off dynamic data much easier.

[cdienger]

Quite a bit of info in the last couple of posts. Thanks @mcapra and @tacolover101! Let us know if you have any further questions @amitw.

[amitw]

Thanks for the info guys.

I'm trying to think of way to query ElasticSeach with HTTP GET, like this:

GET /test-log-YYYY.MM.DD/_count HTTP/1.1
Host: test.kibana.com:9200
User-Agent: curl/7.58.0
Accept: */*
Content-Length: 132
Content-Type: application/x-www-form-urlencoded

{ "query": { "bool": { "must": [{ "match_phrase": { "message": "time out" }}, { "range": { "date": { "gte": "now-15m" } } } ] } } }


HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 59

{"count":0,"_shards":{"total":5,"successful":5,"failed":0}}

where YYYY.MM.DD is the current date? Then it should raise alarm in case the returned count value is greater than zero, and remove alarm if it returns to zero.

It should query ElasticSearch index called test-lot-YYYY.MM.DD for all events added during last 15 minutes and containing text “time out”.
Can Nagios do that?

Thanks

[cdienger]

The best way to create this would be through a query on the dashboards and create an alert on the result.

Under dashboards run a query for:

message:"time out"

and the use the 'create an alert' button (bell in the top right) to create an alert based on queries of the current dashboard.

https://library.nagios.com/library/prod ... og-events/

[mcapra]

Using the plugin provided by @tacolover101, you'd probably need to leverage some sort of wrapper script to get the correct index name based on the current date. Or modify the script itself.

You could also bake a date execution into your Nagios command definition, but you may need to do some clever escaping.

Code: Select all

[root@capra_nag bin]# echo "check_elasticsearch.py --host <host> --index \"logstash-$(date '+%Y.%m.%d')\" --query <query> --warning <warning> --critical <critical>"
check_elasticsearch.py --host <host> --index "logstash-2018.05.07" --query <query> --warning <warning> --critical <critical>

[tmcdonald]

Thanks for the assist, @mcapra and @tacolover!

@amitw, please let us know if you need further assistance.

[amitw]

THANKS ALOT!
will test it and update you accordingly

[tmcdonald]

We'll keep this open for you.