Nagios Support Forum

Hello,

We receive a lot of elasticsearch "query rejected" messages in nagios.
E.g.:
2018-06-21 03:50:11,996][DEBUG][action.count ] [04c4efb4-9365-45d3-9c7b-162e3cbcc051] [logstash-2017.07.16][2], node[ac07Vu3JSl6vogmMAT4AYA], [P], s[STARTED]: failed to execute [[[]][], source[{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1529545511000,"to":1529545811000}}},{"fquery":{"query":{"query_string":{"query":"syslog_severity_code:([0 TO 3])"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"syslog_facility:(\"local1\")"}},"_cache":true}}]}}}}}]]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@2de55b4a

I tried to monitor the search queue from the thread pool (for a few days).
I couldn't see anything abnormal (it does not look like the queue capacity is reached).
Indeed, the rejected counter is high.
"search" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 57558, "largest" : 2, "completed" : 164822}

I am using nagios Log Server on one node only:
Nagios Log Server 1.4.4
Elasticsearch 1.6.0
Logstash 1.5.1
Kibana 3.1.1-nagios3

Q: Could someone offer a detailed explanation on the above DEBUG message from the elasticsearch server?

One last note: a quick search on the Nagios Forum indicates a few relevant discussion threads, but these threads are only available for "paid" nagios accounts.

Thank you.
Regards,
Liviu

Here is a ES that describes the message

https://discuss.elastic.co/t/rejected-e ... 1000/89954

How much memory and CPU's does this node have?

@scottwilkerson
Hi, Scott,

Thank you for your reply.
Yes, I also found this link, but I really can't tell which solution applies to us.

I have 2 nagios deployments (completely independent, separate), both behave the same (a lot of "rejected" messages in the logs).
Deployment1:

• 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
  2 GB RAM

Deployment2:

• 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
  4 GB RAM

Important question: does this message mean that some messages are not written in the ES database, i.e. some messages received by the nagios system are lost?

Thank you.
Liviu

I'm not sure how many messages you are trying to push through each, but this isn't nearly enough memory for a log server deployment, I would recommend 8GB Minimum ideally 32-64GB along with fast disks, like SSD's.

li_alm wrote: Important question: does this message mean that some messages are not written in the ES database, i.e. some messages received by the nagios system are lost?

No, these are searches that cannot be performed, however without increasing the RAM it could be possible messages could be lost at some point.