elasticsearch "rejected execution (queue capacity 1000)" mes

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
li_alm
Posts: 19
Joined: Thu Oct 13, 2016 4:44 am

elasticsearch "rejected execution (queue capacity 1000)" mes

Post by li_alm »

Hello,

We receive a lot of elasticsearch "query rejected" messages in nagios.
E.g.:
2018-06-21 03:50:11,996][DEBUG][action.count ] [04c4efb4-9365-45d3-9c7b-162e3cbcc051] [logstash-2017.07.16][2], node[ac07Vu3JSl6vogmMAT4AYA], [P], s[STARTED]: failed to execute [[[]][], source[{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1529545511000,"to":1529545811000}}},{"fquery":{"query":{"query_string":{"query":"syslog_severity_code:([0 TO 3])"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"syslog_facility:(\"local1\")"}},"_cache":true}}]}}}}}]]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@2de55b4a

I tried to monitor the search queue from the thread pool (for a few days).
I couldn't see anything abnormal (it does not look like the queue capacity is reached).
Indeed, the rejected counter is high.
"search" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 57558, "largest" : 2, "completed" : 164822}

I am using nagios Log Server on one node only:
Nagios Log Server 1.4.4
Elasticsearch 1.6.0
Logstash 1.5.1
Kibana 3.1.1-nagios3

Q: Could someone offer a detailed explanation on the above DEBUG message from the elasticsearch server?

One last note: a quick search on the Nagios Forum indicates a few relevant discussion threads, but these threads are only available for "paid" nagios accounts.

Thank you.
Regards,
Liviu
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: elasticsearch "rejected execution (queue capacity 1000)"

Post by scottwilkerson »

Here is a ES that describes the message

https://discuss.elastic.co/t/rejected-e ... 1000/89954

How much memory and CPU's does this node have?
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
li_alm
Posts: 19
Joined: Thu Oct 13, 2016 4:44 am

Re: elasticsearch "rejected execution (queue capacity 1000)"

Post by li_alm »

@scottwilkerson
Hi, Scott,

Thank you for your reply.
Yes, I also found this link, but I really can't tell which solution applies to us.

I have 2 nagios deployments (completely independent, separate), both behave the same (a lot of "rejected" messages in the logs).
Deployment1:
  • 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
    2 GB RAM
Deployment2:
  • 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
    4 GB RAM
Important question: does this message mean that some messages are not written in the ES database, i.e. some messages received by the nagios system are lost?

Thank you.
Liviu
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: elasticsearch "rejected execution (queue capacity 1000)"

Post by scottwilkerson »

I'm not sure how many messages you are trying to push through each, but this isn't nearly enough memory for a log server deployment, I would recommend 8GB Minimum ideally 32-64GB along with fast disks, like SSD's.
li_alm wrote: Important question: does this message mean that some messages are not written in the ES database, i.e. some messages received by the nagios system are lost?
No, these are searches that cannot be performed, however without increasing the RAM it could be possible messages could be lost at some point.
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
Locked