Hello,
We receive a lot of elasticsearch "query rejected" messages in nagios.
E.g.:
2018-06-21 03:50:11,996][DEBUG][action.count ] [04c4efb4-9365-45d3-9c7b-162e3cbcc051] [logstash-2017.07.16][2], node[ac07Vu3JSl6vogmMAT4AYA], [P], s[STARTED]: failed to execute [[[]][], source[{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1529545511000,"to":1529545811000}}},{"fquery":{"query":{"query_string":{"query":"syslog_severity_code:([0 TO 3])"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"syslog_facility:(\"local1\")"}},"_cache":true}}]}}}}}]]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@2de55b4a
I tried to monitor the search queue from the thread pool (for a few days).
I couldn't see anything abnormal (it does not look like the queue capacity is reached).
Indeed, the rejected counter is high.
"search" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 57558, "largest" : 2, "completed" : 164822}
I am using nagios Log Server on one node only:
Nagios Log Server 1.4.4
Elasticsearch 1.6.0
Logstash 1.5.1
Kibana 3.1.1-nagios3
Q: Could someone offer a detailed explanation on the above DEBUG message from the elasticsearch server?
One last note: a quick search on the Nagios Forum indicates a few relevant discussion threads, but these threads are only available for "paid" nagios accounts.
Thank you.
Regards,
Liviu
elasticsearch "rejected execution (queue capacity 1000)" mes
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: elasticsearch "rejected execution (queue capacity 1000)"
Here is a ES that describes the message
https://discuss.elastic.co/t/rejected-e ... 1000/89954
How much memory and CPU's does this node have?
https://discuss.elastic.co/t/rejected-e ... 1000/89954
How much memory and CPU's does this node have?
Re: elasticsearch "rejected execution (queue capacity 1000)"
@scottwilkerson
Hi, Scott,
Thank you for your reply.
Yes, I also found this link, but I really can't tell which solution applies to us.
I have 2 nagios deployments (completely independent, separate), both behave the same (a lot of "rejected" messages in the logs).
Deployment1:
Thank you.
Liviu
Hi, Scott,
Thank you for your reply.
Yes, I also found this link, but I really can't tell which solution applies to us.
I have 2 nagios deployments (completely independent, separate), both behave the same (a lot of "rejected" messages in the logs).
Deployment1:
- 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
2 GB RAM
- 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
4 GB RAM
Thank you.
Liviu
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: elasticsearch "rejected execution (queue capacity 1000)"
I'm not sure how many messages you are trying to push through each, but this isn't nearly enough memory for a log server deployment, I would recommend 8GB Minimum ideally 32-64GB along with fast disks, like SSD's.
No, these are searches that cannot be performed, however without increasing the RAM it could be possible messages could be lost at some point.li_alm wrote: Important question: does this message mean that some messages are not written in the ES database, i.e. some messages received by the nagios system are lost?