Dropping action in logs

[CFT6Server]

I am seeing these in our logstash logs. Is this of concern? Are the logs being dropped? I suspect it cannot keep up and dropping?

{:timestamp=>"2016-01-14T09:43:56.583000-0800", :message=>"failed action with response of 400, dropping action:

[jolson]

I suspect it cannot keep up and dropping?

A failing an action with a response of 400 typically means that elasticsearch rejected the insertion request (logstash gives ES events via the API).

A return code of 400 means " The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications." - it's possible that the logstash request was malformed somehow (this can happen if perhaps there is memory corruption/a bug/Elasticsearch was too loaded.

I suspect that is not the case on your server, as I remember the load averages seemed fine. Was there any more to the log that you posted, or is that the end of it? Typically Logstash will report a specific event and give an idea as to why it was dropped. It would be useful to see more of that log if possible.

To cover our bases, I'd like to see the output of the following commands:

Code: Select all

curl 'localhost:9200/_cat/nodes?v'
curl -XGET localhost:9200/_nodes/jvm?pretty
top | head -n5
free -m
df -h
df -i