Logstash service don't start

[achiricozzi]

Hi all,
I have the problem with our cluster log server nagios. The Logstash Collector doesn't work.
This is the logstash file(/var/log/logstash/logstash.log):
{:timestamp=>"2016-04-27T18:26:46.421000+0200", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-04-27T18:26:46.421000+0200", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-04-27T18:26:46.421000+0200", :message=>"retrying failed action with response code: 503", :level=>:warn}

I have:
Nagios Log Server 1.4.0
Elasticsearch 1.6.0
Logstash 1.5.1
Kibana 3.1.1-nagios3
Can you help me, please?
Thanks.

[rkennedy]

Can you post the complete logstash.log file?

Additionally, what kind of resources do you have allocated to this machine? How much data of logs do you have coming in each day?

[achiricozzi]

I have attached the log file.
This is a resource of machine:
top - 14:26:12 up 23:04, 1 user, load average: 0.10, 0.14, 0.10
Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.4%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16459680k total, 15891772k used, 567908k free, 108984k buffers
Swap: 262136k total, 0k used, 262136k free, 6219220k cached
This is a Cluster Statistics:

162,636,389 -- Documents
77.8GB -- Primary Size
77.8GB -- Total Size
2Data -- Instances
1172 -- Total Shards
110 -- Indices
Thanks.

[hsmith]

Can I please see the output of the follow commands?

Code: Select all

service elasticsearch status
curl 'localhost:9200/_cluster/health?level=indices&pretty'
tail /var/log/elaticsearch/<HIT TAB TO AUTOCOMPLETE>.log
free -m

Is this a cluster?

[achiricozzi]

In attached the result.

[hsmith]

A few things I can see from this file,

You're running low on disk space, that watermark error needs to be resolved. Check this document out: http://stackoverflow.com/questions/3336 ... xceeded-on

You have a few indices that are red, you'll have to remove them as they can cause issues.

logstash-2016.04.16
logstash-2016.04.27
logstash-2016.04.26

See if removing these helps.

Code: Select all

curl -XDELETE 'http://localhost:9200/logstash-2016.04.16/'
curl -XDELETE 'http://localhost:9200/logstash-2016.04.26/'
curl -XDELETE 'http://localhost:9200/logstash-2016.04.27/'

[Box293]

This KB article of ours explains how to resolve the watermark issue @hsmith raised.

https://support.nagios.com/kb/article.php?id=469