Page 1 of 1
Logstash service don't start
Posted: Wed Apr 27, 2016 12:30 pm
by achiricozzi
Hi all,
I have the problem with our cluster log server nagios. The Logstash Collector doesn't work.
This is the logstash file(/var/log/logstash/logstash.log):
{:timestamp=>"2016-04-27T18:26:46.421000+0200", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-04-27T18:26:46.421000+0200", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-04-27T18:26:46.421000+0200", :message=>"retrying failed action with response code: 503", :level=>:warn}
I have:
Nagios Log Server 1.4.0
Elasticsearch 1.6.0
Logstash 1.5.1
Kibana 3.1.1-nagios3
Can you help me, please?
Thanks.
Re: Logstash service don't start
Posted: Wed Apr 27, 2016 2:00 pm
by rkennedy
Can you post the complete logstash.log file?
Additionally, what kind of resources do you have allocated to this machine? How much data of logs do you have coming in each day?
Re: Logstash service don't start
Posted: Thu Apr 28, 2016 8:13 am
by achiricozzi
I have attached the log file.
This is a resource of machine:
top - 14:26:12 up 23:04, 1 user, load average: 0.10, 0.14, 0.10
Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.4%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16459680k total, 15891772k used, 567908k free, 108984k buffers
Swap: 262136k total, 0k used, 262136k free, 6219220k cached
This is a Cluster Statistics:
162,636,389 -- Documents
77.8GB -- Primary Size
77.8GB -- Total Size
2Data -- Instances
1172 -- Total Shards
110 -- Indices
Thanks.
Re: Logstash service don't start
Posted: Thu Apr 28, 2016 9:31 am
by hsmith
Can I please see the output of the follow commands?
Code: Select all
service elasticsearch status
curl 'localhost:9200/_cluster/health?level=indices&pretty'
tail /var/log/elaticsearch/<HIT TAB TO AUTOCOMPLETE>.log
free -m
Is this a cluster?
Re: Logstash service don't start
Posted: Thu Apr 28, 2016 10:03 am
by achiricozzi
In attached the result.
Re: Logstash service don't start
Posted: Thu Apr 28, 2016 11:05 am
by hsmith
A few things I can see from this file,
You're running low on disk space, that watermark error needs to be resolved. Check this document out:
http://stackoverflow.com/questions/3336 ... xceeded-on
You have a few indices that are red, you'll have to remove them as they can cause issues.
logstash-2016.04.16
logstash-2016.04.27
logstash-2016.04.26
See if removing these helps.
Code: Select all
curl -XDELETE 'http://localhost:9200/logstash-2016.04.16/'
curl -XDELETE 'http://localhost:9200/logstash-2016.04.26/'
curl -XDELETE 'http://localhost:9200/logstash-2016.04.27/'
Re: Logstash service don't start
Posted: Thu Apr 28, 2016 7:18 pm
by Box293
This KB article of ours explains how to resolve the watermark issue
@hsmith raised.
https://support.nagios.com/kb/article.php?id=469