logstash stopped

[pccwglobalit]

one of logstash was stopped and there are a lot of messages like this

{:timestamp=>"2016-05-07T04:55:07.527000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.527000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.527000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.527000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.527000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.527000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.528000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.528000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.528000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.528000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.528000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}
{:timestamp=>"2016-05-07T04:55:07.528000+0000", :message=>"retrying failed action with response code: 503", :level=>:warn}

after i restart logstash, the log is coming in again.

1. The logstash status is running. We cannot see two days log. That means we lost two days log?
2. how can we make sure logstash is receiving logs and working?
3. can we have a query in command line to query host log status? as one of host will keep to send log in every seconds and we want to check if the log is sending to logstash.

[hsmith]

1. The logstash status is running. We cannot see two days log. That means we lost two days log?

Possibly. Some syslog daemons will wait to send the logs until the remote host is listening as to not lose anyone. Some will not. You can check the dates logstash was down for to see if those logs populated.

2. how can we make sure logstash is receiving logs and working?

These commands should all give you hints as to what's going on:

Code: Select all

service logstash status
netstat -tl | grep 5544
tcpdump port 5544

3. can we have a query in command line to query host log status? as one of host will keep to send log in every seconds and we want to check if the log is sending to logstash.

I don't fully understand this question. Do you mean can you have an alert or something fire if you don't receive a log from a host every X minutes?

[pccwglobalit]

the logstash daemon is running but it just stopped to process logs. i have one script to check the status of logstash and it shows logstash is running.

for third question, i just want to make sure logstash is receiving logs.

[hsmith]

Let's try to make the following changes on your system:

Go to Administration > Global > Global Configuration

On this screen, expand your Syslog (Default) Input

Delete what is in there, and put this in:

Code: Select all

tcp {
    port => 5544
    type => syslog
  }
  udp {
    port => 5544
    type => syslog
  }

Now on the other side of that screen, create a new filter that looks like this:

Code: Select all

  if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
  }

Apply the configuration, and check to see if you're still witnessing the same behavior.

Sometimes Logstash will drop logs with timestamp formats it doesn't understand. This should work around that.