Nagios Support Forum • "Logstash Daemon dead but pid file exists"

Page 1 of 2

"Logstash Daemon dead but pid file exists"

Posted: Wed Apr 06, 2016 8:42 am

by Dante

Hi,
I have a stream of log messages of type "Failed password for root from %IP% port %port_number% ssh2".
How can I get a list of values of %IP% which generated > 10 reports like this?
How can I set an alert which will trigger when a new %IP% will match the condition above?

Re: How to group logs by IP inside the message?

Posted: Wed Apr 06, 2016 9:26 am

by hsmith

It's not pretty, but one of my servers has a grok filter I created for this:

Code: Select all

if [type] == 'syslog' {

   grok {

match => [ 'message', 'Failed password for invalid user %{GREEDYDATA:user} +from %{GREEDYDATA:ip} +port %{GREEDYDATA:port} %{GREEDYDATA:service}' ]


match => [ 'message', 'Failed password for %{GREEDYDATA:user} +from %{GREEDYDATA:ip} +port %{GREEDYDATA:port} %{GREEDYDATA:service}' ]


}

}

Please note that using greedydata for everything is not good practice, and I just wanted to be quick.

Dante wrote:How can I get a list of values of %IP% which generated > 10 reports like this?

Once you apply that filter, you can create a chart to do exactly this, here's an example from my SSH failed login dashboard:

NLS SSH Dashboard Example.png

Dante wrote: How can I set an alert which will trigger when a new %IP% will match the condition above?

To my knowledge, there's not a great way to alert only when a new unique value is found in a log. I can consult with my team and get back to you, but I don't think this is going to be an easy requirement to meet.

(also I have no idea why there's a blue marker on China in the map right now, we are in Saint Paul, Minnesota.)

Re: How to group logs by IP inside the message?

Posted: Wed Apr 06, 2016 10:33 am

by Dante

Thank you for the detailed reply.

I was following instructions on "Configuring Input Filters" but I run into problems.

When I got to the "Verify Configuration" stage it showed "OK!" but after the "Apply Configuration", I get the following in my console.

Code: Select all

sudo service logstash status
Logstash Daemon dead but pid file exists

I tried to roll back by deliting the filter that I have created, but logstash is still going down after about a minute and I can't see and logs from hosts.

Code: Select all

$ tail -100 /var/log/logstash/logstash.log
{:timestamp=>"2016-04-04T12:15:42.777000+0100", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}
{:timestamp=>"2016-04-04T12:15:43.243000+0100", :message=>"Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}

These logs seem to be irrelevant since they are from 2 days ago.
Rebooting NLS server didn't help.

How can I prevent logstash from going down?

Re: How to group logs by IP inside the message?

Posted: Wed Apr 06, 2016 10:45 am

by jolson

I'd like the following information from you:

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

Code: Select all

cat /etc/sysconfig/logstash

The above information will help us give you a proper recommendation. Thanks!

Re: How to group logs by IP inside the message?

Posted: Wed Apr 06, 2016 10:58 am

by Dante

Code: Select all

$ cat /usr/local/nagioslogserver/logstash/etc/conf.d/* | grep -v '^$\|^\s*\#'
output {
    elasticsearch {
        cluster => ''
        host => 'localhost'
        index_type => '%{type}'
        node_name => '1e28f93a-63b7-42c1-bca3-d23fcb6eef44'
        protocol => 'http'
        workers => 4
    }
}

Code: Select all

$ cat /etc/sysconfig/logstash | grep -v '^$\|^\s*\#'
APP_DIR=/usr/local/nagioslogserver
LS_HOME="$APP_DIR/logstash"
ES_CLUSTER=$(cat $APP_DIR/var/cluster_uuid)
LS_JAVA_OPTS="-Djava.io.tmpdir=$APP_DIR/tmp"
LS_USER=nagios
LS_GROUP=nagios
LS_CONF_DIR="$LS_HOME/etc/conf.d"
LS_OPTS=" -w 4"
if [ "x$1" == "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" ];then
	GET_LOGSTASH_CONFIG_MESSAGE=$( php /usr/local/nagioslogserver/scripts/get_logstash_config.php )
	GET_LOGSTASH_CONFIG_RETURN=$?
	if [ "$GET_LOGSTASH_CONFIG_RETURN" != "0" ]; then
		echo $GET_LOGSTASH_CONFIG_MESSAGE
		exit 1
	fi
fi

Re: How to group logs by IP inside the message?

Posted: Wed Apr 06, 2016 12:59 pm

by hsmith

There should be more in that first command..

What's output of a ls -l /usr/local/nagioslogserver/logstash/etc/conf.d command?

Those configuration files shouldn't have gone missing..

Re: How to group logs by IP inside the message?

Posted: Thu Apr 07, 2016 3:41 am

by Dante

Here is the full output with comments and empty lines included.

Code: Select all

[ec2-user@log ~]$ ls -l /usr/local/nagioslogserver/logstash/etc/conf.d
total 12
-rw-rw-r-- 1 apache apache 218 Apr  6 16:18 000_inputs.conf
-rw-rw-r-- 1 apache apache 220 Apr  6 16:18 500_filters.conf
-rw-rw-r-- 1 apache apache 493 Apr  6 16:18 999_outputs.conf

Code: Select all

[ec2-user@log ~]$ cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 06 Apr 2016 16:18:48 +0100
#

#
# Global inputs
#



#
# Local inputs
#


# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 06 Apr 2016 16:18:48 +0100
#

#
# Global filters
#



#
# Local filters
#


# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 06 Apr 2016 16:18:48 +0100
#

#
# Required output for Nagios Log Server
#

output {
    elasticsearch {
        cluster => ''
        host => 'localhost'
        index_type => '%{type}'
        node_name => '1e28f93a-63b7-42c1-bca3-d23fcb6eef44'
        protocol => 'http'
        workers => 4
    }
}

#
# Global outputs
#



#
# Local outputs
#

Code: Select all

[ec2-user@log ~]$ cat /etc/sysconfig/logstash
###############################
# Default settings for logstash
###############################

# Override Java location
#JAVACMD=/usr/bin/java

# Set a home directory
APP_DIR=/usr/local/nagioslogserver
LS_HOME="$APP_DIR/logstash"

# set ES_CLUSTER
ES_CLUSTER=$(cat $APP_DIR/var/cluster_uuid)

# Arguments to pass to java
#LS_HEAP_SIZE="256m"
LS_JAVA_OPTS="-Djava.io.tmpdir=$APP_DIR/tmp"

# Logstash filter worker threads
#LS_WORKER_THREADS=1

# pidfiles aren't used for upstart; this is for sysv users.
#LS_PIDFILE=/var/run/logstash.pid

# user id to be invoked as; for upstart: edit /etc/init/logstash.conf
LS_USER=nagios
LS_GROUP=nagios

# logstash logging
#LS_LOG_FILE=/var/log/logstash/logstash.log
#LS_USE_GC_LOGGING="true"

# logstash configuration directory
LS_CONF_DIR="$LS_HOME/etc/conf.d"

# Open file limit; cannot be overridden in upstart
#LS_OPEN_FILES=2048

# Nice level
#LS_NICE=0

# Increate Filter workers to 4 threads
LS_OPTS=" -w 4"

if [ "x$1" == "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" ];then
	GET_LOGSTASH_CONFIG_MESSAGE=$( php /usr/local/nagioslogserver/scripts/get_logstash_config.php )
	GET_LOGSTASH_CONFIG_RETURN=$?
	if [ "$GET_LOGSTASH_CONFIG_RETURN" != "0" ]; then
		echo $GET_LOGSTASH_CONFIG_MESSAGE
		exit 1
	fi

Re: How to group logs by IP inside the message?

Posted: Thu Apr 07, 2016 6:26 am

by Dante

I tried to run the binary to see more output.

Code: Select all

[ec2-user@log logstash]$ sudo bin/logstash --verbose -f etc/conf.d

After a minute I get:

Code: Select all

Pipeline started {:level=>:info}

After another minute I get:

Code: Select all

New Elasticsearch output {:cluster=>"", :host=>"localhost", :port=>"9200", :embedded=>false, :protocol=>"http", :level=>:info}
Automatic template management enabled {:manage_template=>"true", :level=>:info}
Using mapping template {:template=>"{  \"template\" : \"logstash-*\",  \"settings\" : {    \"index.refresh_interval\" : \"5s\"  },  \"mappings\" : {    \"_default_\" : {       \"_all\" : {\"enabled\" : true},       \"dynamic_templates\" : [ {         \"string_fields\" : {           \"match\" : \"*\",           \"match_mapping_type\" : \"string\",           \"mapping\" : {             \"type\" : \"string\", \"index\" : \"analyzed\", \"omit_norms\" : true,               \"fields\" : {                 \"raw\" : {\"type\": \"string\", \"index\" : \"not_analyzed\", \"ignore_above\" : 256}               }           }         }       } ],       \"properties\" : {         \"@version\": { \"type\": \"string\", \"index\": \"not_analyzed\" },         \"geoip\"  : {           \"type\" : \"object\",             \"dynamic\": true,             \"path\": \"full\",             \"properties\" : {               \"location\" : { \"type\" : \"geo_point\" }             }         }       }    }  }}", :level=>:info}
Pipeline shutdown complete. {:level=>:info}
[ec2-user@log logstash]$

One discrepancy that I noticed is that my configuration checks out in the command line.

Code: Select all

[ec2-user@log logstash]$ sudo bin/logstash --verbose -f etc/conf.d --configtest
Configuration OK

But the web interface says "There was an error in the configuration!" when I try to verify configuration.

Code: Select all

Error: Expected one of #, input, filter, output at line 64, column 1 (byte 662) after #
# Logstash Configuration File
...............
#
# Local outputs
#



1

Here line 64 points to the last line with character "1" which is not actually in the configuration file.

Could you please try to reproduce the issue by using the EC2 image that NLS provide?

Re: "Logstash Daemon dead but pid file exists"

Posted: Thu Apr 07, 2016 9:38 am

by hsmith

I don't believe this is specific to the AWS image.

Can I see what's under your Administration > Global > Global Configuration > View > All Files Combined output?

Also, this is the image running 1.4.0, correct?

Re: "Logstash Daemon dead but pid file exists"

Posted: Thu Apr 07, 2016 9:57 am

by Dante

All Files Combined:

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Thu, 07 Apr 2016 15:54:56 +0100
#

#
# Global Configuration
#



#
# Local Configuration
#

Code: Select all

[ec2-user@log logstash]$ sudo bin/logstash -V
logstash 1.4.2