syslog udp listener died

[Box293]

When the server is rebooted, the logstash service fails to start, I see this in the /var/log/logstash/logstash.log:

Code: Select all

{:timestamp=>"2015-07-17T09:32:30.422000+1000", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin.  For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}
{:timestamp=>"2015-07-17T09:32:30.512000+1000", :message=>"Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}
{:timestamp=>"2015-07-17T09:32:41.320000+1000", :message=>"syslog udp listener died", :address=>"0.0.0.0:514", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.java:1521:in `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}

If I start the service in the web interface it remains started and all is well.

I am listening on privileged ports, I have followed both methods in this guide and the problem persists:
https://assets.nagios.com/downloads/nag ... Server.pdf

Code: Select all

cat /etc/sysconfig/logstash
###############################
# Default settings for logstash
###############################

# Override Java location
#JAVACMD=/usr/bin/java

# Set a home directory
APP_DIR=/usr/local/nagioslogserver
LS_HOME="$APP_DIR/logstash"

# set ES_CLUSTER
ES_CLUSTER=$(cat $APP_DIR/var/cluster_uuid)

# Arguments to pass to java
#LS_HEAP_SIZE="256m"
LS_JAVA_OPTS="-Djava.io.tmpdir=$APP_DIR/tmp"

# Logstash filter worker threads
#LS_WORKER_THREADS=1

# pidfiles aren't used for upstart; this is for sysv users.
#LS_PIDFILE=/var/run/logstash.pid

# user id to be invoked as; for upstart: edit /etc/init/logstash.conf
LS_USER=root
LS_GROUP=nagios

# logstash logging
#LS_LOG_FILE=/var/log/logstash/logstash.log
#LS_USE_GC_LOGGING="true"

# logstash configuration directory
LS_CONF_DIR="$LS_HOME/etc/conf.d"

# Open file limit; cannot be overridden in upstart
#LS_OPEN_FILES=2048

# Nice level
#LS_NICE=0

# Increase Filter workers to 4 threads
LS_OPTS=" -w 4"

if [ "x$1" == "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" ];then
	GET_LOGSTASH_CONFIG_MESSAGE=$( php /usr/local/nagioslogserver/scripts/get_logstash_config.php )
	GET_LOGSTASH_CONFIG_RETURN=$?
	if [ "$GET_LOGSTASH_CONFIG_RETURN" != "0" ]; then
		echo $GET_LOGSTASH_CONFIG_MESSAGE
		exit 1
	fi
fi

setcap 'cap_net_bind_service=+ep' $(readlink -f $(which java))

Nagios Log Server 2015R1.4

[ssax]

I found this:

https://github.com/elastic/logstash/issues/1587

It looks like setting _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true in the logstash init script fixed it for a couple people.

[Box293]

In /etc/init.d/logstash I tried this:

Code: Select all

LS_JAVA_OPTS="-Djava.io.tmpdir=${LS_HOME}/tmp"
_JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true

And I get this:

Code: Select all

service logstash start
/etc/init.d/logstash: line 40: _JAVA_OPTIONS:: command not found
Starting Logstash Daemon:                                [  OK  ]

I've tried:

Code: Select all

_JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true"
or
LS_JAVA_OPTS="-Djava.net.preferIPv4Stack=true"
or
LS_JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true"

None of them work, logstash does not start on boot.

Am I defining it correctly in the init script?

[jolson]

If I start the service in the web interface it remains started and all is well.

That is very interesting - when you're starting the process, are you logged in as the root user or as nagios? You will need to start the process as the root user.

Are you certain that no other instances of Logstash are running on the server before you attempt to start it?

Code: Select all

ps -ef | grep logstash

Nothing else listening on UDP 514?

Code: Select all

netstat -na | grep 514

When you start Logstash from the Web GUI, does it start as the root or nagios user? Does it bind to 514 properly?

Code: Select all

ps -ef | grep logstash | grep -v grep | cut -f1 -d' '
netstat -na | grep 514

[ssax]

What if you do this:

Code: Select all

export _JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true"

[Box293]

I can run the command, it appears in the list when I execute export.

However when I reboot the server it no longer appears in the list when I execute export.

This problem only occurs on server boot up. Once the server is booted I can go into Nagios Logserver and start the Logstash Collected from the GUI and it remains started.

[jolson]

However when I reboot the server it no longer appears in the list when I execute export.

You can make a change like this persistent by adding it to your .bashrc file:

Code: Select all

echo 'export _JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true"' >> ~/.bashrc

After you reboot the server, is logstash running in any capacity, i.e., as any user?

[Box293]

Adding it to the .bashrc made it persist through a reboot, however logstash did not start on bootup.

These commands were executed after a reboot:

Code: Select all

[root@lsproduction01 ~]# export
declare -x G_BROKEN_FILENAMES="1"
declare -x HISTCONTROL="ignoredups"
declare -x HISTSIZE="1000"
declare -x HOME="/root"
declare -x HOSTNAME="lsproduction01.box293.local"
declare -x LANG="en_US.UTF-8"
declare -x LESSOPEN="|/usr/bin/lesspipe.sh %s"
declare -x LOGNAME="root"
declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:"
declare -x MAIL="/var/spool/mail/root"
declare -x OLDPWD
declare -x PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"
declare -x PWD="/root"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SSH_CLIENT="10.25.254.50 56472 22"
declare -x SSH_CONNECTION="10.25.254.50 56472 10.25.5.80 22"
declare -x SSH_TTY="/dev/pts/0"
declare -x TERM="xterm"
declare -x USER="root"
declare -x _JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true"

[root@lsproduction01 ~]# service logstash status
Logstash Daemon dead but pid file exists

[root@lsproduction01 ~]# service logstash start
Starting Logstash Daemon:                                  [  OK  ]

[root@lsproduction01 ~]# service logstash status
Logstash Daemon (pid  2090) is running...

Logstash will remain running until the next reboot. Note: after a reboot I can start it at the command line OR through the web interface.

As per the original post, I see the syslog udp listener died error in /var/log/logstash/logstash.log on every boot.

[jolson]

Using your logstash configuration on my own test box (also running CentOS 7) I experienced no issues. I also have a syslog port listening on 514 which starts up appropriately on boot.

The only thing that I can imagine is that either:
1. Port 514 is temporarily being used by some other process on your NLS node when logstash tries to start
2. Logstash tries to start as the 'nagios' user initially.

Let's check your init.d startup files to ensure that they look proper:

Code: Select all

cat /etc/init.d/logstash

Code: Select all

cat /etc/rc.d/init.d/logstash

[Box293]

Using your logstash configuration on my own test box (also running CentOS 7) I experienced no issues. I also have a syslog port listening on 514 which starts up appropriately on boot.

I am on CentOS 6.6.

I upgraded to 2015R2.1 today and this problem has gone away. What I did notice was this in the upgrade:

Upgrading Logstash...
Applying Nagios patches to Logstash...
Restarting Logstash Daemon: [ OK ]
WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
[ OK ]
Logstash upgraded OK
patching file /usr/lib/python2.6/site-packages/curator/curator.py
patching file /usr/lib/python2.6/site-packages/curator/curator.py
Archive: sourceguardian/ixed4.lin.x86-64.zip
inflating: /usr/lib64/php/modules/ixed.5.3.lin
Sourceguardian extension found for PHP version 5.3
Sourceguardian extension already in php.ini
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
Upgrading Kibana...
Kibana upgraded OK
Stopping elasticsearch: [ OK ]
Starting elasticsearch: [ OK ]

Nagios Log Server Upgrade Complete!

Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
You can access the Nagios Log Server web interface by visiting:
http://10.25.5.80/nagioslogserver/

After a reboot it came up OK.

I even removed the entry in .bashrc, rebooted and logstash started on boot.

So it looks like this must have been resolved somehow in the upgrade.