Logstash service not running. Reboot did not correct

[krobertson71]

Reboot the service did not correct. I can see one error:

The Logstash.log file says it was created on August 25th but contains no data.

This is a service down situation.

[mcapra]

Can you share the full output of:

Code: Select all

/usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d

[krobertson71]

Here is the output. After running this I can see many events now flowing into NLS and filling in the last few days as well. In the GUI it still shows the log stash service as RED.

Code: Select all

Sep 06, 2016 11:13:25 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Logstash startup completed
Sep 06, 2016 11:13:27 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Sep 06, 2016 11:13:27 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Sep 06, 2016 11:13:27 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Sep 06, 2016 11:13:27 AM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Received an event that has a different character encoding than you configured. {:text=>"<13>Sep  4 20:46:39 10.0.101.35 \\xA6\\u0003\\n", :expected_charset=>"UTF-8", :level=>:warn}

[krobertson71]

Also note the we are running a two cluster node. The log stash server on 02 is fine, but 01 is the primary collector.

[mcapra]

That's good news then that logstash isn't encountering any fatal errors.

Can you share the output of the following:

Code: Select all

service logstash restart

As well as the contents of the following file:

Code: Select all

/etc/init.d/logstash

Be sure that the previously started logstash instance is not still running.

[krobertson71]

service logstash restart:

Code: Select all

sudo /etc/init.d/logstash restart
Restarting Logstash Daemon:                                [FAILED]
WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp
                                                           [  OK  ]
[nagios@nagilgp01 nagioslogserver_log]$ Sep 06, 2016 12:16:00 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Sep 06, 2016 12:16:01 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Sep 06, 2016 12:16:01 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Sep 06, 2016 12:16:01 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []
Sep 06, 2016 12:16:01 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [b2733b10-233a-4593-9428-85145cd54c77] loaded [], sites []

/etc/init.d/logstash

Code: Select all

cat /etc/init.d/logstash 
#! /bin/sh
#
#       /etc/rc.d/init.d/logstash
#
#       Starts Logstash as a daemon
#
# chkconfig: 2345 90 10
# description: Starts Logstash as a daemon.

### BEGIN INIT INFO
# Provides: logstash
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: S 0 1 6
# Short-Description: Logstash
# Description: Starts Logstash as a daemon.
### END INIT INFO

. /etc/rc.d/init.d/functions

NAME=logstash
DESC="Logstash Daemon"
DEFAULT=/etc/sysconfig/$NAME

if [ `id -u` -ne 0 ]; then
   echo "You need root privileges to run this script"
   exit 1
fi

# The following variables can be overwritten in $DEFAULT
PATH=/bin:/usr/bin:/sbin:/usr/sbin

# See contents of file named in $DEFAULT for comments
LS_USER=logstash
LS_GROUP=logstash
LS_HOME=/usr/local/nagioslogserver
LS_HEAP_SIZE="500m"
LS_JAVA_OPTS="-Djava.io.tmpdir=${LS_HOME}/tmp"
LS_LOG_FILE=/var/log/logstash/$NAME.log
LS_CONF_DIR=/etc/logstash/conf.d
LS_OPEN_FILES=16384
LS_NICE=19
LS_OPTS=""
LS_PIDFILE=/var/run/$NAME/$NAME.pid
LS_PIDDIR=/var/run/$NAME

# End of variables that can be overwritten in $DEFAULT

if [ -f "$DEFAULT" ]; then
  . "$DEFAULT"
fi

# Define other required variables
PID_FILE=${LS_PIDFILE}

# Make sure the pid directory actually exists (CentOS/RHEL 7)
if [ ! -d $LS_PIDDIR ]; then
   install -d -m 0755 -o $LS_USER -g $LS_USER $LS_PIDDIR
fi

DAEMON="$LS_HOME/bin/logstash"
DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

#
# Function that starts the daemon/service
#
do_start()
{

  if [ -z "$DAEMON" ]; then
    echo "not found - $DAEMON"
    exit 1
  fi

  if pidofproc -p "$PID_FILE" >/dev/null; then
    failure
    exit 99
  fi

  # Prepare environment
  HOME="${HOME:-$LS_HOME}"
  JAVA_OPTS="${LS_JAVA_OPTS}"
  ulimit -n ${LS_OPEN_FILES}
  cd "${LS_HOME}"
  export PATH HOME JAVA_OPTS LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING
  test -n "${JAVACMD}" && export JAVACMD

  nice -n ${LS_NICE} runuser -s /bin/sh -c "exec $DAEMON $DAEMON_OPTS" ${LS_USER} > /dev/null 1>&1 < /dev/null &

  RETVAL=$?
  local PID=$!
  # runuser forks rather than execing our process.
  usleep 500000
  JAVA_PID=$(ps axo ppid,pid | awk -v "ppid=$PID" '$1==ppid {print $2}')
  PID=${JAVA_PID:-$PID}
  echo $PID > $PID_FILE
  chown $LS_USER:$LS_GROUP $PID_FILE
  [ $PID = $JAVA_PID ] && success
}

#
# Function that stops the daemon/service
#
do_stop()
{
    killproc -p $PID_FILE $DAEMON
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f ${PID_FILE}
}

case "$1" in
  start)
    echo -n "Starting $DESC: "
    do_start
    touch /var/run/logstash/$NAME
    ;;
  stop)
    echo -n "Stopping $DESC: "
    do_stop
    rm /var/run/logstash/$NAME
    ;;
  restart|reload)
    echo -n "Restarting $DESC: "
    do_stop
    do_start
    ;;
  status)
    echo -n "$DESC"
    status -p $PID_FILE
    exit $?
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|status|restart}" >&2
    exit 3
    ;;
esac

echo
exit 0

[rkennedy]

What are the permissions on these files?

Code: Select all

ls -la /var/run/logstash/logstash.pid
ls -la /var/run/elasticsearch/elasticsearch.pid

Additionally, what happens if you attempt to load both of these pages from your browser? (replace x.x.x.x with the hostname / IP you're trying to access with).

Code: Select all

http://x.x.x.x/nagioslogserver/api/system/status?subsystem=elasticsearch
http://x.x.x.x/nagioslogserver/api/system/status?subsystem=logstash

Finally, do you have anything in front of the two NLS machines?

[krobertson71]

Actually it is now all green. The api call shows Logstash is up and running. It just took a really long time to "Green Up".

This command you had me run is what started the data flowing in again.

/usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d

I am assuming that was a reinitializing of the logstash service.

Thanks for the help.

[mcapra]

If the aforementioned pid files have incorrect permissions, this would explain why the init.d script is failing. There could also be some math in terms of allocating the memory available to Java that is done within the init.d file.

You can certainly run the logstash binary directly every time (as I detailed in my previous post), but the stock NLS environment is going to try to start logstash via the init.d script every time the system reboots.

[mcapra]

Lets see what the following commands give us:

Code: Select all

service logstash stop

service logstash start