Cisco Switch Logs

[ssavaso]

Log stash file:
###############################
# Default settings for logstash
###############################

# Override Java location
#JAVACMD=/usr/bin/java

# Set a home directory
APP_DIR=/usr/local/nagioslogserver
LS_HOME="$APP_DIR/logstash"

# set ES_CLUSTER
ES_CLUSTER=$(cat $APP_DIR/var/cluster_uuid)

# Arguments to pass to java
#LS_HEAP_SIZE="256m"
LS_JAVA_OPTS="-Djava.io.tmpdir=$APP_DIR/tmp"

# Logstash filter worker threads
#LS_WORKER_THREADS=1

# pidfiles aren't used for upstart; this is for sysv users.
#LS_PIDFILE=/var/run/logstash.pid

# user id to be invoked as; for upstart: edit /etc/init/logstash.conf
LS_USER=root
LS_GROUP=nagios

# logstash logging
#LS_LOG_FILE=/var/log/logstash/logstash.log
#LS_USE_GC_LOGGING="true"

# logstash configuration directory
LS_CONF_DIR="$LS_HOME/etc/conf.d"

# Open file limit; cannot be overridden in upstart
#LS_OPEN_FILES=2048

# Nice level
#LS_NICE=0

# Increase Filter workers to 4 threads
LS_OPTS=" -w 4"

if [ "x$1" == "xstart" -o "x$1" == "xrestart" -o "x$1" == "xreload" ];then
GET_LOGSTASH_CONFIG_MESSAGE=$( php /usr/local/nagioslogserver/scripts/get_logstash_config.php )
GET_LOGSTASH_CONFIG_RETURN=$?
if [ "$GET_LOGSTASH_CONFIG_RETURN" != "0" ]; then
echo $GET_LOGSTASH_CONFIG_MESSAGE
exit 1
fi
fi

setcap 'cap_net_bind_service=+ep' $(readlink -f $(which java))
[root@localhost ~]#

Permissions:
[root@localhost ~]# ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/
total 12
-rw-r--r-- 1 nagios users 294 Apr 1 13:08 000_inputs.conf
-rw-r--r-- 1 nagios users 978 Apr 1 13:08 500_filters.conf
-rw-r--r-- 1 nagios users 534 Apr 1 13:08 999_outputs.conf
[root@localhost ~]#

[jolson]

Please try the following:

First, we'll turn logstash off on the CLI:

Code: Select all

service logstash stop

Next, we'll purge any configuration files that logstash has:

Code: Select all

rm -f /usr/local/nagioslogserver/logstash/etc/conf.d/*.conf

Now, we'll go to the webGUI. Please remove any local configurations that you may have - ensure that the only configs you have defined are in 'global configuration'. Double check the syntax and ensure that there aren't any duplicate ports.

Now, let's start a tail on your logstash.log file:

Code: Select all

tail -f /var/log/logstash/logstash.log

After you have started this tail, please Reset your jobs (Administration -> Command Subsystem -> Reset All Jobs) and Apply Configuration.

Once the configuration has been applied, please verify that your logstash config files were added to your node properly (you may need to open a second shell, since you have one that is using the tail -f command):

Code: Select all

ls -l /usr/local/nagioslogserver/logstash/etc/conf.d/
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

Verify that the configs match the configs in your GUI. Restart logstash:

Code: Select all

service logstash start

This procedure should hopefully correct the problems we're having. The logstash tail will give us any errors that logstash might encounter.

[ssavaso]

The logstash will no longer stay running and nothing reported from the logstash tail other then the message below.

[root@localhost ~]# tail -f /var/log/logstash/logstash.log
{:timestamp=>"2015-04-01T13:08:15.590000-0600", :message=>"Using milestone 1 input plugin 'syslog'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.4.2/plugin-milestones", :level=>:warn}

[jolson]

ssavaso, I believe that this is a bug. It appears to occur when you replicate a global configuration in local configuration. The symptom is that 'Apply Configuration' stops writing Global Configurations. I am troubleshooting this bug in-house and have reported it to our developers - I'll write back once I have more details about what might be happening. You can either try and troubleshoot it on your side, or re-make your Nagios Log Server box. Thank you!

[ssavaso]

I really don't think that is the problem. Try to delete the default syslog source then added it back then I accidentally added another syslog source with port 514. That is when I think the system broke. I could not ever delete the duplicate syslog source and the input config file was blank from there out. In order to get logs to collect I added a local instance and then the logs started to collect but if you delete that local input it stops collecting logs and will not allow logstash to continue running. To add two inputs with the same syslog's with the same port should not be allowed. The system should catch that and or let you delete it. I even tried to blank out the duplicate syslog input and it still would not let me delete it.

[jolson]

I have just verified in my lab that Globally defined inputs can stop being pushed when the default syslog input is removed from the system. I am talking to a developer about this right now - I'll report back with what I find out.

[jolson]

We have fixed this bug, and the fix will be included in the next release of Nagios Log Server. Due to the nature of this bug, that release will be coming soon - I will update this thread once more when the release is official.

[jolson]

The latest version of Nagios Log Server has been released. Please download the latest version here: http://library.nagios.com/library/produ ... /downloads

[rpaulmorris]

I am having the same issue my logstash log contains these entries.
[root@localhost ~]# tail -f /var/log/logstash/logstash.log
{:timestamp=>"2015-10-02T14:44:38.585000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}
{:timestamp=>"2015-10-02T14:46:34.282000+0000", :message=>"You are using a deprecated config setting \"charset\" set in tcp. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Use the codec setting instead. For example: input { tcp { codec => plain { charset => \"UTF-8\" } } If you have any questions about this, please visit the #logstash channel on freenode irc.", :name=>"charset", :plugin=><LogStash::Inputs::Tcp --->, :level=>:warn}
{:timestamp=>"2015-10-02T14:46:37.375000+0000", :message=>"Could not start TCP server: Address in use", :host=>"0.0.0.0", :port=>5544, :level=>:error}
{:timestamp=>"2015-10-02T14:46:37.393000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}
{:timestamp=>"2015-10-02T14:49:01.684000+0000", :message=>"You are using a deprecated config setting \"charset\" set in tcp. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Use the codec setting instead. For example: input { tcp { codec => plain { charset => \"UTF-8\" } } If you have any questions about this, please visit the #logstash channel on freenode irc.", :name=>"charset", :plugin=><LogStash::Inputs::Tcp --->, :level=>:warn}
{:timestamp=>"2015-10-02T14:49:05.712000+0000", :message=>"Could not start TCP server: Address in use", :host=>"0.0.0.0", :port=>5544, :level=>:error}
{:timestamp=>"2015-10-02T14:49:05.714000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}
{:timestamp=>"2015-10-02T14:49:31.465000+0000", :message=>"You are using a deprecated config setting \"charset\" set in tcp. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Use the codec setting instead. For example: input { tcp { codec => plain { charset => \"UTF-8\" } } If you have any questions about this, please visit the #logstash channel on freenode irc.", :name=>"charset", :plugin=><LogStash::Inputs::Tcp --->, :level=>:warn}
{:timestamp=>"2015-10-02T14:49:33.553000+0000", :message=>"Could not start TCP server: Address in use", :host=>"0.0.0.0", :port=>5544, :level=>:error}
{:timestamp=>"2015-10-02T14:49:33.562000+0000", :message=>"The error reported is: \n Address already in use - bind - Address already in use"}

[jolson]

What version of Nagios Log Server are you running? Be sure to update to the latest: http://library.nagios.com/library/produ ... /downloads

I would also like to see your logstash configuration. Access the Web GUI and navigate to 'Administration -> Global Configuration -> View -> all files combined' and send me the result.