Nagios Support Forum

Hi ,

I am trying to add the new source to the Nagios Log server. But i am not able to see the increase in the count . when checked the logs found the below error .

"{:timestamp=>"2017-07-12T10:08:50.632000-0400", :message=>"An error occurred. Closing connection", :client=>"157.96.179.26:59433", :exception=>#<LogStash::ShutdownSignal: LogStash::ShutdownSignal>, :backtrace=>["org/jruby/RubyIO.javain `sysread'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:164:in `read'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:112:in `handle_socket'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:147:in `client_thread'"], :level=>:error}
{:timestamp=>"2017-07-12T10:08:50.634000-0400", :message=>"An error occurred. Closing connection", :client=>"156.71.175.9:51756", :exception=>#<LogStash::ShutdownSignal: LogStash::ShutdownSignal>, :backtrace=>["org/jruby/RubyIO.javain `sysread'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:164:in `read'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:112:in `handle_socket'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:147:in `client_thread'"], :level=>:error}

As per the previous post did checked the LS_OPEN_FILES
I have changed the value to 65535 long back .Is that can i increase the LS_open_files to large number.

LS_USER=logstash
LS_GROUP=logstash
LS_HOME=/usr/local/nagioslogserver
LS_HEAP_SIZE="1000m"
LS_JAVA_OPTS="-Djava.io.tmpdir=${LS_HOME}/tmp"
LS_LOG_FILE=/var/log/logstash/$NAME.log
LS_CONF_DIR=/etc/logstash/conf.d
LS_OPEN_FILES=65535
LS_NICE=19
LS_OPTS=""
LS_PIDFILE=/var/run/$NAME/$NAME.pid
LS_PIDDIR=/var/run/$NAME

please suggest

How many log sources are currently configured? Can you upload the config files found in /usr/local/nagioslogserver/logstash/etc/conf.d or PM them to me? I'd also be curious to see the config of the source you're trying to add as well and the /var/log/httpd/access_log and /var/log/httpd/error_log may show something useful while you run:
and then try adding the new source.

Hi ,

Please find the attachment of config files found in /usr/local/nagioslogserver/logstash/etc/conf.d ,source conf file and details of /var/log/httpd/access_log and /var/log/httpd/error_log

And also attached logstash.log and elasticsearch . Please check the logs as well .


here are details of access_log and error_log while adding the source

[root@SESKLNGLSIPD01 /]# tail -f /var/log/httpd/access_log
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_aliases?ignore_missing=true HTTP/1.1" 200 38 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_mapping HTTP/1.1" 200 52200 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search HTTP/1.1" 200 124 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search?search_type=count HTTP/1.1" 200 179 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:39 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_aliases?ignore_missing=true HTTP/1.1" 200 38 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:40 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_mapping HTTP/1.1" 200 52200 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:40 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search HTTP/1.1" 200 124 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:40 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search?search_type=count HTTP/1.1" 200 180 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:41 -0400] "POST /nagioslogserver/api/system/status HTTP/1.1" 200 82 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:41 -0400] "POST /nagioslogserver/api/system/status HTTP/1.1" 200 87 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
^C
[root@SESKLNGLSIPD01 /]# ^C
[root@SESKLNGLSIPD01 /]# tail -f /var/log/httpd/error_log
[Wed Jul 12 13:00:25 2017] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jul 12 13:00:25 2017] [notice] Digest: generating secret for digest authentication ...
[Wed Jul 12 13:00:25 2017] [notice] Digest: done
[Wed Jul 12 13:00:25 2017] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations

curl: (7) Failed to connect to 2600:f03c:91ff:fe18:849c: Network is unreachable

curl: (7) Failed to connect to 2600:f03c:91ff:fe18:849c: Network is unreachable

curl: (7) Failed to connect to 2600:f03c:91ff:fe18:849c: Network is unreachable




I have just gone through some of the posts in Nagios Support . So I am forwarding you the details of the command .

/usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d



[root@SESKLNGLSIPD01 /]# /usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d
syslog listener died {:protocol=>:tcp, :address=>"0.0.0.0:5544", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:853:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:152:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:101:in `run'"], :level=>:warn}
syslog listener died {:protocol=>:udp, :address=>"0.0.0.0:5544", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:135:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}
Could not start TCP server: Address in use {:host=>"0.0.0.0", :port=>3515, :level=>:error}
The error reported is:
Address already in use - bind - Address already in use

Hi Anish,

If you're adding a new log source you'll want to make sure to use a port that is not already in use. For example, instead of 5544 you could use 5555:

syslog {
type => 'syslog'
port => 5555
}

Tried but still no go still receiving the same logs that i have attached before

So if you remove the Input that you added (I'm assuming it's the ESXi input), does the error go away? If not, then it has nothing to do with your input.

Having said that, log in to the NLS box and make sure you're actually receiving data on the port in question, and that the firewall is open for that port: Where XXX is the port for the Input in NLS

[root@SESKLNGLSIPD01 /]# iptables -L -v -n | grep 5544
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:5544
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5544
[root@SESKLNGLSIPD01 /]# iptables -L -v -n | grep 1514
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:1514
17766 5685K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1514
[root@SESKLNGLSIPD01 /]#
[root@SESKLNGLSIPD01 /]# tcpdump -n port 5544
-bash: tcpdump: command not found

Is it the port 5544 stuff that you're not able to receive data on? If so, that's not what I thought you had problems with. I thought you were trying to add a new source. 5544 is the default source.

Hi yes ,

I was not able to receive the data through the port 5544 . But i was able to receive the data before it stopped suddenly .


[root@SESKLNGLSIPD01 /]# tcpdump -v port 5544
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

^C
0 packets captured
16 packets received by filter
0 packets dropped by kernel
[root@SESKLNGLSIPD01 /]#

Try stopping logstash:
and then check if port 5544 is in use:
You can identify the PID/program that is using port 5544(if there is one) and try killing it with:
Restart logstash with:
Restarting logstash alone may be enough to clear this up, but the above will also check for potential conflicts.