Not able to add New Source to Nagios Log server

anish · Post by **anish** » Wed Jul 12, 2017 9:36 am

Hi ,

I am trying to add the new source to the Nagios Log server. But i am not able to see the increase in the count . when checked the logs found the below error .

"{:timestamp=>"2017-07-12T10:08:50.632000-0400", :message=>"An error occurred. Closing connection", :client=>"157.96.179.26:59433", :exception=>#<LogStash::ShutdownSignal: LogStash::ShutdownSignal>, :backtrace=>["org/jruby/RubyIO.java

in `sysread'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:164:in `read'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:112:in `handle_socket'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:147:in `client_thread'"], :level=>:error}
{:timestamp=>"2017-07-12T10:08:50.634000-0400", :message=>"An error occurred. Closing connection", :client=>"156.71.175.9:51756", :exception=>#<LogStash::ShutdownSignal: LogStash::ShutdownSignal>, :backtrace=>["org/jruby/RubyIO.java

in `sysread'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:164:in `read'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:112:in `handle_socket'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:147:in `client_thread'"], :level=>:error}

As per the previous post did checked the LS_OPEN_FILES
I have changed the value to 65535 long back .Is that can i increase the LS_open_files to large number.

LS_USER=logstash
LS_GROUP=logstash
LS_HOME=/usr/local/nagioslogserver
LS_HEAP_SIZE="1000m"
LS_JAVA_OPTS="-Djava.io.tmpdir=${LS_HOME}/tmp"
LS_LOG_FILE=/var/log/logstash/$NAME.log
LS_CONF_DIR=/etc/logstash/conf.d
LS_OPEN_FILES=65535
LS_NICE=19
LS_OPTS=""
LS_PIDFILE=/var/run/$NAME/$NAME.pid
LS_PIDDIR=/var/run/$NAME

please suggest

Post by **cdienger** » Wed Jul 12, 2017 11:38 am

How many log sources are currently configured? Can you upload the config files found in /usr/local/nagioslogserver/logstash/etc/conf.d or PM them to me? I'd also be curious to see the config of the source you're trying to add as well and the /var/log/httpd/access_log and /var/log/httpd/error_log may show something useful while you run:

Code: Select all

tail -f /var/log/httpd/access_log
tail -f /var/log/httpd/error_log

and then try adding the new source.

anish · Post by **anish** » Thu Jul 13, 2017 7:40 am

Hi ,

Please find the attachment of config files found in /usr/local/nagioslogserver/logstash/etc/conf.d ,source conf file and details of /var/log/httpd/access_log and /var/log/httpd/error_log

And also attached logstash.log and elasticsearch . Please check the logs as well .

here are details of access_log and error_log while adding the source

[root@SESKLNGLSIPD01 /]# tail -f /var/log/httpd/access_log
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_aliases?ignore_missing=true HTTP/1.1" 200 38 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_mapping HTTP/1.1" 200 52200 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search HTTP/1.1" 200 124 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:36 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search?search_type=count HTTP/1.1" 200 179 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:39 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_aliases?ignore_missing=true HTTP/1.1" 200 38 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:40 -0400] "GET /nagioslogserver/api/backend/logstash-2017.07.13/_mapping HTTP/1.1" 200 52200 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:40 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search HTTP/1.1" 200 124 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:40 -0400] "POST /nagioslogserver/api/backend/logstash-2017.07.13/_search?search_type=count HTTP/1.1" 200 180 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:41 -0400] "POST /nagioslogserver/api/system/status HTTP/1.1" 200 82 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
172.18.213.170 - - [13/Jul/2017:08:44:41 -0400] "POST /nagioslogserver/api/system/status HTTP/1.1" 200 87 "http://sesklnglsipd01/nagioslogserver/dashboard" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
^C
[root@SESKLNGLSIPD01 /]# ^C
[root@SESKLNGLSIPD01 /]# tail -f /var/log/httpd/error_log
[Wed Jul 12 13:00:25 2017] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jul 12 13:00:25 2017] [notice] Digest: generating secret for digest authentication ...
[Wed Jul 12 13:00:25 2017] [notice] Digest: done
[Wed Jul 12 13:00:25 2017] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 configured -- resuming normal operations

curl: (7) Failed to connect to 2600

:f03c:91ff:fe18:849c: Network is unreachable

curl: (7) Failed to connect to 2600

:f03c:91ff:fe18:849c: Network is unreachable

I have just gone through some of the posts in Nagios Support . So I am forwarding you the details of the command .

/usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d

[root@SESKLNGLSIPD01 /]# /usr/local/nagioslogserver/logstash/bin/logstash -f /usr/local/nagioslogserver/logstash/etc/conf.d
syslog listener died {:protocol=>:tcp, :address=>"0.0.0.0:5544", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in `initialize'", "org/jruby/RubyIO.java:853:in `new'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:152:in `tcp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:101:in `run'"], :level=>:warn}
syslog listener died {:protocol=>:udp, :address=>"0.0.0.0:5544", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:135:in `udp_listener'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:117:in `server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-0.1.6/lib/logstash/inputs/syslog.rb:97:in `run'"], :level=>:warn}
Could not start TCP server: Address in use {:host=>"0.0.0.0", :port=>3515, :level=>:error}
The error reported is:
Address already in use - bind - Address already in use

Post by **cdienger** » Thu Jul 13, 2017 2:19 pm

Hi Anish,

If you're adding a new log source you'll want to make sure to use a port that is not already in use. For example, instead of 5544 you could use 5555:

syslog {
type => 'syslog'
port => 5555
}

anish · Post by **anish** » Fri Jul 14, 2017 8:19 am

Tried but still no go still receiving the same logs that i have attached before

Post by **eloyd** » Fri Jul 14, 2017 8:27 am

So if you remove the Input that you added (I'm assuming it's the ESXi input), does the error go away? If not, then it has nothing to do with your input.

Having said that, log in to the NLS box and make sure you're actually receiving data on the port in question, and that the firewall is open for that port:

Code: Select all

iptables -L -v -n | grep XXX
tcpdump -n port XXX

Where XXX is the port for the Input in NLS

anish · Post by **anish** » Fri Jul 14, 2017 9:21 am

[root@SESKLNGLSIPD01 /]# iptables -L -v -n | grep 5544
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:5544
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5544
[root@SESKLNGLSIPD01 /]# iptables -L -v -n | grep 1514
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:1514
17766 5685K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1514
[root@SESKLNGLSIPD01 /]#
[root@SESKLNGLSIPD01 /]# tcpdump -n port 5544
-bash: tcpdump: command not found

Post by **eloyd** » Fri Jul 14, 2017 9:28 am

Code: Select all

yum -y install tcpdump

Is it the port 5544 stuff that you're not able to receive data on? If so, that's not what I thought you had problems with. I thought you were trying to add a new source. 5544 is the default source.

anish · Post by **anish** » Fri Jul 14, 2017 9:52 am

Hi yes ,

I was not able to receive the data through the port 5544 . But i was able to receive the data before it stopped suddenly .

[root@SESKLNGLSIPD01 /]# tcpdump -v port 5544
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

^C
0 packets captured
16 packets received by filter
0 packets dropped by kernel
[root@SESKLNGLSIPD01 /]#

Post by **cdienger** » Fri Jul 14, 2017 10:40 am

Try stopping logstash:

Code: Select all

service logstash stop

and then check if port 5544 is in use:

Code: Select all

netstat -nap | grep 5544

You can identify the PID/program that is using port 5544(if there is one) and try killing it with:

Code: Select all

kill <PID>

Restart logstash with:

Code: Select all

service logstash start

Restarting logstash alone may be enough to clear this up, but the above will also check for potential conflicts.

Nagios Support Forum

Not able to add New Source to Nagios Log server

Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server

Re: Not able to add New Source to Nagios Log server