Hosts dropped to 1

[sreinhardt]

OK well, let's run those netstat and ps commands after stopping the logstash service as I had requested. Ideally it should indicate what process is using that port. Just incase you're not aware, the netstat output should look like:

Code: Select all

netstat -naop | grep 5544

tcp 0 0 :::5544 :::* LISTEN 15303/java off (0.00/0/0)
tcp 0 0 ::1:5544 ::1:41714 ESTABLISHED 15303/java off (0.00/0/0)
tcp 0 0 ::1:41714 ::1:5544 ESTABLISHED 844/rsyslogd off (0.00/0/0)
udp 0 0 :::5544 :::* 15303/java off (0.00/0/0)

btw this is what a properly running nls server looks like when pretty much stock

The last parts showing 15303/java are process id's and names. Once we have logstash stopped and the netstat run, take the offending pid still using 5544 and run this:

Code: Select all

ps -ef | grep [found pid]

[itbilling]

Here is the result:

[root@ATCLogServer ~]# netstat -naop | grep 5544
tcp 0 0 127.0.0.1:50275 127.0.0.1:5544 ESTABLISHED 1008/rsyslogd off (0.00/0/0)
tcp 0 0 :::5544 :::* LISTEN 27229/java off (0.00/0/0)
tcp 16797 0 ::ffff:127.0.0.1:5544 ::ffff:127.0.0.1:50275 ESTABLISHED 27229/java off (0.00/0/0)
udp 0 0 :::5544 :::* 27229/java off (0.00/0/0)
[root@ATCLogServer ~]# ps -ef | grep logstash
root 27227 1 0 08:22 pts/1 00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4 nagios
nagios 27229 27227 11 08:22 ? 00:00:24 /usr/bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /usr/local/nagioslogserver/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/usr/local/nagioslogserver/logstash/lib /usr/local/nagioslogserver/logstash/lib/logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
root 27597 26342 0 08:26 pts/1 00:00:00 grep logstash
[root@ATCLogServer ~]# service logstash stop
Stopping Logstash Daemon: [ OK ]

[root@ATCLogServer ~]# netstat -nao | grep 5544
[root@ATCLogServer ~]# ps -ef | grep logstash
root 27620 26342 0 08:26 pts/1 00:00:00 grep logstash
[root@ATCLogServer ~]# service logstash start
Starting Logstash Daemon: [ OK ]
[root@ATCLogServer ~]# netstat -naop | grep 5544
[root@ATCLogServer ~]# ps -ef | grep logstash
root 27634 1 0 08:26 pts/1 00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4 nagios
nagios 27636 27634 6 08:26 ? 00:00:00 /usr/bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /usr/local/nagioslogserver/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/usr/local/nagioslogserver/logstash/lib /usr/local/nagioslogserver/logstash/lib/logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
root 27691 26342 0 08:26 pts/1 00:00:00 grep logstash

[itbilling]

I was able to find the PID and have listed the results of the next command below:

[root@ATCLogServer ~]# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1054/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1100/sendmail
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 28148/sshd
tcp 0 0 :::2056 :::* LISTEN 30227/java
tcp 0 0 ::ffff:172.16.16.128:5544 :::* LISTEN 30227/java
tcp 0 0 :::2057 :::* LISTEN 30227/java
tcp 0 0 ::ffff:127.0.0.1:9200 :::* LISTEN 10128/java
tcp 0 0 :::80 :::* LISTEN 1126/httpd
tcp 0 0 :::9300 :::* LISTEN 10128/java
tcp 0 0 :::22 :::* LISTEN 1054/sshd
tcp 0 0 ::1:6010 :::* LISTEN 28148/sshd
tcp 0 0 :::3515 :::* LISTEN 30227/java
[root@ATCLogServer ~]# ps -ef | grep 30227
root 1643 28172 0 09:34 pts/0 00:00:00 grep 30227
nagios 30227 30225 12 08:51 ? 00:05:32 /usr/bin/java -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /usr/local/nagioslogserver/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/usr/local/nagioslogserver/logstash/lib /usr/local/nagioslogserver/logstash/lib/logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4
[root@ATCLogServer ~]#

[itbilling]

(Brian commenting)

Ok, please help me understand, why are you focused on the Syslog Port 5544?

Here is a snip from the Windows Servers all of them show the same thing...
C:\Program Files (x86)\nxlog\data\nxlog.log

2015-01-07 08:16:52 ERROR couldn't connect to tcp socket on atclogserver:3515; No connection could be made because the target machine actively refused it.
2015-01-07 08:16:52 INFO connecting to atclogserver:3515
2015-01-07 08:16:53 INFO reconnecting in 8 seconds
2015-01-07 08:16:53 ERROR couldn't connect to tcp socket on atclogserver:3515; No connection could be made because the target machine actively refused it.
2015-01-07 08:16:56 INFO connecting to atclogserver:3515
2015-01-07 08:16:57 INFO reconnecting in 16 seconds

If all the clients are looking for TCP 3515, shouldn't we be focusing on that port instead of the 5544?

[sreinhardt]

I was specifically looking for 5544 because of the error mentioned in logstash's log seems directly related to it. Also because you stated that even with logstash stopped, nmap claimed 5544 was still open. With that said, your netstats both pre and post restarting of logstash look great and do not seem to keep 5544 open in between. Could you tar up and post (or pm to one of us) the current logstash log(s) that you have for further review?

[itbilling]

I had sent a message, is there a guide to get the logs needed to pack up and send? I didn't see one in the documentation for Log server.

[sreinhardt]

There is not currently a guide. At the moment, we are just looking for the logstash logs, let's tar those up like so:

Code: Select all

tar czf /tmp/logstash-logs.tar.gz /var/log/logstash/

Then if you could send the resulting /tmp/logstash-logs.tar.gz please.

[itbilling]

Thank you for the instructions, I just sent it in a message.

[sreinhardt]

Got it, and it's in our internal folder for this. I'll check it out and let you know!

[itbilling]

Any update on this? I'd like to get this package in production as quickly as possible.