Not receiving logs from Juniper switch

[nshah]

We were using Kiwi for a log collector. Configured our Juniper devices to send the log to Nagios log server exactly the same as our current log server. But the Nagios Log server is not getting any logs. There is no firewall in between, One difference in Nagios log server is by default it receives log on port 5544.

Juniper devices are set to send log on port 5544, Not sure if anything needs to do on the Nagios Log server? The documentation says we only need to configure on the log sender with the log server IP and port no. I am completely new to Nagios log server, Don't we need to configure anything in Nagios log server? I am stuck here, Any help will be appreciated.

[mbellerue]

It should be exactly that simple. Do you have any logs getting to Nagios Log Server at this time, or is this the first device that you are setting up to send to Log Server?

The first thing we should do is verify that the logs are actually making it to Log Server. Can you ssh into Log Server as root, run the following command, let it run for a couple of minutes, and let me know if it displays any output?

Code: Select all

tcpdump -i any host <IPOfJunpierSwitch> tcp 5544

That should start a network sniffer listening specifically for the IP address of your Juniper switch. If you start seeing data right away, that's good. That means that the logs are being sent to Log Server, and you can ctrl+c to stop the command. If nothing has shown up after a couple of minutes, we will want to find some way to verify that the switch is sending.

[nshah]

Thanks, I can see the logs coming in. How long does it take to show the hosts in Report->Unique hosts?

Although I can see the logs are receiving by the Nagios Log server, I still can't see the devices have been added.

Is there is a way to add port 514 including 5544? There are some devices which we can't change the port to send the log.

[scottwilkerson]

Thanks, I can see the logs coming in. How long does it take to show the hosts in Report->Unique hosts?

Although I can see the logs are receiving by the Nagios Log server, I still can't see the devices have been added.

Is there is a way to add port 514 including 5544? There are some devices which we can't change the port to send the log.

They should show in the unique hosts in an hour or 2


As for adding another port, yes you can add another syslog input, however there are special instructions outlined for listening on ports below 1024

See the following doc
https://assets.nagios.com/downloads/nag ... Server.pdf

[nshah]

Thanks, I have updated the config as per the documents. But I am still not receiving a log.

After TCPdump on that IP (10.x.x.x), This is the output of the TCP dump. Does ICMP require to allow?

15:31:46.609059 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 183
15:31:46.609118 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 235
15:31:49.774742 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 128
15:31:49.774824 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 164
15:31:49.775211 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 127
15:31:49.775252 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 163
15:31:49.775678 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 130
15:31:49.775701 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 166
15:31:56.608244 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 223
15:31:56.608319 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 259
15:31:56.608409 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 204
15:31:56.608448 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 240
15:31:56.609040 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 143
15:31:56.609067 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 179
15:31:56.609144 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 209
15:31:56.609162 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 245
15:31:56.609935 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 251
15:31:56.609965 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 287
15:31:56.610035 IP 10.x.x.x.syslog > NagiosLogServer.syslog: SYSLOG local1.debug, length: 147
15:31:56.610059 IP NagiosLogServer > 10.x.x.x: ICMP host NagiosLogServer unreachable - admin prohibited, length 183

These are the steps I took to allow privileged port-
1) Run Logstash as root
2) Add an Input for
syslog {
type => "syslog"
port => 514
}
3) Add firewall rule
firewall-cmd --zone=public --add-port=514/tcp
firewall-cmd --zone=public --add-port=514/tcp --permanent

[scottwilkerson]

I actually don't believe ICMP is required at all but the logs you are showing say unreachable - admin prohibited, can this 10.x.x.x route successfully to the Nagios Log Server machine?

And for clarification on the setup on the Nagios Log Server machine, can you show the output of the following

Code: Select all

netstat -nlp|grep 514
ps -ef|grep logstash

[nshah]

Hi,

Yes, I can ping from NagiosLogServer to 10.x.x.x. I just double confirmed too.

[root@NagiosLogServer ~]# ping 10.x.x.x
PING 10.x.x.x (10.x.x.x) 56(84) bytes of data.
64 bytes from 10.x.x.x: icmp_seq=1 ttl=55 time=4.19 ms
64 bytes from 10.x.x.x: icmp_seq=2 ttl=55 time=4.21 ms
64 bytes from 10.x.x.x: icmp_seq=3 ttl=55 time=10.3 ms
64 bytes from 10.x.x.x: icmp_seq=4 ttl=55 time=24.0 ms
64 bytes from 10.x.x.x: icmp_seq=5 ttl=55 time=90.9 ms
64 bytes from 10.x.x.x: icmp_seq=6 ttl=55 time=2.20 ms
64 bytes from 10.x.x.x: icmp_seq=7 ttl=55 time=7.92 ms
64 bytes from 10.x.x.x: icmp_seq=8 ttl=55 time=3.06 ms
64 bytes from 10.x.x.x: icmp_seq=9 ttl=55 time=1.66 ms

Here is the output of the command-

[root@NagiosLogServer ~]# netstat -nlp | grep 514
tcp6 0 0 :::514 :::* LISTEN 29698/java
udp6 0 0 :::514 :::* 29698/java

[root@NagiosLogServer ~]# ps -ef | grep logstash
root 23913 23749 0 13:05 pts/0 00:00:00 grep --color=auto logstash
root 29696 1 0 Oct13 ? 00:00:00 runuser -s /bin/sh -c exec /usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4 root
root 29698 29696 2 Oct13 ? 00:36:02 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/usr/local/nagioslogserver/tmp -XX:HeapDumpPath=/usr/local/nagioslogserver/logstash/heapdump.hprof -Xbootclasspath/a:/usr/local/nagioslogserver/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/local/nagioslogserver/logstash/vendor/jruby -Djruby.lib=/usr/local/nagioslogserver/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /usr/local/nagioslogserver/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /usr/local/nagioslogserver/logstash/etc/conf.d -l /var/log/logstash/logstash.log -w 4

[scottwilkerson]

This all looks ok, I think the unreachable - admin prohibited error might just be from tcpdump

I noticed you just opened 514 for tcp in the firewall commands and not udp.
Are you sure 10.x.x.x is sending the syslogs as TCP and not UDP?

Also, Just for clarification, I see this is only listening on IPv6 ports, 10.x.x.x is sending to the IPv6 address correct?

[nshah]

Thank you! Yes, You are right it was using 514 UDP port. It's working now

[scottwilkerson]

Thank you! Yes, You are right it was using 514 UDP port. It's working now

Great!

Locking thread