Unable to receive incoing syslog on port 514

[dlukinski]

Hi

I have configured some tests from the network switches to the Nagios Log (incoming on port 514)
syslog is reaching NAgios LOG server, but does not appear in configuration or the database even is enabled:

1. Logstash is currently collecting locally on: 10.102.36.164 tcp: 5544, 2056, 5545, 2057, 3515, 3516, 4444, 4445, 4446, 4447, 4448, 4450udp: 4444, 4445, 4446, 4447, 4448, 4450, 5544, 5545 - 514 is missing (but active)

2. Active Syslog (514) config
----------------------------------
tcp {
port => 514
type => syslog
}
udp {
port => 514
type => syslog
}
---------------------------------

3. TCP Dump

[root@fikc-naglsprod01 ~]# tcpdump port 514 -X
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:08:21.374919 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 77
0x0000: 4500 0069 12de 0000 3111 0d72 0a2a 3001 E..i....1..r.*0.
0x0010: 0a66 24a4 0202 0202 0055 451c 3c31 353e .f$......UE.<15>
0x0020: 204a 616e 2031 3620 3136 3a30 383a 3230 .Jan.16.16:08:20
0x0030: 2031 302e 3432 2e34 382e 3120 4c4c 4450 .10.42.48.1.LLDP
0x0040: 3a20 4c4c 4450 2064 656c 6179 2074 696d :.LLDP.delay.tim
0x0050: 6572 2066 6f72 2070 6f72 7420 3a20 3136 er.for.port.:.16
0x0060: 3420 6578 7069 7265 64 4.expired
16:08:21.375977 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 80
0x0000: 4500 006c 12e0 0000 3111 0d6d 0a2a 3001 E..l....1..m.*0.
0x0010: 0a66 24a4 0202 0202 0058 592c 3c31 353e .f$......XY,<15>
0x0020: 204a 616e 2031 3620 3136 3a30 383a 3230 .Jan.16.16:08:20
0x0030: 2031 302e 3432 2e34 382e 3120 4c4c 4450 .10.42.48.1.LLDP
0x0040: 3a20 4c4c 4450 206c 6c64 7020 7265 6672 :.LLDP.lldp.refr
0x0050: 6573 6820 706b 7420 7365 6e74 206f 7574 esh.pkt.sent.out
0x0060: 2070 6f72 7420 3a20 3138 3320 .port.:.183.
16:08:21.376690 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 80
0x0000: 4500 006c 12e2 0000 3111 0d6b 0a2a 3001 E..l....1..k.*0.
0x0010: 0a66 24a4 0202 0202 0058 5934 3c31 353e .f$......XY4<15>
0x0020: 204a 616e 2031 3620 3136 3a30 383a 3230 .Jan.16.16:08:20
0x0030: 2031 302e 3432 2e34 382e 3120 4c4c 4450 .10.42.48.1.LLDP
0x0040: 3a20 4c4c 4450 206c 6c64 7020 7265 6672 :.LLDP.lldp.refr
0x0050: 6573 6820 706b 7420 7365 6e74 206f 7574 esh.pkt.sent.out
0x0060: 2070 6f72 7420 3a20 3230 3220 .port.:.202.
16:08:22.375106 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 80
0x0000: 4500 006c 12e4 0000 3111 0d69 0a2a 3001 E..l....1..i.*0.
0x0010: 0a66 24a4 0202 0202 0058 5932 3c31 353e .f$......XY2<15>
0x0020: 204a 616e 2031 3620 3136 3a30 383a 3231 .Jan.16.16:08:21
0x0030: 2031 302e 3432 2e34 382e 3120 4c4c 4450 .10.42.48.1.LLDP
0x0040: 3a20 4c4c 4450 206c 6c64 7020 7265 6672 :.LLDP.lldp.refr
0x0050: 6573 6820 706b 7420 7365 6e74 206f 7574 esh.pkt.sent.out
0x0060: 2070 6f72 7420 3a20 3131 3320 .port.:.113.
16:08:22.916660 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 74
0x0000: 4500 0066 12e6 0000 3111 0d6d 0a2a 3001 E..f....1..m.*0.
0x0010: 0a66 24a4 0202 0202 0052 9325 3c31 353e .f$......R.%<15>
0x0020: 204a 616e 2031 3620 3136 3a30 383a 3231 .Jan.16.16:08:21
0x0030: 2031 302e 3432 2e34 382e 3120 4c4c 4450 .10.42.48.1.LLDP
0x0040: 3a20 4c4c 4450 206c 6c64 7020 706b 7420 :.LLDP.lldp.pkt.
0x0050: 7265 6365 6976 6564 206f 6e20 706f 7274 received.on.port
0x0060: 203a 2031 3032 .:.102

[scottwilkerson]

Because port 514 is a privileged port there is an additional step required to get it working outlined in this document
https://assets.nagios.com/downloads/nag ... Server.pdf

[dlukinski]

Because port 514 is a privileged port there is an additional step required to get it working outlined in this document
https://assets.nagios.com/downloads/nag ... Server.pdf

Hi

We've just discovered that the method # 2 kills logstash daemon and breaks Java, so we've implemented Method # 1 - (with Tom)
Now, i get incoming syslog from a specific IP:


[root@fikc-naglsprod01 logstash]# tcpdump port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:25:59.156321 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 76
18:25:59.156552 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 80
18:26:00.531346 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 73
18:26:00.531526 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 97
18:26:00.531699 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 71
18:26:00.792857 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 73
18:26:00.793053 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 97
18:26:00.793222 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 71
18:26:00.794221 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 73

But, NAGIOS LOG does not see it still:

IP Address
10.42.48.1

Not logs found. No logs from that host in the database. The sender's firewall may be blocking the logs or the sender may be misconfigured.

Thanks

(worried that if I am to follow other advise given in a forum, may break our installation again for whatever reason)

[scottwilkerson]

Which of the 2 methods did you follow?

what is the output of the following

Code: Select all

tail -50 /var/log/logstash/logstash.log

[dlukinski]

Which of the 2 methods did you follow?

what is the output of the following

Code: Select all

tail -50 /var/log/logstash/logstash.log

We've just discovered that the method # 2 kills logstash daemon and breaks Java, so we've implemented Method # 1 - (with Tom)
Now, i get incoming syslog from a specific IP:


[root@fikc-naglsprod01 logstash]# tcpdump port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:25:59.156321 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 76
18:25:59.156552 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 80
18:26:00.531346 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 73
18:26:00.531526 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 97
18:26:00.531699 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 71
18:26:00.792857 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 73
18:26:00.793053 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 97
18:26:00.793222 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 71
18:26:00.794221 IP 10.42.48.1.syslog > fikc-naglsprod01.konecranes.com.syslog: SYSLOG user.debug, length: 73

But, NAGIOS LOG does not see it still:

IP Address
10.42.48.1

Not logs found. No logs from that host in the database. The sender's firewall may be blocking the logs or the sender may be misconfigured.

Thanks

(worried that if I am to follow other advise given in a forum, may break our installation again for whatever reason)

[scottwilkerson]

it is java that is listening on port 514 right?

Code: Select all

netstat -nlp|grep 514

[dlukinski]

it is java that is listening on port 514 right?

Code: Select all

netstat -nlp|grep 514

Yes, so it seems

[root@fikc-naglsprod01 logstash]# netstat -nlp|grep 514
tcp 0 0 :::514 :::* LISTEN 15171/java
udp 0 0 :::514 :::* 15171/java

[tgriep]

Check to see if the firewall on the server is allowing inbound traffic on port 514.

[dlukinski]

Check to see if the firewall on the server is allowing inbound traffic on port 514.

Please see the last line

iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -p tcp -m tcp --dport 4450 -j ACCEPT
-A INPUT -p udp -m udp --dport 4450 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4448 -j ACCEPT
-A INPUT -p udp -m udp --dport 4448 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 3516 -j ACCEPT
-A INPUT -p udp -m udp --dport 4447 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4447 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4446 -j ACCEPT
-A INPUT -p udp -m udp --dport 4446 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2057 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2056 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5544 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 4444 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 4445 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3515 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9300:9400 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5667 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5666 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3516 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4448 -j ACCEPT
-A INPUT -p udp -m udp --dport 4448 -j ACCEPT
-A INPUT -p udp -m udp --dport 514 -j ACCEPT

[tgriep]

Try disabling the firewall and see if the server starts to receive the logs.
Also, verify that the input is still configured in the Logserver's GUI and that it is enabled.