All Log Entries on same facility/priority/severity

[sreinhardt]

Well, you would almost definitely match the syslog input, as you might expect, but I do not see any filters there that would cause issue. Could you attack all of your /usr/local/nagioslogserver/logstash/etc/conf.d/ files please? You should only have 3 on a default system like yours, but let's make sure and make sure they are being outputted correctly.

[eloyd]

Could you attack all of your /usr/local/nagioslogserver/logstash/etc/conf.d/ files please?

Or attach them, even?

[tmckay]

The 3 files are attached. i could still attack them, if it's more helpful. Please let me know if anything else is needed.

[scottwilkerson]

The syslog input requires RFC3164 syslog format (see http://logstash.net/docs/1.4.2/inputs/syslog ), looking at your sample, this isn't the format you message are coming in, it should be something like this:

Code: Select all

<PRI>Jul 10 12:00:00 192.168.1.1 SyslogGen MESSAGE TEXT

This is also why you will see you have a tag added to the item "_grokparsefailure"

You can either change the format on the sending device, or you could create a new tcp/udp input and add a grok filter to determine the parts of the message.

This site can be useful in determining the correct grok pattern
https://grokdebug.herokuapp.com/

[tmckay]

Nov 19 09:30:40 CST: %LINK-3-UPDOWN: Interface GigabitEthernet4/3, changed state to down

This is the format that the Cisco 7600 is sending to the LS. i have different equipment models and from different vendors all coming in as the exact same priority/severity/facility. Is there a command to make Cisco send syslog in a different format than that above? Also, i can turn the localhost logging back on and see, but i think the localhost logging had the same issue. i can confirm and post back.

[tmckay]

Below is a tcpdump from the LS that shows it is receiving the various logfiles, which match the above listed format. i've done some research and it appears that Cisco syslog output does comply with RFC 3164. You can see below that when i changed facilities, it was referenced in the file, as well as various severities when i enabled interfaces with no layer1 connectivity. Also, i don't know if it's related, but my logstash log is completely empty, and system status never shows me the status of logstash collector or elasticsearch; it just shows spinning wheels. If LS isn't setup with appropriate grok filters for cisco-esque logfile collection OOB, are there any recommended grok filters to try?

Cisco Log Output
Nov 20 20:25:40.846: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0
(64.233.128.6)
Nov 20 20:25:58.505: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to
up
Nov 20 20:26:10.553: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0
(64.233.128.6)
Nov 20 20:26:11.449: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state t
o administratively down







TCP Dump
[tmckay@nocsyslog01 ~]$ sudo tcpdump host 64.233.146.154 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:23:02.835451 IP 64.233.146.154.55156 > www.thesawyersfamily.com.syslog: SYSLOG local1.notice, length: 114
E.........'.@[email protected]<141>22: RITTERLAB: Nov 20 20:23:01.815: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0 (64.233.128.6)
15:25:16.681847 IP 64.233.146.154.55156 > www.thesawyersfamily.com.syslog: SYSLOG local2.notice, length: 114
E.........'.@[email protected]<149>23: RITTERLAB: Nov 20 20:25:15.662: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0 (64.233.128.6)
15:25:31.185651 IP 64.233.146.154.55156 > www.thesawyersfamily.com.syslog: SYSLOG local3.notice, length: 114
E.... ....'.@[email protected]^Q<157>24: RITTERLAB: Nov 20 20:25:30.166: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0 (64.233.128.6)
15:25:41.865550 IP 64.233.146.154.55156 > www.thesawyersfamily.com.syslog: SYSLOG local3.notice, length: 114
E....
....&.@[email protected]<157>25: RITTERLAB: Nov 20 20:25:40.846: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0 (64.233.128.6)
15:25:59.525257 IP 64.233.146.154.55156 > www.thesawyersfamily.com.syslog: SYSLOG local3.error, length: 103
E.........' @[email protected]*.<155>26: RITTERLAB: Nov 20 20:25:58.505: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
15:26:11.573169 IP 64.233.146.154.55156 > www.thesawyersfamily.com.syslog: SYSLOG local3.notice, length: 114
E.........&.@[email protected]<157>27: RITTERLAB: Nov 20 20:26:10.553: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0 (64.233.128.6)
15:26:11.573999 IP 64.233.146.154.55156 > www.thesawyersfamily.com.syslog: SYSLOG local3.notice, length: 123
....&.@[email protected]..... <157>28: RITTERLAB: Nov 20 20:26:11.449: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down

[scottwilkerson]

Better than the tcpdump you are sending, would bea copy of the message field that is being received in Log Server from the Cisco device.

Thanks

[sreinhardt]

Can you send me a packet capture of that instead please? I'm not sure if those are valid non-printable headers at the front of your messages or what is going on there. Going to be far easier to look at the packets themselves if you don't mind.

Code: Select all

tcpdump -nnXSs 0 -w /tmp/syslog.pcap host 64.233.146.154

As swilkerson mentioned, message fields would be fantastic as well.

[tmckay]

Thanks! The pcap is attached as well, but with a .txt extension, so change it back to pcap. i'm still reviewing if anyone else has seen similar behavior with logstash. will update if i find anything that improves results.

Message contents
<157>240: RITTERLAB: Nov 21 14:07:39.548: %SYS-5-CONFIG_I: Configured from console by tmckay on vty0 (64.233.128.6)

[sreinhardt]

thanks, I'm going to compare this against some of my incoming cisco logs. I don't have zhone, but I do have several switches and firewalls to test against. What IOS version are you presently running?