All Log Entries on same facility/priority/severity

[tmckay]

Is anyone seeing where you've pointed several equipment to the LS using different facilities, and it all goes to the same facility? i'm also seeing all events come through as severity 5, priority 13 events, when the device is sending out severity 3 and 4 logs. The raw file in LS shows the same severity 5, priority 13 across multiple equipment vendors. i'm using syslog exports from Cisco and Zhone, and they are all identically catergorized as severity 5 (notice), priority 13. Any ideas? The logs are being sent through a forwarded port to a NAT'd address.

[abrist]

Have you created any custom filters/rules for these sources?

[tmckay]

Have you created any custom filters/rules for these sources?

No. They were doing this immediately upon installation, as well as the 127.0.0.1 log entries. i fixed that, but all of my log entries are still coded same facility, severity and priority. Is there a place to grab the raw syslog file from CLI and examine what was received by the system?

[tmckay]

No, and we reloaded from a brand new image out from behind the NAT. Still seeing the same thing at the moment. It doesn't help when building alerts if there is no severity segregation once the logfile is processed.

[tmckay]

i'm including a tcpdump of a sample severity 3 entry that is being picked up as a severity 5. Ignore the inaccurate rDNS reference to thesawyerfamily.com

14:04:48.288228 IP (tos 0x0, ttl 255, id 23, offset 0, flags [none], proto UDP (17), length 187)
64.233.128.3.58449 > http://www.thesawyersfamily.com.syslog: SYSLOG, length: 159
Facility local0 (16), Severity error (3)
Msg: 408337: SLOT 1: Nov 17 13:04:46.284 CST: %SPA_CHOC_DSX-3-HDLC_CTRL_ERR: SPA 1/0: 204978 TX Chnl Queue Overflow events on HDLC Controller were encountered.
E.........8\@[email protected]<131>408337: SLOT 1: Nov 17 13:04:46.284 CST: %SPA_CHOC_DSX-3-HDLC_CTRL_ERR: SPA 1/0: 204978 TX Chnl Queue Overflow events on HDLC Controller were encountered.
........Tu..N.@.........@..;..................
........Tu..N.@.........@.....................
........Tu..N.@[email protected]..................
........Tu..N.@.........@..$..................
........Tu..N.@.........@.....................
........Tu..N.@.........@.....................
........Tu..N.@[email protected]..................
........Tu..N.@.........@..;..................
........Tu..N.@.........@..$..................

[eloyd]

Oh, I so wanna hack the Sawyers now...

[sreinhardt]

Off the top of my head, it sounds like either a bad filter is in place(which it sounds like you do not believe there is) or that no filtering may be being done and somehow elasticsearch\logstash believe it should. Could you go to a log view page, and open any one of those events that is mis-tagged and send us a screenshot. The same of your current filters and this input expanded would be very helpful.

[tmckay]

Please see the attached. i've not modified any of the files in the server, other than add the alerts.php that fixes the repetitious "alert is OK" and change to udp 514 for port. This issue has persisted across 2 different vm environments, one in which it was straight "out of the box" from Nagios site. This product appears to have much promise. Thanks.

[sreinhardt]

I'm sorry by anything showing the filters, I meant under admin->global config. Aside from that, I see that your entry has a tag of _grokparsefailure. This should mean that it is getting tagged as at least partially matching a grok filter, but either the rest did not match as well, or the filter it's self was not syntactically correct. The latter ideally shouldn't be too much of an option with verification after config writing.

So, let's check your grok filters, maybe you have a global filter for syslog type or another tag that is being matched.

[tmckay]

It just hit me you might want this screenshot. I'm not certain where i'd check my grok settings, if it's not here.