Page 2 of 2
Re: JSON and JSON_LINES delimiter
Posted: Mon Apr 11, 2016 4:28 pm
by scottwilkerson
Andrew,
If you are going to change the format that syslog is sending messages, you will need to configure a different Input on a different port in Log Server because the default syslog input expects the format for the message to be RFC3164
Actually, you absolutely can send logs to a Redis server and add an input to pull logs from the Redis queue
Here's the input doc
https://www.elastic.co/guide/en/logstas ... redis.html
Re: JSON and JSON_LINES delimiter
Posted: Mon Apr 11, 2016 4:30 pm
by hsmith
Can you try to switch this to a different input that uses the json codec? You can see our Windows Event Log one for example.
Take a look at this:
https://www.elastic.co/guide/en/logstas ... -json.html
Let me know if this doesn't make sense, or you need some further clarification.
Re: JSON and JSON_LINES delimiter
Posted: Mon Apr 11, 2016 4:31 pm
by vAJ
I am. We send JSON logs to TCP/2057 and the input filter uses the JSON codec.
Re: JSON and JSON_LINES delimiter
Posted: Mon Apr 11, 2016 4:34 pm
by vAJ
My JSON input (i think it's default)
Code: Select all
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
Re: JSON and JSON_LINES delimiter
Posted: Mon Apr 11, 2016 4:37 pm
by hsmith
It's still coming through as one log, though?
Can I see a screenshot of it? This multiline stuff gets tricky.
Re: JSON and JSON_LINES delimiter
Posted: Mon Apr 11, 2016 5:04 pm
by vAJ
I regret hijacking Fabian's thread...
I've been working with two different configs here. One with the JSON template, one without.
In the last rsyslog config I posted with just a straight forwarding of the log , I get the entire json message as a syslog event:
json_log_message.JPG
When we tried using the JSON template config in rsyslogd, it sent each line of JSON as a separate message.
json_messy.JPG
Re: JSON and JSON_LINES delimiter
Posted: Tue Apr 12, 2016 11:40 am
by jolson
Your log is being tagged with both a json parsefailure and a grok parsefailure. I am interested in seeing the filter that you have in place that's causing the grokparsefailure. Could you send us that information please?
Code: Select all
cat /usr/local/nagioslogserver/logstash/etc/conf.d/*
Re: JSON and JSON_LINES delimiter
Posted: Tue Apr 12, 2016 2:08 pm
by vAJ
I'll try to get that. I'm not worried about the parsing yet, I just need to get the right rsyslogd config.
Re: JSON and JSON_LINES delimiter
Posted: Tue Apr 12, 2016 4:35 pm
by hsmith
Unless I'm missing something, from what I understand, you want rsyslog to space it out and look nicer, instead of just sending it like one chunk of text like it is doing right now?
Re: JSON and JSON_LINES delimiter
Posted: Tue Apr 12, 2016 4:39 pm
by jolson
Also, it's worth noting that the json decoding doesn't have to be done at the input level, it can be done at the filter level:
https://www.elastic.co/guide/en/logstas ... -json.html
This way you could design a filter to strip the beginning syslog message and then parse the actualy JSON using the filter above.