JSON and JSON_LINES delimiter

[scottwilkerson]

Andrew,

If you are going to change the format that syslog is sending messages, you will need to configure a different Input on a different port in Log Server because the default syslog input expects the format for the message to be RFC3164

Found this as well http://www.rsyslog.com/coupling-with-lo ... #more-2356 but NLS doesn't use redis.

Actually, you absolutely can send logs to a Redis server and add an input to pull logs from the Redis queue
Here's the input doc
https://www.elastic.co/guide/en/logstas ... redis.html

[hsmith]

Can you try to switch this to a different input that uses the json codec? You can see our Windows Event Log one for example.

Take a look at this: https://www.elastic.co/guide/en/logstas ... -json.html

Let me know if this doesn't make sense, or you need some further clarification.

[vAJ]

I am. We send JSON logs to TCP/2057 and the input filter uses the JSON codec.

[vAJ]

My JSON input (i think it's default)

Code: Select all

tcp {
    type => 'import_json'
    tags => 'import_json'
    port => 2057
    codec => json
}

[hsmith]

It's still coming through as one log, though?

Can I see a screenshot of it? This multiline stuff gets tricky.

[vAJ]

I regret hijacking Fabian's thread...

I've been working with two different configs here. One with the JSON template, one without.

In the last rsyslog config I posted with just a straight forwarding of the log , I get the entire json message as a syslog event:

json_log_message.JPG

When we tried using the JSON template config in rsyslogd, it sent each line of JSON as a separate message.

json_messy.JPG

[jolson]

Your log is being tagged with both a json parsefailure and a grok parsefailure. I am interested in seeing the filter that you have in place that's causing the grokparsefailure. Could you send us that information please?

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/*

[vAJ]

I'll try to get that. I'm not worried about the parsing yet, I just need to get the right rsyslogd config.

[hsmith]

Unless I'm missing something, from what I understand, you want rsyslog to space it out and look nicer, instead of just sending it like one chunk of text like it is doing right now?

[jolson]

Also, it's worth noting that the json decoding doesn't have to be done at the input level, it can be done at the filter level: https://www.elastic.co/guide/en/logstas ... -json.html

This way you could design a filter to strip the beginning syslog message and then parse the actualy JSON using the filter above.