JSON files

[mcapra]

There may be some rsyslog configurations to consider. From the CLI of the machine that is sending the JSON file, can you share the outputs of:

Code: Select all

ls -al /etc/rsyslog.d/
cat /etc/rsyslog.d/*.conf

Any chance I'd be able to get these outputs? rsyslog might be applying a template that is malforming the JSON.

[opene2]

Code: Select all

ls -l /etc/rsyslog.d/

-rw-r--r-- 1 root root 736 Oct 18 16:02 90-nagioslogserver_var_tmp_logs_test.log.conf

Code: Select all

cat /etc/rsyslog.d/*.conf

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/tmp/logs/test.log
$InputFileTag import_json:
$InputFileStateFile nls-state-var_tmp_logs_test.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'import_json' then @@dev444.dev.e2open.com:2057
if $programname == 'import_json' then ~

[mcapra]

It definitely looks like the default rsyslog configuration template is messing up the JSON. I have this received as a message:

Code: Select all

<133>Oct 18 15:55:19 localhost import_json: {"Var1":"Foo","Var2":"Bar","Var3":"Alpha","Var4":"Beta"}

Which tells me that what is likely happening is that the generic rsyslog template is adding the date, host, and programname to the beginning of our json. That's less than ideal, so we'll need to alter the specific file's configuration to remove those parts from our message. I'm working on a few possible solutions for this; Stay tuned

[mcapra]

Two possible solutions: modify the logstash input, or remove the default template from the rsyslog configuration.

remove the template from the rsyslog configuration

The simpler solution of the two for sure. Modify your rsyslog config (90-nagioslogserver_var_tmp_logs_test.log.conf) to include a template for the messages and apply that template to your output. This involves two changes:

Code: Select all

# create the template, just take the raw message with no additional info
$template cleanJson,"%rawmsg%"

...

# apply the template to your output
if $programname == 'import_json' then @@192.168.67.4:2057;cleanJson

I have modified your provided configuration file accordingly:

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for import_json
$InputFileName /var/tmp/logs/test.log
$InputFileTag import_json:
$InputFileStateFile nls-state-var_tmp_logs_test.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# template to send raw message by itself 
$template cleanJson,"%rawmsg%"

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'import_json' then @@dev444.dev.e2open.com:2057;cleanJson
if $programname == 'import_json' then ~

Be sure to do a service rsyslog restart when modifying your rsyslog configurations. Applying this produced the following events moving forward:

2016_10_18_16_43_15_Dashboard_Nagios_Log_Server.png

modify the logstash input

This solution is pretty hacky and definitely the less ideal solution, but I am providing it in the event that the first solution doesn't work.

I used the following input rule as a replacement for "Import Files - JSON (Default)":

Code: Select all

tcp {
    type => 'import_json'
    tags => 'import_json'
    port => 2057
}

Then crafted a filter to sanitize the message and parse it as JSON:

Code: Select all

if [type] == 'import_json' {
	mutate {
		gsub => [
		  # remove everything before first JSON bracket
		  "message", "^[^{]*{", "{"
		]
	  }
	  json {
          source => "message"
        }
}

Which produced the following event:

2016_10_18_16_29_16_Dashboard_Nagios_Log_Server.png

[opene2]

I will try these options in a day or two and let you know the results.

[dwhitfield]

Sounds good. Let us know any updates!

[opene2]

Both options worked for me. Only drawback with option 1 is that I can't see the original message in the dashboard.

Thanks for your help.

[dwhitfield]

You are very welcome. Is it ok if we lock this thread?

[opene2]

Yes