Page 1 of 2
JSON files
Posted: Fri Oct 14, 2016 9:48 am
by opene2
I am trying to import JSON formatted log files into NLS. I am getting ""_jsonparsefailure" error. Could someone please tell me what I am doing wrong with in input and/or filter.
Sample log file:
{"Var1":"Foo","Var2":Bar,"Var3":"Alpha","Var4":"Beta"}
{"Var1":"Foo","Var2":Bar,"Var3":"Alpha","Var4":"Beta"}
Input:
tcp {
type => 'jason_test'
tags => 'jason_test'
port => 2100
}
Filter:
if [type] == 'jason_test' {
json {
source => "message"
}
}
Re: JSON files
Posted: Fri Oct 14, 2016 9:53 am
by mcapra
You shouldn't actually need a filter. Setting the codec on your input to JSON should be sufficient. Here's the default JSON input that comes with NLS:
Code: Select all
tcp {
type => 'import_json'
tags => 'import_json'
port => 2057
codec => json
}
This doesn't always work well with multi-line "pretty" JSON though, for the record.
Re: JSON files
Posted: Fri Oct 14, 2016 10:52 am
by opene2
I've tried "import_json" but no luck. I am seeing "_jsonparsefailure,import_json" tags in the dashboard. I've also tried changing codec like the following.
codec => json_lines
Re: JSON files
Posted: Fri Oct 14, 2016 1:47 pm
by mcapra
Are you sure you're sending properly formatted JSON? I notice the JSON in your original post that you provided as an example is malformed. If you could provide a few actual events from the log file, I could more accurately figure out what's going on.
If I send a generic udp packet to the JSON input like so:
Code: Select all
nc -w 1 -u 192.168.67.4 2057 <<< '{"Var1":"Foo","Var2":"Bar","Var3":"Alpha","Var4":"Beta"}'
I get the following event on a fresh install of NLS:
2016_10_14_13_46_18_Dashboard_Nagios_Log_Server.png
Re: JSON files
Posted: Mon Oct 17, 2016 8:17 am
by opene2
If I use netcat command, it works.
Code: Select all
nc -w 1 <nls_server> 2057 <<< `cat /var/tmp/logs/test.log`
If I use "setup-linux.sh" command, I am getting "_jsonparsefailure" error.
Code: Select all
bash setup-linux.sh -s <nls_server> -p 2057 -f "/var/tmp/logs/test.log" -t import_json
Re: JSON files
Posted: Mon Oct 17, 2016 10:37 am
by rkennedy
Could you show us the data contained in test.log so we can attempt to replicate this in house?
Re: JSON files
Posted: Mon Oct 17, 2016 10:52 am
by opene2
This is what I have in test.log file. I am using NLS 1.4.2
Code: Select all
{"Var1":"Foo","Var2":"Bar","Var3":"Alpha","Var4":"Beta"}
Re: JSON files
Posted: Mon Oct 17, 2016 4:54 pm
by mcapra
There may be some rsyslog configurations to consider. From the CLI of the machine that is sending the JSON file, can you share the outputs of:
Code: Select all
ls -al /etc/rsyslog.d/
cat /etc/rsyslog.d/*.conf
Re: JSON files
Posted: Mon Oct 17, 2016 5:00 pm
by avandemore
This works for me when importing from a file: /var/www/html/nagioslogserver/www/scripts/setup-linux.sh -s localhost -p 2057 -f /tmp/log.log -t import_json
Re: JSON files
Posted: Mon Oct 17, 2016 5:41 pm
by opene2
Problem is not with importing the file into Nagios Log Server. After importing the file, Nagios Log Server is not applying the filters properly. If you take a closer look at your output, you would see "_jasonparsefailure" in tags.
Following is the "nc" command and screenshot from Nagios Log Server. You can see that NLS properly assigned fields Var1, Var2 etc..
Code: Select all
nc -w 1 localhost 2057 <<< '{"Var1":"Foo","Var2":"Bar","Var3":"Alpha","Var4":"Beta"}'