Page 1 of 1
Log Server Internal Logs parsefailure
Posted: Fri Aug 26, 2016 3:40 pm
by CFT6Server
So I haven't checked one of our NLS instance and looks like it had stopped working. Once I restarted the server (single node instance), I started seeing import_json/jsonparsefailure messages for local logs. I don't think anything have changed here, so any ideas?
The host field also looks suspicious.
importjson.png
Re: Log Server Internal Logs parsefailure
Posted: Mon Aug 29, 2016 9:56 am
by mcapra
Can you share your inputs and filters on this instance?
Also what is suspicious about the host field? Are you expecting a host other than localhost?
Re: Log Server Internal Logs parsefailure
Posted: Mon Aug 29, 2016 1:35 pm
by CFT6Server
I do find the host field odd as well. No additional configuration was done for any localhost syslogs.
Config attached. (FYI - I removed part of the config to remove some hostnames etc)
Re: Log Server Internal Logs parsefailure
Posted: Mon Aug 29, 2016 2:17 pm
by mcapra
Are these entries persisting through multiple indices?
I would do a tcpdump on the import_json port (2057) and see what sort of traffic is coming in through that port. tcpdump -i eth0 'port 2057' changing eth0 to reflect the logical name of the network interface. Very strange that a syslog message would make it through that path.
And at the risk of sounding like a broken record: Can you show us your elasticsearch log? Feel free to PM it to me if you have security concerns. It should be located at /var/log/elasticsearch/<cluster id>.log
Re: Log Server Internal Logs parsefailure
Posted: Sun Sep 04, 2016 9:34 pm
by eloyd
Your host field is the localhost in IPv6 notation. Nothing weird about that unless it's incorrect. I'm confused about your filters though. I see you're doing raw json input for what looks like syslog input (literally, this looks like output from /var/log/syslog or /var/log/messages) so I'm not sure why you're doing that. Short of knowing what the specific input host really is (I have to assume that IPv6 localhost is correct) then it appears that none of your filters are actually triggering, since the type and tags and host don't match any of your filter matches.
You also do this a lot:
Code: Select all
if "_grokparsefailure_sysloginput" in [tags]{
mutate {
remove_tag => ["_grokparsefailure_sysloginput"]
}
}
Which, I understand, but I'm not sure. Why not fix the grokparse failure instead of remove the warning message, and even so, you can just ignore it if you're not going to fix it.
Re: Log Server Internal Logs parsefailure
Posted: Tue Sep 06, 2016 9:28 am
by tmcdonald
@eloyd thanks as always for the input!