Log Server Internal Logs parsefailure

[CFT6Server]

So I haven't checked one of our NLS instance and looks like it had stopped working. Once I restarted the server (single node instance), I started seeing import_json/jsonparsefailure messages for local logs. I don't think anything have changed here, so any ideas?

The host field also looks suspicious.

importjson.png

[mcapra]

Can you share your inputs and filters on this instance?

Also what is suspicious about the host field? Are you expecting a host other than localhost?

[CFT6Server]

I do find the host field odd as well. No additional configuration was done for any localhost syslogs.

Config attached. (FYI - I removed part of the config to remove some hostnames etc)

[mcapra]

Are these entries persisting through multiple indices?

I would do a tcpdump on the import_json port (2057) and see what sort of traffic is coming in through that port. tcpdump -i eth0 'port 2057' changing eth0 to reflect the logical name of the network interface. Very strange that a syslog message would make it through that path.

And at the risk of sounding like a broken record: Can you show us your elasticsearch log? Feel free to PM it to me if you have security concerns. It should be located at /var/log/elasticsearch/<cluster id>.log

[eloyd]

Your host field is the localhost in IPv6 notation. Nothing weird about that unless it's incorrect. I'm confused about your filters though. I see you're doing raw json input for what looks like syslog input (literally, this looks like output from /var/log/syslog or /var/log/messages) so I'm not sure why you're doing that. Short of knowing what the specific input host really is (I have to assume that IPv6 localhost is correct) then it appears that none of your filters are actually triggering, since the type and tags and host don't match any of your filter matches.

You also do this a lot:

Code: Select all

       if "_grokparsefailure_sysloginput" in [tags]{
            mutate {
               remove_tag => ["_grokparsefailure_sysloginput"]
            }
        }

Which, I understand, but I'm not sure. Why not fix the grokparse failure instead of remove the warning message, and even so, you can just ignore it if you're not going to fix it.

[tmcdonald]

@eloyd thanks as always for the input!