Nagios LS - Break message into fields

[jameshanguyen]

Hi,
So I receive logs from my firewall.
Each message is like this:

<30>device="SFW" date=2019-10-29 time=16:24:23 timezone="+07" device_name="XG230" device_id=C9867FFFPM287E7 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=106 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=8 application="" application_risk=0 application_technology="" application_category="" in_interface="Port1" out_interface="Port2" src_mac=00:00:00:00:00:00 src_ip=10.10.10.10 src_country_code=R1 dst_ip=42.115.194.133 dst_country_code=USA protocol="TCP" src_port=54312 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=105.69.29.106 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="1423586464" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

How could I configure the filter (or what I should do generally) to break this message into fields ?
Currently it has only these fields which don't help much:
@timestamp
@version
_id
_index
_type
alert_names
facility
facility_label
highlight
host
message
priority
severity
severity_label
tags
type

[scottwilkerson]

You could add a filter like this replacing xxx.xxx.xxx.xxx with the hosts IP address

Code: Select all

if [host] == 'xxx.xxx.xxx.xxx' {
    kv { }
}

This will split everything into key/value pairs on the = symbol
https://www.elastic.co/guide/en/logstas ... rs-kv.html

[jameshanguyen]

Hi scottwilkerson,
After adding the filter and applying configuration, I didn't receive any more logs in dashboard.
Could you please tell me how I can debug it ?
Thanks.

[scottwilkerson]

You added it as a filter correct?

[jameshanguyen]

Wow,
It didn't work yesterday after I added it as a filter.
But after 8 hours now it works.
I don't really understand why it works, but it's good to see that.
Thank you very much for your help.

[scottwilkerson]

Wow,
It didn't work yesterday after I added it as a filter.
But after 8 hours now it works.
I don't really understand why it works, but it's good to see that.
Thank you very much for your help.

Glad it is working.

There is a slight possibility that something made yesterdays index have bad mapping for one of the fields.

Glad it is sorted!

Locking thread