Nagios LS - Logstash filter dissect

[jameshanguyen]

Hi there,
I'm new to Nagios LS and just mount the virtual machine Nagios Log Server trial (downloaded from Nagios).
So we receive the logs from our firewall and the "message" is like this:

<30>device="SFW" date=2019-10-29 time=16:24:23 timezone="+07" device_name="XG230" device_id=C9867FFFPM287E7 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=106 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=8 application="" application_risk=0 application_technology="" application_category="" in_interface="Port1" out_interface="Port2" src_mac=00:00:00:00:00:00 src_ip=10.10.10.10 src_country_code=R1 dst_ip=42.115.194.133 dst_country_code=USA protocol="TCP" src_port=54312 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=105.69.29.106 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="1423586464" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

I would like to remove the unnecessary time information in the message and break the message in to fields.
So I put this in filter:

if [type] == 'fwlog' {
dissect {
mapping => {
"message" => "%{} %{} %{} %{} %{[@metadata][restOfLine]}"
}
}
kv {
source => "[@metadata][restOfLine]"
}
}

However, it didn't pass the verification:

{:timestamp=>"2019-10-29T16:33:47.919000+0700", :message=>"The given configuration is invalid. Reason: Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load the dissect filter plugin resulted in this error: load error: jruby_dissector -- java.lang.NoClassDefFoundError: com/logstash/ext/JrubyEventExtLibrary$RubyEvent", :level=>:fatal}

It's very odd because when I run "/usr/local/nagioslogserver/logstash/bin/logstash-plugin list --verbose" I see:

logstash-filter-dissect (1.0.6)

Could someone please help ?

[cdienger]

I'm seeing the same behavior and looking into it, but it isn't listed in the Logstash 2.4.1 which is what NLS uses:

https://www.elastic.co/guide/en/logstas ... ugins.html

[jameshanguyen]

Hi @cdienger
So dissect is not included in Logstash 2.4.1 which is what NLS uses.
I installed successfully the dissect filter manually (/usr/local/nagioslogserver/logstash/bin/logstash-plugin install logstash-filter-dissect). But I still got the same error for the verification.

How can I break the message into fields ?

If I replace the filter by this:

if [type] == 'fwlog' {
kv { }
}

or by this:

if [type] == 'fwlog' {
mutate { gsub => [ "message", "^<.+>", "" ] }
kv { }
}

Then in dashboard it didn't show new logs.

If I removed "kv { }" and kept the "mutate { gsub => [ "message", "^<.+>", "" ] }" then the dashboard showed new logs like this (the <30> at the beginning of the message was removed).

device="SFW" date=2019-10-29 time=16:24:23 timezone="+07" device_name="XG230" device_id=C9867FFFPM287E7 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=106 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=8 application="" application_risk=0 application_technology="" application_category="" in_interface="Port1" out_interface="Port2" src_mac=00:00:00:00:00:00 src_ip=10.10.10.10 src_country_code=R1 dst_ip=42.115.194.133 dst_country_code=USA protocol="TCP" src_port=54312 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=105.69.29.106 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="1423586464" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

[cdienger]

I tested this and it seems to do what you're looking for:

Code: Select all

if [type] == 'import_raw' {
mutate { gsub => [ "message", "^<.+>", "" ] 
}
kv { }
mutate { remove_field => [ "date", "time", "timezone" ]
}
}

I'm not sure why the kv{} filter wouldn't work for you, but debugging may help identify why:

Edit /etc/init.d/logstash and change line 64 from:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and restart the service with:

Code: Select all

systemctl daemon-reload
service logstash restart

The output is sent to /var/log/logstash/logstash.log. Make sure to revert the changes when done.

[jameshanguyen]

Thank you very much for your help.
As I wrote in this topic: https://support.nagios.com/forum/viewto ... 70#p296270
The "kv {}" didn't work at first, but then it worked 8 hours later.
I don't really know why.
However I will try your solution for other network devices (e.g. switches).
I appreciate much your detailed instruction.

[scottwilkerson]

Thank you very much for your help.
As I wrote in this topic: https://support.nagios.com/forum/viewto ... 70#p296270
The "kv {}" didn't work at first, but then it worked 8 hours later.
I don't really know why.
However I will try your solution for other network devices (e.g. switches).
I appreciate much your detailed instruction.

Glad it is now working!

Locking thread