Hi there,
I'm new to Nagios LS and just mount the virtual machine Nagios Log Server trial (downloaded from Nagios).
So we receive the logs from our firewall and the "message" is like this:
<30>device="SFW" date=2019-10-29 time=16:24:23 timezone="+07" device_name="XG230" device_id=C9867FFFPM287E7 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=106 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=8 application="" application_risk=0 application_technology="" application_category="" in_interface="Port1" out_interface="Port2" src_mac=00:00:00:00:00:00 src_ip=10.10.10.10 src_country_code=R1 dst_ip=42.115.194.133 dst_country_code=USA protocol="TCP" src_port=54312 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=105.69.29.106 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="1423586464" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0
I would like to remove the unnecessary time information in the message and break the message in to fields.
So I put this in filter:
if [type] == 'fwlog' {
dissect {
mapping => {
"message" => "%{} %{} %{} %{} %{[@metadata][restOfLine]}"
}
}
kv {
source => "[@metadata][restOfLine]"
}
}
However, it didn't pass the verification:
{:timestamp=>"2019-10-29T16:33:47.919000+0700", :message=>"The given configuration is invalid. Reason: Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load the dissect filter plugin resulted in this error: load error: jruby_dissector -- java.lang.NoClassDefFoundError: com/logstash/ext/JrubyEventExtLibrary$RubyEvent", :level=>:fatal}
It's very odd because when I run "/usr/local/nagioslogserver/logstash/bin/logstash-plugin list --verbose" I see:
logstash-filter-dissect (1.0.6)
Could someone please help ?
Nagios LS - Logstash filter dissect
Re: Nagios LS - Logstash filter dissect
I'm seeing the same behavior and looking into it, but it isn't listed in the Logstash 2.4.1 which is what NLS uses:
https://www.elastic.co/guide/en/logstas ... ugins.html
https://www.elastic.co/guide/en/logstas ... ugins.html
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 12
- Joined: Mon Oct 28, 2019 2:54 am
Re: Nagios LS - Logstash filter dissect
Hi @cdienger
So dissect is not included in Logstash 2.4.1 which is what NLS uses.
I installed successfully the dissect filter manually (/usr/local/nagioslogserver/logstash/bin/logstash-plugin install logstash-filter-dissect). But I still got the same error for the verification.
How can I break the message into fields ?
If I replace the filter by this:
if [type] == 'fwlog' {
kv { }
}
or by this:
if [type] == 'fwlog' {
mutate { gsub => [ "message", "^<.+>", "" ] }
kv { }
}
Then in dashboard it didn't show new logs.
If I removed "kv { }" and kept the "mutate { gsub => [ "message", "^<.+>", "" ] }" then the dashboard showed new logs like this (the <30> at the beginning of the message was removed).
device="SFW" date=2019-10-29 time=16:24:23 timezone="+07" device_name="XG230" device_id=C9867FFFPM287E7 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=106 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=8 application="" application_risk=0 application_technology="" application_category="" in_interface="Port1" out_interface="Port2" src_mac=00:00:00:00:00:00 src_ip=10.10.10.10 src_country_code=R1 dst_ip=42.115.194.133 dst_country_code=USA protocol="TCP" src_port=54312 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=105.69.29.106 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="1423586464" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0
So dissect is not included in Logstash 2.4.1 which is what NLS uses.
I installed successfully the dissect filter manually (/usr/local/nagioslogserver/logstash/bin/logstash-plugin install logstash-filter-dissect). But I still got the same error for the verification.
How can I break the message into fields ?
If I replace the filter by this:
if [type] == 'fwlog' {
kv { }
}
or by this:
if [type] == 'fwlog' {
mutate { gsub => [ "message", "^<.+>", "" ] }
kv { }
}
Then in dashboard it didn't show new logs.
If I removed "kv { }" and kept the "mutate { gsub => [ "message", "^<.+>", "" ] }" then the dashboard showed new logs like this (the <30> at the beginning of the message was removed).
device="SFW" date=2019-10-29 time=16:24:23 timezone="+07" device_name="XG230" device_id=C9867FFFPM287E7 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=0 fw_rule_id=106 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=8 application="" application_risk=0 application_technology="" application_category="" in_interface="Port1" out_interface="Port2" src_mac=00:00:00:00:00:00 src_ip=10.10.10.10 src_country_code=R1 dst_ip=42.115.194.133 dst_country_code=USA protocol="TCP" src_port=54312 dst_port=443 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip=105.69.29.106 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Start" connid="1423586464" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0
Re: Nagios LS - Logstash filter dissect
I tested this and it seems to do what you're looking for:
I'm not sure why the kv{} filter wouldn't work for you, but debugging may help identify why:
Edit /etc/init.d/logstash and change line 64 from:
to:
and restart the service with:
The output is sent to /var/log/logstash/logstash.log. Make sure to revert the changes when done.
Code: Select all
if [type] == 'import_raw' {
mutate { gsub => [ "message", "^<.+>", "" ]
}
kv { }
mutate { remove_field => [ "date", "time", "timezone" ]
}
}
Edit /etc/init.d/logstash and change line 64 from:
Code: Select all
DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"
Code: Select all
DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"
Code: Select all
systemctl daemon-reload
service logstash restart
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
-
- Posts: 12
- Joined: Mon Oct 28, 2019 2:54 am
Re: Nagios LS - Logstash filter dissect
Thank you very much for your help.
As I wrote in this topic: https://support.nagios.com/forum/viewto ... 70#p296270
The "kv {}" didn't work at first, but then it worked 8 hours later.
I don't really know why.
However I will try your solution for other network devices (e.g. switches).
I appreciate much your detailed instruction.
As I wrote in this topic: https://support.nagios.com/forum/viewto ... 70#p296270
The "kv {}" didn't work at first, but then it worked 8 hours later.
I don't really know why.
However I will try your solution for other network devices (e.g. switches).
I appreciate much your detailed instruction.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Nagios LS - Logstash filter dissect
Glad it is now working!jameshanguyen wrote:Thank you very much for your help.
As I wrote in this topic: https://support.nagios.com/forum/viewto ... 70#p296270
The "kv {}" didn't work at first, but then it worked 8 hours later.
I don't really know why.
However I will try your solution for other network devices (e.g. switches).
I appreciate much your detailed instruction.
Locking thread