Nagios Support Forum

Posted: **Wed May 03, 2017 11:20 am**

I am forwarding kiwi syslog messages to Nagios LS...and I am receiving the events with no problem, however, I need some help setting up filters to match
host=Original Address=10.29.22.26 NYUNY085KB1 (example) . Hostname will always be 11 digits after the IP Address..but some entries does not have the hostname...only IP.

2017-05-03T11:13:49.946-05:00 192.168.216.14 syslog <12>May 3 07:13:50 10.29.22.26 Kiwi_Syslog_Server Original Address=10.29.22.26 NYUNY085KB1: PMOAIL-HC slot2 In-Line2_Out Alm IL2 Tx Pwr Low Warning ON
2017-05-03T11:13:49.744-05:00 192.168.216.14 syslog <131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513309 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
2017-05-03T11:13:49.543-05:00 192.168.216.14 syslog <131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513306 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
2017-05-03T11:13:49.543-05:00 192.168.216.14 syslog <131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513307 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
2017-05-03T11:13:49.543-05:00 192.168.216.14 syslog <131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513305 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
2017-05-03T11:13:49.543-05:00 192.168.216.14 syslog <131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513308 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 5 file-id 5]: Log file cf3:\act\ac...
2017-05-03T11:13:49.136-05:00 192.168.216.14 syslog <131>May 3 07:13:50 10.96.2.66 Kiwi_Syslog_Server Original Address=10.96.2.66 May 3 12:13:50 10.96.2.66 VAVIR019AT1: 503788 Base SECURITY-MINOR-ssh_user_logout-2010 [aluadmin]: User aluadmin from 192.168.232.22 logge...

Posted: **Wed May 03, 2017 12:05 pm**

Can you provide some of the raw log files you're trying to capture, as well as the rsyslog rules used to ship the log to Nagios Log Server?

Step 1 is being able to isolate your Kiwi traffic. Some people will do this with separate input rules running on separate ports. This would allow you to designate specific ports as being responsible for specific traffic. 2090 for Kiwi traffic, 2091 for Netgear traffic, etc etc.

If you were to change nothing at all, one way to isolate the traffic might be the "host" field originally set by the default Logstash input rule for on 5544 (using the "syslog" input type). It looks like the "host" of these entries all originate from 192.168.216.14 currently, so the start of your filter rule might look something like this:

Code: Select all

if [host] == "192.168.216.14" {
   #do my kiwi filtering here
}

Step 2 might be to figure out how your messages are structured. It looks like these are the raw messages received from rsyslog:

Code: Select all

<12>May 3 07:13:50 10.29.22.26 Kiwi_Syslog_Server Original Address=10.29.22.26 NYUNY085KB1: PMOAIL-HC slot2 In-Line2_Out Alm IL2 Tx Pwr Low Warning ON
<131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513309 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
<131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513306 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
<131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513307 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
<131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513305 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...
<131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513308 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 5 file-id 5]: Log file cf3:\act\ac...
<131>May 3 07:13:50 10.96.2.66 Kiwi_Syslog_Server Original Address=10.96.2.66 May 3 12:13:50 10.96.2.66 VAVIR019AT1: 503788 Base SECURITY-MINOR-ssh_user_logout-2010 [aluadmin]: User aluadmin from 192.168.232.22 logge...

You can match them up to the "message" field, since this is typically what Nagios Log Server received without any additional processing; Just the message itself.

I don't have the complete log entries, since it looks like this was copied from the GUI and was likely truncated. But working with what I have now, I've built the following grok rule:

Code: Select all

<131>May 3 07:13:50 10.21.26.80 Kiwi_Syslog_Server Original Address=10.21.26.80 May 3 12:13:50 10.21.26.80 NYUNY843SP1: 513309 Base LOGGER-MINOR-tmnxLogFileDeleted-2009 [acct-log-id 14 file-id 16]: Log file cf3:\act\...

Address=%{DATA:original_address} %{GREEDYDATA}

Toss these into the top and bottom windows on this grok debugger tool:
http://grokdebug.herokuapp.com/

And you should see how the individual fields are pulled out:

2017_05_03_12_02_59_Grok_Debugger.png

Essentially, grok filters use regex matches to pull fields out of your raw messages. In the above example, I pulled out the "original_address" field. The end result of all this might be a grok filter defined like so:

Code: Select all

if [host] == "192.168.216.14" {
    grok {
            match => [ 'message', 'Address=%{DATA:original_address} %{GREEDYDATA}']
        }
}

You'd need to build it up from there with additional regular expressions and pre-existing patterns (i used the DATA and GREEDYDATA patterns above) to get all the fields you want.

Posted: **Mon May 08, 2017 12:51 pm**

I really appreciate your reply.

I tried the filter that you suggested, but it will still not replace the host to the "Original Address"

Here is the RAW:

Code: Select all

{
  "_index": "logstash-2017.05.08",
  "_type": "syslog",
  "_id": "AVvpLelhmDPeuieCW2uc",
  "_score": null,
  "_source": {
    "message": "<131>May  8 08:49:16 10.21.5.138 Kiwi_Syslog_Server Original Address=10.21.5.138 May  8 13:49:17 10.21.5.138 PAWPA597AC1: 530930 Base LOGGER-MINOR-tmnxLogFileRollover-2008 [acct-log-id 13 file-id 15]:  Log file cf1:\\act\\act1315-20170508-133417.xml.gz on compact flash cf1: has been rolled over",
    "@version": "1",
    "@timestamp": "2017-05-08T17:49:11.260Z",
    "type": "syslog",
    "host": "192.168.216.14",
    "tags": [
      "_grokparsefailure_sysloginput"
    ],
    "priority": 0,
    "severity": 0,
    "facility": 0,
    "facility_label": "kernel",
    "severity_label": "Emergency"
  },
  "sort": [
    1494265751260,
    1494265751260
  ]
}

Posted: **Mon May 08, 2017 3:46 pm**

The rule provided would have created a new field called "original_address" and was not intended to replace the value in the existing host field.

That said, double check the filter rule - it seems there's an error in there somewhere as the output you provided did not include an original_address field. Once you have that field available and populating, you can replace the values in the host field with:

mutate {
replace => { "host" => "%{original_address}" }
}

or you could remove the host field and rename name the original_address field:

mutate { remove_field => [ "host" ] }

mutate {rename => { "original_address" => "host" } }

Nagios Support Forum

Map fields from Kiwi

Map fields from Kiwi

Re: Map fields from Kiwi

Re: Map fields from Kiwi

Re: Map fields from Kiwi