Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Filtering on IP ranges

https://support.nagios.com/forum/viewtopic.php?t=40029

Page 1 of 1

Filtering on IP ranges

Posted: Tue Aug 30, 2016 3:45 am
by ljorg
Hi,

We're using Nagios Log Server to handle firewall logs from pfSense, and it's mostly working great. I have one issue though when trying to filter IP ranges.

This works:

field: dest_ip
query: [130.226.228.1 TO 130.226.228.128]

This doesn't

field: dest_ip
query: [130.226.228.64 TO 130.226.228.128]

Am I missing something obvious?

Re: Filtering on IP ranges

Posted: Tue Aug 30, 2016 9:28 am
by mcapra
Those look fine to me. Can you run the following from the NLS CLI and see if it returns any entries:

Code: Select all

curl -XPOST "http://localhost:9200/_search?pretty" -d '{"query":{"filtered":{"filter":{"range":{"dest_ip":{"from":"130.226.228.64","to":"130.226.228.128"}}}}}}}'

Re: Filtering on IP ranges

Posted: Thu Sep 01, 2016 9:11 am
by ljorg
No, nothing:

Code: Select all

 curl -XPOST "http://localhost:9200/_search?pretty" -d '{"query":{"filtered":{"filter":{"range":{"dest_ip":{"from":"130.226.228.64","to":"130.226.228.128"}}}}}}}'
{
  "took" : 27,
  "timed_out" : false,
  "_shards" : {
    "total" : 216,
    "successful" : 216,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}
If I change the last octet of the first IP address in the range to .1, I get 314,537 hits. Only (the first?) 10 are displayed so I can't confirm whether addresses from the .64-.128 range is included, but I suspect they are.

Even weirder: If I search in the range .1 to .63, I get 1,262,388 hits.

Re: Filtering on IP ranges

Posted: Thu Sep 01, 2016 11:30 am
by mcapra
It might have something to do with the field type being used on the back-end by elasticsearch.

Run the following from the CLI and share the output:

Code: Select all

curl -XGET 'http://localhost:9200/logstash-2016.08.31/_mapping/'
Might be a bit large depending on your environment.

Re: Filtering on IP ranges

Posted: Fri Sep 02, 2016 1:46 am
by ljorg
Here you go:

Code: Select all

{"logstash-2016.08.31":{"mappings":{"_default_":{"dynamic_templates":[{"string_fields":{"mapping":{"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"index":"not_analyzed","ignore_above":256,"type":"string"}}},"match":"*","match_mapping_type":"string"}}],"_all":{"enabled":true},"properties":{"@version":{"type":"string","index":"not_analyzed"},"geoip":{"dynamic":"true","properties":{"location":{"type":"geo_point"}}}}},"syslog":{"dynamic_templates":[{"string_fields":{"mapping":{"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"index":"not_analyzed","ignore_above":256,"type":"string"}}},"match":"*","match_mapping_type":"string"}}],"_all":{"enabled":true},"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"@version":{"type":"string","index":"not_analyzed"},"action":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"class":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"data_length":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"dest_ip":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"dest_port":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"direction":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"facility":{"type":"long"},"facility_label":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"flags":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"flow_label":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"geoip":{"dynamic":"true","properties":{"location":{"type":"geo_point"}}},"hop_limit":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"host":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"id":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"interface":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"ip_version":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"length":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"logsource":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"message":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"offset":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"pid":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"priority":{"type":"long"},"program":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"protocol":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"protocol_id":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"reason":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"rest":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"rule_number":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"severity":{"type":"long"},"severity_label":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"src_ip":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"src_port":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"sub_rule_number":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"tags":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"timestamp":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"tos":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"tracker_id":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"ttl":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"type":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}}}}}}}

Re: Filtering on IP ranges

Posted: Fri Sep 02, 2016 10:16 am
by mcapra
The problem is the dest_ip field is a string type. Trying to use a range operator like TO on a string is going to produce some weird behavior for sure.

Code: Select all

"dest_ip":{  
                  "type":"string",
                  "norms":{  
                     "enabled":false
                  },
                  "fields":{  
                     "raw":{  
                        "type":"string",
                        "index":"not_analyzed",
                        "ignore_above":256
                     }
                  }
               }
Ideally we would cast this field to a IPv4 type and solve all of our problems right there, but that type is not available in the version of elasticsearch Nagios Log Server is currently using to my knowledge. You could certainly give it a try though:
https://www.elastic.co/guide/en/elastic ... nt/ip.html

You could also probably write a grok filter to pull each octet into it's own int field and filter that way.

Re: Filtering on IP ranges

Posted: Mon Sep 05, 2016 2:06 am
by ljorg
I think I am already casting it as an IP type. This is part of my input filter:

Code: Select all

		
grok {
	  match => [ 'message', '%{WORD:tos},(\d+)?,%{INT:ttl},%{INT:id},%{INT:offset},%{WORD:flags},%{INT:protocol_id},%{WORD:protocol},%{INT:length},%{IP:src_ip},%{IP:dest_ip},%{GREEDYDATA:rest3}' ]
}
So the problem must be the version of elastic search running inside Nagios Log Server. When will it be updated to support the IP type?

Re: Filtering on IP ranges

Posted: Tue Sep 06, 2016 7:54 am
by eloyd
I've been waiting for an answer to the following question for a long time, for other reasons. It would be nice if NLS, which I consider a Nagios flagship product, were using "mostly current" versions of ES and LS.
ljorg wrote:When will it be updated to support the IP type?

Re: Filtering on IP ranges

Posted: Tue Sep 06, 2016 9:17 am
by mcapra
2.0 is intended to contain more modern versions of the ELK components. Unfortunately I am unable to give a specific release date at this time.
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited