Filtering on IP ranges

[ljorg]

Hi,

We're using Nagios Log Server to handle firewall logs from pfSense, and it's mostly working great. I have one issue though when trying to filter IP ranges.

This works:

field: dest_ip
query: [130.226.228.1 TO 130.226.228.128]

This doesn't

field: dest_ip
query: [130.226.228.64 TO 130.226.228.128]

Am I missing something obvious?

[mcapra]

Those look fine to me. Can you run the following from the NLS CLI and see if it returns any entries:

Code: Select all

curl -XPOST "http://localhost:9200/_search?pretty" -d '{"query":{"filtered":{"filter":{"range":{"dest_ip":{"from":"130.226.228.64","to":"130.226.228.128"}}}}}}}'

[ljorg]

No, nothing:

Code: Select all

 curl -XPOST "http://localhost:9200/_search?pretty" -d '{"query":{"filtered":{"filter":{"range":{"dest_ip":{"from":"130.226.228.64","to":"130.226.228.128"}}}}}}}'
{
  "took" : 27,
  "timed_out" : false,
  "_shards" : {
    "total" : 216,
    "successful" : 216,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

If I change the last octet of the first IP address in the range to .1, I get 314,537 hits. Only (the first?) 10 are displayed so I can't confirm whether addresses from the .64-.128 range is included, but I suspect they are.

Even weirder: If I search in the range .1 to .63, I get 1,262,388 hits.

[mcapra]

It might have something to do with the field type being used on the back-end by elasticsearch.

Run the following from the CLI and share the output:

Code: Select all

curl -XGET 'http://localhost:9200/logstash-2016.08.31/_mapping/'

Might be a bit large depending on your environment.

[ljorg]

Here you go:

Code: Select all

{"logstash-2016.08.31":{"mappings":{"_default_":{"dynamic_templates":[{"string_fields":{"mapping":{"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"index":"not_analyzed","ignore_above":256,"type":"string"}}},"match":"*","match_mapping_type":"string"}}],"_all":{"enabled":true},"properties":{"@version":{"type":"string","index":"not_analyzed"},"geoip":{"dynamic":"true","properties":{"location":{"type":"geo_point"}}}}},"syslog":{"dynamic_templates":[{"string_fields":{"mapping":{"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"index":"not_analyzed","ignore_above":256,"type":"string"}}},"match":"*","match_mapping_type":"string"}}],"_all":{"enabled":true},"properties":{"@timestamp":{"type":"date","format":"dateOptionalTime"},"@version":{"type":"string","index":"not_analyzed"},"action":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"class":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"data_length":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"dest_ip":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"dest_port":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"direction":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"facility":{"type":"long"},"facility_label":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"flags":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"flow_label":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"geoip":{"dynamic":"true","properties":{"location":{"type":"geo_point"}}},"hop_limit":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"host":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"id":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"interface":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"ip_version":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"length":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"logsource":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"message":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"offset":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"pid":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"priority":{"type":"long"},"program":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"protocol":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"protocol_id":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"reason":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"rest":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"rule_number":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"severity":{"type":"long"},"severity_label":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"src_ip":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"src_port":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"sub_rule_number":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"tags":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"timestamp":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"tos":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"tracker_id":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"ttl":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},"type":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}}}}}}}

[mcapra]

The problem is the dest_ip field is a string type. Trying to use a range operator like TO on a string is going to produce some weird behavior for sure.

Code: Select all

"dest_ip":{  
                  "type":"string",
                  "norms":{  
                     "enabled":false
                  },
                  "fields":{  
                     "raw":{  
                        "type":"string",
                        "index":"not_analyzed",
                        "ignore_above":256
                     }
                  }
               }

Ideally we would cast this field to a IPv4 type and solve all of our problems right there, but that type is not available in the version of elasticsearch Nagios Log Server is currently using to my knowledge. You could certainly give it a try though:
https://www.elastic.co/guide/en/elastic ... nt/ip.html

You could also probably write a grok filter to pull each octet into it's own int field and filter that way.

[ljorg]

I think I am already casting it as an IP type. This is part of my input filter:

Code: Select all

		
grok {
	  match => [ 'message', '%{WORD:tos},(\d+)?,%{INT:ttl},%{INT:id},%{INT:offset},%{WORD:flags},%{INT:protocol_id},%{WORD:protocol},%{INT:length},%{IP:src_ip},%{IP:dest_ip},%{GREEDYDATA:rest3}' ]
}

So the problem must be the version of elastic search running inside Nagios Log Server. When will it be updated to support the IP type?

[eloyd]

I've been waiting for an answer to the following question for a long time, for other reasons. It would be nice if NLS, which I consider a Nagios flagship product, were using "mostly current" versions of ES and LS.

When will it be updated to support the IP type?

[mcapra]

2.0 is intended to contain more modern versions of the ELK components. Unfortunately I am unable to give a specific release date at this time.