Unique values in a field

[opene2]

Is it possible to get total count of unique values in a field?

For example, I have a log the contains "clientIP:<IP_Address>". I can get how many times each IP address appears. What I am looking for is the total number of unique IP addresses.

[rkennedy]

Yes, I have something similar set up to detect failed SSH attempts. It allows me to aggregate and see who the top brute forcers are. It works pretty well.

The important thing we'll need to do, is get the clientIP & IP_Address field isolated so that all variables can be inputted there. This can be done by adding a grok to match your data, and push the data -> field.

Could you show us a screenshot of how the data is currently inputted from your NLS dashboard so that we can see all the populated variables currently?

[opene2]

Partial log file.

Code: Select all

20/Oct/2016:04:02:18 uuid:eca88c67-16fd-47f4-81f7-18204f35f560 clientIP:203.166.320.2
20/Oct/2016:04:02:18 uuid:b2431f27-c683-418d-b600-93410de1691f clientIP:85.0.73.186
20/Oct/2016:04:02:19 uuid:a224384c-df07-4a6d-9968-ba330ba8a6fc clientIP:67.51.99.47
20/Oct/2016:04:02:24 uuid:294dfc51-6d7a-4d71-af04-8a0c6238d1cf clientIP:214.69.154.224

Filter

Code: Select all

if [type] == 'unique_test' {
   grok {
      match => [ 'message', '%{DATA:datetime} uuid:%{DATA:uuid} clientIP:%{IPORHOST:clientIP}' ]
   }
}

Dashboard 1

NLS1.png

Dashboard 2

NLS2.png

Let me know if you need anything else.

[mcapra]

I answered this in a customer post previously. The short answer is that it's not possible right now, but I did file a feature request for it and it is within the realm of possibility for a future version.

I can certainly think of a few ways to do this, but there isn't anything built-in to Nagios Log Server (or elasticsearch to my knowledge) that can tackle this very effectively.

In MySQL, you would essentially need to do a COUNT on a nested SELECT DISTINCT to accomplish this. Might look something like this:

Code: Select all

SELECT COUNT(SELECT DISTINCT name FROM table) AS name_count FROM table;

Which would return the total number of distinct "name" items as the "name_count" field. However, there doesn't really exist a COUNT function in elasticsearch that behaves in that fashion. There are options for doing the equivalent of a SELECT DISTINCT and counting the occurrences of those distinct values:

Code: Select all

[root@localhost ~]# curl -XGET 'http://localhost:9200/_all/_search?search_type=count&pretty' -d '{"aggs":{"distinct_name":{"terms":{"field":"name"}}}}'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 41,
    "successful" : 41,
    "failed" : 0
  },
  "hits" : {
    "total" : 274844,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "distinct_name" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [ {
        "key" : "dave",
        "doc_count" : 6
      }, {
        "key" : "bob",
        "doc_count" : 5
      }, {
        "key" : "alice",
        "doc_count" : 1
      }, {
        "key" : "jeff",
        "doc_count" : 1
      }, {
        "key" : "kim",
        "doc_count" : 1
      } ]
    }
  }
}

What this API call has done is ask for the count of every distinct value for the "name" field. While there are several occurrences of the name "dave" for example, only one single buckets entry is created for "dave" occurrences. Ideally elasticsearch would have some method by which we could tally up the count of unique keys for each bucket, but this functionality does not exist.

It's more of a limitation within elasticsearch than it is Nagios Log Server, though it could be solved within Nagios Log Server. If I had to solve this problem, I would write a PHP script to parse that JSON object as a PHP array and get the size of the buckets array.

I've filed a feature request for this (ID 9839). I can see where that sort of information/alerting would be useful, with the example of "how many unique users are accessing ContentX" coming to mind when considering DDOS detection.

[opene2]

Thanks for the update. Looking forward to the next release that contains this feature. You can go ahead and lock this thread.