Query to count the number of occurrences a string?

[gormank]

I'm trying to make an NXI check on NLS that will alert when some log data doesn't show up. So I'm trying to make a query to count the number of occurrences a string, which I'll put in the NLS wizard in NXI.

Can you suggest a way to make a query to count the number of occurrences for the past x minutes of a string in NLS? Sort of like a select count(*) where message like 'somelog.log'; in SQL.

I tried using the following but it didn't work.
https://www.elastic.co/guide/en/elastic ... count.html

I suppose I could just snag the last few minutes of data in NLS and grep somelog.log but was hoping for a less gruesome solution...

Thanks!

[pbroste]

Hello @gormank

Thanks for reaching out, are you looking for a number of matches or logs that match a certain *filter*. I want to test this scenario on my test VM, please review the linked support article and let me know if think that this is a possible solution to test.

https://assets.nagios.com/downloads/nag ... ilters.pdf

Thanks,
Perry

[gormank]

I'm looking for a text query (not from the web GUI) that looks for a count of occurrences of message:somelog.log for the last 5 minutes.
I've read the doc you linked and it's more geared toward the GUI, which isn't much help.
Below are examples of my testing that aren't working.

curl --insecure -XGET 'https://solktxeshgw-v-pmeslog-01/nagios ... c211f4ec6d' -d '{_count?q=message:PolicyNetCacheWs_i.cpp}'

#!/usr/bin/sh

curl --insecure -XGET 'https://<hostname>/nagioslogserver/api/backend/logstash-2022.01.26/_search?pretty&token=<token>' -d \
'{
"facets": {
"terms": {
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "message:PolicyNetCacheWs_i.cpp"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"from":"now-5m",
"to":"now",
}
}
}
]
}
}
}
}
}
}
}
}
}'

[pbroste]

Hello @gormank

Could do something like this as only example by placing into script:

I am using this from my test VM:

Code: Select all

curl -sk -X GET "localhost:9200/logstash-2022.01.27/_count?"

MY example response:

{"count":511434,"_shards":{"total":5,"successful":5,"failed":0}}

What I am filtering:

Code: Select all

RESULT=$(curl -s -X GET "localhost:9200/logstash-2022.01.27/_count?" | grep -o 'total.*' | cut -f2- -d: | awk -F "," '{print $1}')

My example result:

>echo $RESULT
5

Adjusting the filter to get the results you desire to pull, including any text to obtain the desired outcome. This is one of the probably many options that are possible.

Thanks,
Perry

[gormank]

The count only works for the total number of messages, not the query part, which is the reason I'm asking for support.

[gormank]

You can reject my last post...
Actually, I see your syntax is different than what I was using, and works when I add the query. Now the count is correct. Can you help with limiting the time range to the last five minutes?

[nags-01 ~]$ curl --insecure -XGET 'https://log-01/nagioslogserver/api/back ... oken=token'; echo
{
"count": 15497789,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
}
}

[pbroste]

Hello @gormank

This is what I was able to cobble together while experimenting with api calls on my VM:

Code: Select all

curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "range" : {
            "date" : {
                "gte" : "now-5m/m",
                "lt" :  "now/m"
            }
        }
    }
}'

Results:

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 52,
"successful" : 52,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

Here is a link that references time range:
https://www.elastic.co/guide/en/elastic ... #date-math

Thanks,
Perry

[gormank]

After more than a bit of trial and error last week I figured out that when these queries are given bad info they seem to sort of ignore the bad query parts and just dump out a sampling of info. I had a few quoting issues and bad time formats. I eventually made a query in the UI, snagged that from the inspect button and started editing that.
Once I got those worked out with a _search, I just replaced it with _count and get the desired info in a pretty concise format.
I used your example of the time range and it now looks like I have what I'm looking for. Below is the output and my little test script.

Thanks much and feel free to close this one.

{
"count": 52265,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
}
}

#!/usr/bin/sh

host=solktxeshsc-v-pweslog-01
token=a148b6a40d996f0c8c1b0547daa06963bc9cb542

curl --insecure -X GET -H 'Content-Type: application/json' \
"https://${host}/nagioslogserver/api/backend/logstash-2022.01.31/_count?pretty&token=${token}" -d '{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"query_string": {
"query": "cmpservice.log"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"@timestamp": {
"gte" : "now-5m/m",
"lt" : "now/m"
}
}
}
]
}
}
}
}
}'

echo

[pbroste]

Excellent, and thanks for posting details to help others.

Perry