How to build query to show values over time

[gsl_ops_practice]

Hello,

We are successfully able to use the GUI interface to change the graph to show us values of a specific field over time, as per the screenshot. This is a big help in problem analysis, I can build custom dashboards and parse data quickly for patterns. In this case, the application logs at specific time intervals the number of CLI requests over the same interval. I would like to be notified if this happens in NagiosXI, if the number of CLI requests over this time period exceeded a pre-defined threshold.

There is a difference between a dashboard and a query. I am not yet able to have a query to get the same information, that NagiosXI can pull via the API.

The only thing I am able to do via NagiosXI is run queries that count the number of events over time with specified parameters, not the specific values recorded for each event over time.

Thank you,
Alex

[mcapra]

I don't think this is do-able with the way alerts currently work, but there might be a way around it in the query itself. Can you post the dashboard and query you're currently using so I can take a look? You can get them like so:

2017_02_01_15_00_37_Dashboard_Nagios_Log_Server.png

2017_02_01_14_59_59_Dashboard_Nagios_Log_Server.png

[gsl_ops_practice]

I am pasting the simplest possible view:

Code: Select all

{
  "title": "My Default Dashboard",
  "edit_type": "",
  "services": {
    "query": {
      "list": {
        "0": {
          "query": "*",
          "alias": "",
          "color": "#4D89F9",
          "id": 0,
          "pin": false,
          "type": "lucene",
          "enable": true
        }
      },
      "ids": [
        0
      ]
    },
    "filter": {
      "list": {
        "0": {
          "type": "time",
          "field": "@timestamp",
          "from": "now-24h",
          "to": "now",
          "mandate": "must",
          "active": true,
          "alias": "",
          "id": 0
        }
      },
      "ids": [
        0
      ]
    }
  },
  "rows": [
    {
      "title": "Graph",
      "height": "300px",
      "editable": true,
      "collapse": false,
      "collapsable": true,
      "panels": [
        {
          "span": 12,
          "editable": true,
          "group": [
            "default"
          ],
          "type": "histogram",
          "mode": "max",
          "time_field": "@timestamp",
          "value_field": "CLIRequests",
          "auto_int": true,
          "resolution": 100,
          "interval": "10m",
          "fill": 3,
          "linewidth": 3,
          "timezone": "browser",
          "spyable": true,
          "zoomlinks": true,
          "bars": true,
          "stack": true,
          "points": false,
          "lines": false,
          "legend": true,
          "x-axis": true,
          "y-axis": true,
          "percentage": false,
          "interactive": true,
          "queries": {
            "mode": "all",
            "ids": [
              0
            ]
          },
          "title": "Events over time",
          "intervals": [
            "auto",
            "1s",
            "1m",
            "5m",
            "10m",
            "30m",
            "1h",
            "3h",
            "12h",
            "1d",
            "1w",
            "1M",
            "1y"
          ],
          "options": true,
          "tooltip": {
            "value_type": "cumulative",
            "query_as_alias": true
          },
          "scale": 1,
          "y_format": "none",
          "grid": {
            "max": null,
            "min": 0
          },
          "annotate": {
            "enable": false,
            "query": "*",
            "size": 20,
            "field": "_type",
            "sort": [
              "_score",
              "desc"
            ]
          },
          "pointradius": 5,
          "show_query": true,
          "legend_counts": true,
          "zerofill": true,
          "derivative": false
        }
      ],
      "notice": false
    },
    {
      "title": "Events",
      "height": "350px",
      "editable": true,
      "collapse": false,
      "collapsable": true,
      "panels": [
        {
          "title": "All events",
          "error": false,
          "span": 12,
          "editable": true,
          "group": [
            "default"
          ],
          "type": "table",
          "size": 50,
          "pages": 5,
          "offset": 0,
          "sort": [
            "@timestamp",
            "desc"
          ],
          "style": {
            "font-size": "9pt"
          },
          "overflow": "min-height",
          "fields": [
            "@timestamp",
            "host",
            "TotalRequests"
          ],
          "localTime": true,
          "timeField": "@timestamp",
          "highlight": [
            "*"
          ],
          "sortable": true,
          "header": true,
          "paging": true,
          "spyable": true,
          "queries": {
            "mode": "all",
            "ids": [
              0
            ]
          },
          "field_list": true,
          "status": "Stable",
          "trimFactor": 900,
          "normTimes": true,
          "all_fields": false
        }
      ],
      "notice": false
    }
  ],
  "editable": true,
  "failover": false,
  "index": {
    "interval": "day",
    "pattern": "[logstash-]YYYY.MM.DD",
    "default": "NO_TIME_FILTER_OR_INDEX_PATTERN_NOT_MATCHED",
    "warm_fields": true
  },
  "style": "light",
  "panel_hints": true,
  "pulldowns": [
    {
      "type": "query",
      "collapse": false,
      "notice": false,
      "query": "*",
      "pinned": true,
      "history": [],
      "remember": 10,
      "enable": true
    },
    {
      "type": "filtering",
      "collapse": false,
      "notice": true,
      "enable": true
    }
  ],
  "nav": [
    {
      "type": "timepicker",
      "collapse": false,
      "notice": false,
      "status": "Stable",
      "time_options": [
        "5m",
        "15m",
        "1h",
        "2h",
        "6h",
        "12h",
        "24h",
        "2d",
        "7d",
        "30d"
      ],
      "refresh_intervals": [
        "5s",
        "10s",
        "30s",
        "1m",
        "5m",
        "15m",
        "30m",
        "1h",
        "2h",
        "1d"
      ],
      "timefield": "@timestamp",
      "now": true,
      "filter_id": 0,
      "enable": true
    }
  ],
  "loader": {
    "save_gist": false,
    "save_elasticsearch": true,
    "save_local": true,
    "save_default": true,
    "save_temp": true,
    "save_temp_ttl_enable": true,
    "save_temp_ttl": "30d",
    "load_gist": true,
    "load_elasticsearch": true,
    "load_elasticsearch_size": 20,
    "load_local": true,
    "hide": false
  },
  "refresh": false,
  "dash_type": "elasticsearch"
}

[mcapra]

Could I also see the query you're currently using for this dashboard? One of the images in my previous post shows how to download a query.

[gsl_ops_practice]

Interesting...if I save the query and load it, I don't get the same results, the query doesn't save what I am trying to do, the only way that I found so far is to save the entire dashboard.

Here is the query that I saved, as you can see it doesn't have any mention of CLIRequests:

Code: Select all

{"name":"clirequests_test_query","raw":"{\"query\":{\"filtered\":{\"query\":{\"bool\":{\"should\":[{\"query_string\":{\"query\":\"*\"}}]}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from\":1485979163014,\"to\":1486065563015}}}]}}}}}","services":"{\"query\":{\"list\":{\"0\":{\"query\":\"*\",\"alias\":\"\",\"color\":\"#4D89F9\",\"id\":0,\"pin\":false,\"type\":\"lucene\",\"enable\":true}},\"ids\":[0]},\"filter\":{\"list\":{\"0\":{\"type\":\"time\",\"field\":\"@timestamp\",\"from\":\"now-24h\",\"to\":\"now\",\"mandate\":\"must\",\"active\":true,\"alias\":\"\",\"id\":0}},\"ids\":[0]}}","created_by":"nagiosadmin","created_id":"1","show_everyone":0,"imported":0}

I am also attaching the query that appears when I click on the "Inspect" button on the top right of the histogram.

Code: Select all

curl -XGET 'http://maskedipaddress/nagioslogserver/api/backend/logstash-2017.02.02,logstash-2017.02.01/_search?pretty&token=NNNNNNNNNNNNNNNNNNNNNNN' -d '{
  "facets": {
    "0": {
      "date_histogram": {
        "key_field": "@timestamp",
        "value_field": "CLIRequests",
        "interval": "10m"
      },
      "global": true,
      "facet_filter": {
        "fquery": {
          "query": {
            "filtered": {
              "query": {
                "query_string": {
                  "query": "*"
                }
              },
              "filter": {
                "bool": {
                  "must": [
                    {
                      "range": {
                        "@timestamp": {
                          "from": 1485979333943,
                          "to": 1486065733943
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        }
      }
    }
  },
  "size": 0
}'

[mcapra]

Currently, there isn't a way to alert on queries using criteria beyond "how many documents did this query return". Facets, which is what your query is currently using to draw the lovely graph, return multiple result sets based on the provided criteria (in your case, every 10 minutes for the provided time period). Our alerts aren't able to examine multiple result sets right now.

My assumption is you're looking to alert on sudden spikes or prolonged peaks? I can certainly see the use in having that sort of alert. I'll see about filing a feature request for something like that.

[gsl_ops_practice]

Thanks for trying to figure this out. I am hoping this feature will make it into a release in the not-too-distant future. Please consider this request resolved.