Nagios Support Forum

Posted: **Mon Aug 24, 2015 6:51 am**

Hi,
After upgrading to Nagios Logserver 2015R2.2 i se a lot of this fault in /var/log/elasticsearch/...

It happens when i create a new dashboard, add a row and a panel of type histogram.
If a copy an already defined dashboard with an histogram on it and modify it, theres no faults.

It complains over (key) field [@timestamp] not found, but no faults on other panel types f.ex table with the same events.

[2015-08-24 13:39:55,685][DEBUG][action.search.type ] [2ec2a4b3-7137-48ce-bdba-ed07a1fd603f] [nagioslogserver_log][4], node[OwNqjZ2HQYy1YGzvlOJZsA], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@31c737a0] lastShard [true]
org.elasticsearch.search.SearchParseException: [nagioslogserver_log][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"0":{"date_histogram":{"field":"@timestamp","interval":"30s"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1440412750065,"to":1440416350065}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:735)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:560)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:532)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:294)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:172)
at org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:719)
... 9 more

thanks,
Peter Calum
TDC Denmark

Posted: **Mon Aug 24, 2015 9:49 am**

Interesting - I tested this on a R2.2 lab box, and I'm not experiencing the same symptoms. Is there anything special that you're doing to the histogram, or are you leaving it at the default settings?

It would be interesting to know the attributes of your '@timestamp' field for today:

Code: Select all

curl 'localhost:9200/logstash-2015.08.24/_mapping?pretty'

Note that you should replace '08.24' with the month/day you experienced the problem.

Posted: **Mon Aug 24, 2015 10:26 am**

Hi, here's the printout (problem was today)

Code: Select all

[root@khk26dsg4 ~]# curl 'localhost:9200/logstash-2015.08.24/_mapping?pretty'
{
  "logstash-2015.08.24" : {
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string",
              "fields" : {
                "raw" : {
                  "index" : "not_analyzed",
                  "ignore_above" : 256,
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ],
        "_all" : {
          "enabled" : true
        },
        "properties" : {
          "@version" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "geoip" : {
            "dynamic" : "true",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          }
        }
      },
      "syslog" : {
        "dynamic_templates" : [ {
          "string_fields" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "type" : "string",
              "fields" : {
                "raw" : {
                  "index" : "not_analyzed",
                  "ignore_above" : 256,
                  "type" : "string"
                }
              }
            },
            "match" : "*",
            "match_mapping_type" : "string"
          }
        } ],
        "_all" : {
          "enabled" : true
        },
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "@version" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "day" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "facility" : {
            "type" : "long"
          },
          "facility_label" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "geoip" : {
            "dynamic" : "true",
            "properties" : {
              "location" : {
                "type" : "geo_point"
              }
            }
          },
          "host" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "logsource" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "message" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "month" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "pid" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "priority" : {
            "type" : "long"
          },
          "program" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "severity" : {
            "type" : "long"
          },
          "severity_label" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "syslog_facility" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "syslog_facility_code" : {
            "type" : "long"
          },
          "syslog_pri" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "syslog_severity" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "syslog_severity_code" : {
            "type" : "long"
          },
          "tags" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "timestamp" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          },
          "type" : {
            "type" : "string",
            "norms" : {
              "enabled" : false
            },
            "fields" : {
              "raw" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    }
  }
}

Posted: **Mon Aug 24, 2015 11:52 am**

I don't see any problems so far. How are you going about creating this new dashboard - can you give me a couple of exact reproduction steps? I'd like to see if I can get this to reproduce in a test environment.

What's interesting to me is that your error is reporting that the 'nagioslogserver_log' is at fault - this log is used exclusively for audit reporting, so I'm not exactly sure how it's getting tied up in this procedure. Maybe you can shed some light on the problem?

Posted: **Mon Aug 24, 2015 12:57 pm**

Hi,

make a tail -f at the active logfile in /var/log/elasticsearch from ssh session

ex
tail -f 12094021-aef2-4684-90a0-86455fdb760f.log

Login to Nagios logserver

1- Select 'empty dashboard' under dashboards
2 - Set timefilter to 6 hours
3 - Add a row
4 - Add a panel to the row of type histogram
5 - Give panel a name and click save panel

Now you should get the errors in your ssh session from tail

Hope this is enough

thanks,
Peter

Posted: **Mon Aug 24, 2015 1:59 pm**

Peter,

Thank you for the details reproduction steps. This is certainly a bug, and the bug lies in the 'Empty Dashboard' - it's set to parse all indices (including the audit log), not just the daily indices that are normally parsed. To correct this, you can click 'Configure Dashboard' and change your Index settings as follows:

2015-08-24 13_56_12-Dashboard • Nagios Log Server - Firefox Developer Edition.png

This should correct the issue. We will have this fixed entirely in our next release - as far as I can tell, only 'Empty Dashboard' is affected. Thank you for your report!

Nagios Support Forum

Histogram errors after upgrading to 2015R2.2

Histogram errors after upgrading to 2015R2.2

Re: Histogram errors after upgrading to 2015R2.2

Re: Histogram errors after upgrading to 2015R2.2

Re: Histogram errors after upgrading to 2015R2.2

Re: Histogram errors after upgrading to 2015R2.2

Re: Histogram errors after upgrading to 2015R2.2