Hi,
After upgrading to Nagios Logserver 2015R2.2 i se a lot of this fault in /var/log/elasticsearch/...
It happens when i create a new dashboard, add a row and a panel of type histogram.
If a copy an already defined dashboard with an histogram on it and modify it, theres no faults.
It complains over (key) field [@timestamp] not found, but no faults on other panel types f.ex table with the same events.
[2015-08-24 13:39:55,685][DEBUG][action.search.type ] [2ec2a4b3-7137-48ce-bdba-ed07a1fd603f] [nagioslogserver_log][4], node[OwNqjZ2HQYy1YGzvlOJZsA], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@31c737a0] lastShard [true]
org.elasticsearch.search.SearchParseException: [nagioslogserver_log][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"0":{"date_histogram":{"field":"@timestamp","interval":"30s"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1440412750065,"to":1440416350065}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:735)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:560)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:532)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:294)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:172)
at org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:719)
... 9 more
thanks,
Peter Calum
TDC Denmark
Histogram errors after upgrading to 2015R2.2
Histogram errors after upgrading to 2015R2.2
Peter Calum
Re: Histogram errors after upgrading to 2015R2.2
Interesting - I tested this on a R2.2 lab box, and I'm not experiencing the same symptoms. Is there anything special that you're doing to the histogram, or are you leaving it at the default settings?
It would be interesting to know the attributes of your '@timestamp' field for today:
Note that you should replace '08.24' with the month/day you experienced the problem.
It would be interesting to know the attributes of your '@timestamp' field for today:
Code: Select all
curl 'localhost:9200/logstash-2015.08.24/_mapping?pretty'Re: Histogram errors after upgrading to 2015R2.2
Hi, here's the printout (problem was today)
Code: Select all
[root@khk26dsg4 ~]# curl 'localhost:9200/logstash-2015.08.24/_mapping?pretty'
{
"logstash-2015.08.24" : {
"mappings" : {
"_default_" : {
"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"ignore_above" : 256,
"type" : "string"
}
}
},
"match" : "*",
"match_mapping_type" : "string"
}
} ],
"_all" : {
"enabled" : true
},
"properties" : {
"@version" : {
"type" : "string",
"index" : "not_analyzed"
},
"geoip" : {
"dynamic" : "true",
"properties" : {
"location" : {
"type" : "geo_point"
}
}
}
}
},
"syslog" : {
"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"ignore_above" : 256,
"type" : "string"
}
}
},
"match" : "*",
"match_mapping_type" : "string"
}
} ],
"_all" : {
"enabled" : true
},
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"@version" : {
"type" : "string",
"index" : "not_analyzed"
},
"day" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"facility" : {
"type" : "long"
},
"facility_label" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"geoip" : {
"dynamic" : "true",
"properties" : {
"location" : {
"type" : "geo_point"
}
}
},
"host" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"logsource" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"month" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"pid" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"priority" : {
"type" : "long"
},
"program" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"severity" : {
"type" : "long"
},
"severity_label" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"syslog_facility" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"syslog_facility_code" : {
"type" : "long"
},
"syslog_pri" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"syslog_severity" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"syslog_severity_code" : {
"type" : "long"
},
"tags" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"timestamp" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "string",
"norms" : {
"enabled" : false
},
"fields" : {
"raw" : {
"type" : "string",
"index" : "not_analyzed",
"ignore_above" : 256
}
}
}
}
}
}
}
}Peter Calum
Re: Histogram errors after upgrading to 2015R2.2
I don't see any problems so far. How are you going about creating this new dashboard - can you give me a couple of exact reproduction steps? I'd like to see if I can get this to reproduce in a test environment.
What's interesting to me is that your error is reporting that the 'nagioslogserver_log' is at fault - this log is used exclusively for audit reporting, so I'm not exactly sure how it's getting tied up in this procedure. Maybe you can shed some light on the problem?
What's interesting to me is that your error is reporting that the 'nagioslogserver_log' is at fault - this log is used exclusively for audit reporting, so I'm not exactly sure how it's getting tied up in this procedure. Maybe you can shed some light on the problem?
Re: Histogram errors after upgrading to 2015R2.2
Hi,
make a tail -f at the active logfile in /var/log/elasticsearch from ssh session
ex
tail -f 12094021-aef2-4684-90a0-86455fdb760f.log
Login to Nagios logserver
1- Select 'empty dashboard' under dashboards
2 - Set timefilter to 6 hours
3 - Add a row
4 - Add a panel to the row of type histogram
5 - Give panel a name and click save panel
Now you should get the errors in your ssh session from tail
Hope this is enough
thanks,
Peter
make a tail -f at the active logfile in /var/log/elasticsearch from ssh session
ex
tail -f 12094021-aef2-4684-90a0-86455fdb760f.log
Login to Nagios logserver
1- Select 'empty dashboard' under dashboards
2 - Set timefilter to 6 hours
3 - Add a row
4 - Add a panel to the row of type histogram
5 - Give panel a name and click save panel
Now you should get the errors in your ssh session from tail
Hope this is enough
thanks,
Peter
Peter Calum
Re: Histogram errors after upgrading to 2015R2.2
Peter,
Thank you for the details reproduction steps. This is certainly a bug, and the bug lies in the 'Empty Dashboard' - it's set to parse all indices (including the audit log), not just the daily indices that are normally parsed. To correct this, you can click 'Configure Dashboard' and change your Index settings as follows: This should correct the issue. We will have this fixed entirely in our next release - as far as I can tell, only 'Empty Dashboard' is affected. Thank you for your report!
Thank you for the details reproduction steps. This is certainly a bug, and the bug lies in the 'Empty Dashboard' - it's set to parse all indices (including the audit log), not just the daily indices that are normally parsed. To correct this, you can click 'Configure Dashboard' and change your Index settings as follows: This should correct the issue. We will have this fixed entirely in our next release - as far as I can tell, only 'Empty Dashboard' is affected. Thank you for your report!
You do not have the required permissions to view the files attached to this post.