Nagios Support Forum

Hi, I have a problem that has been talked about a lot with the ELK stack but I can't find any guidance here on it. I have used kv in my filters to split out an unknown number of key/value pairs from my logs (from the extension in CEF messages) and when I look at the raw output I can see what should be integers returned surrounded by quotes.

Now, I can fix this with specific grok filters but CEF messages are all different and neither the actual fields or their order are explicitly controlled. Thus, kv offers me the only way to combat this.

Other ELK forums have said to enable dynamic mapping within ES so it detects the type when first stored. This would solve my issue but of course the issue with ES is changing types later - if it detects an integer first then a character in that field in later messages will break it.

I believe kibana, being json based, doesn't care if numbers are quoted and will graph anything you tell it to. I suspect NLS UI would do the same, I think that the issue is the type detection only allowing me to choose bar, table or pie for a text field.

Any advice gladly received.

Thanks, --Chris

(If Nagios Support are reading this, its probably worth mentioning that I am doing a pre-sale eval on this at the moment vs Splunk and that this ability to graph CEF values is a must have for me)

zuzzy wrote:I think that the issue is the type detection only allowing me to choose bar, table or pie for a text field.

Can you elaborate on this? What other options would you be expecting?

Well, AFAIK I'm not allowed to make changes in ES to enable dynamic mapping, so thats one option (and has been suggested as a fix in several places)

Another option is that better data typing is implemented in LogStash, or in the Kibana UI. I know NLS is just LogStash and Kibana with nice UI and integrations, but if I wanted just LogStash and Kibana I'd have gone for ELK already (and not thought about paying) - I was hoping NLS would become a conduit to address some of the annoyances from ELK(like no role based access to data, another pet annoyance). Otherwise, whats the benefit of NLS over ELK?

Dynamic mapping is on by default in Log Server, however, once a field is set to a certain type, it cannot be changed in that index.

If you know a certain field will be an certain type it is best to do a mutate on the field in the logstash filter to force the type.

While JSON isn't typed, the JSON is only used to display the data the is retrieved from elasticsearch which in fact is typed, even with dynamic mappings. The type is just guessed the first time the field is used in the index.

zuzzy wrote:I was hoping NLS would become a conduit to address some of the annoyances from ELK(like no role based access to data, another pet annoyance).

This is on the roadmap, and will be coming in the future.

zuzzy wrote:Otherwise, whats the benefit of NLS over ELK?

Authentication
Alerting (Email, Nagios, SNMP Traps, custom scripts)
Saving of dashboards and queries per user and globally
Saving of queries
Cluster wide logstash configuration (Global and per-instance)
logstash configuration snapshots
Cluster management
Automated Backups
Easy log source setup scripts
Quick actions to search items on Google, Bing, Stack Overflow
Support

Fair point - that was a bit of a 'what have the Romans ever done for us' moment.