Unable to creat graph to show size of events coming in

[krobertson71]

I am trying to create a "size" of graph in byte to show how much is coming in, like in the demo Scott Wilkerson gave at NagCon2014.

When I try to change the panel to Chart Value = Total and Value Field = bytes I am getting the following error:

Code: Select all

 Oops! SearchParseException[[logstash-2015.07.23][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"bytes","interval":"10m"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"*"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1437671318008,"to":1437757718008}}}]}}}}}}}},"size":0}]]]

I have noticed the bytes field is not present when I look through all the possible fields?

Currently I am capturing logs from 60 Windows servers. Any ideas? Or, as in the demo, is this type of search only limited to Apache logs?

[jolson]

Is the number of bytes present anywhere in the logs at all? If so, please send me a few example logs and I'd be happy to write up a quick filter to get this working.

Best,


Jesse

[krobertson71]

These are Windows logs from 2008/2012 so do you want the original event from the host, or is the original event in Logstash somewhere? There is no Byte field in any of the Windows events when I select "All Fields" if that helps.

[jolson]

If you could screenshot a full event in Nagios Log Server, that would be the easiest way to approach this issue. For instance:

2015-07-27 10_56_33-Dashboard • Nagios Log Server - Firefox Developer Edition.png

[krobertson71]

Sorry for taking so long to respond. I had to focus on something in another area.

Here is a screen shot of a Windows event as requested.

nlseventpic.png

[jolson]

To generate a graph based on the size of the event coming in, we'll need to have the 'bytes' field available. Since Windows doesn't do this by default, it would require a lot of tinkering to get this working properly. Some Windows Servers may generate the 'bytes' or similar field - for instance I have seen IIS logs produce fields with this type of data in them.

What types of Windows Servers are these exactly? It may be possible to get this working if they're running IIS or any other application that reports logs with an appropriate field. We'll have to do some tinkering around to get it fully functional, but I see it as a good learning experience.

Some good things to note:
-If you're running IIS or similar on your Windows Servers, check the logs that it outputs by default for any relevant values - maybe bytes, transactions per second, response time, etc.
-Once we find that log file, we can begin exporting it to Nagios Log Server and parsing it however we'd like. Any value included in that log file can have graphs/charts/etc generated from it.

Let me know - thanks!

[krobertson71]

For Apache and other web server logs you do get size information. What we are trying to determine is how much data is "incoming" per hour.

Is there another way to achieve this? I mean I can look and try to average it out, but there are peaks at weird times of the 24 hour time period so makes doing this manually difficult.

[jolson]

The closest thing that I can think of is checking the total size of your index and dividing it by 24 - that would give you an average of the amount of data that ES is taking in per day.

2015-07-31 14_03_44-Index Status • Nagios Log Server - Firefox Developer Edition.png

Another option would be to use Nagios to monitor the disk usage of your Nagios Log Server nodes - that way you could see the disk usage climb over time and calculate *roughly* how much data is incoming per hour.

Do either of those solutions work for you? It is my understanding that there is not a built in way to monitor the amount of data coming in per hour.

[krobertson71]

Sorry for the very late response to this thread.

Yes those will work for now.

Might this be put in as a feature request? Would be nice to see some overall statistics.

[hsmith]

Sorry for the very late response to this thread.

Yes those will work for now.

Might this be put in as a feature request? Would be nice to see some overall statistics.

I'll take a look at existing feature requests, and see if there are any I can tack this on to, otherwise I can get one submitted for you!

Thank you for your feedback.