Page 1 of 3
Log data is incorrectly displayed on the dashboard
Posted: Sun Mar 06, 2016 4:58 pm
by gimeb
I have the following concern regarding displaying of the log data on NLS dashboard. I have correctly parsed the log data from IBM HTTP server (which is IBM version of the apache logs) using the grok filter. The grok debugger is showing that application (EMDCSPAI) is correctly separated from the rest of the message in the following sting 136.133.230.58 - - [04/Mar/2016:15:01:51 -0800] "GET /EMDCSPAI/emazda/dealersystem/parts/css/postion.css HTTP/1.1" 304 -
Below is the grok filter
if [type] == 'ibm-logs' {
grok {
match => [ 'message', '%{IP:client_ip_address} - - \[%{MONTHDAY:day}/%{MONTH:month}/%{YEAR:year}:%{TIME} %{NOTSPACE:class} "%{NOTSPACE:get} /%{WORD:app}/%{GREEDYDATA:info}" %{WORD:error_code}']
}
}
However, on the dashboard when selected to display the “app” field it displays the full message /EMDCSPAI/emazda/dealersystem/parts/css/postion.css
I have attached the detailed explanation and screen prints. Please advise.
Re: Log data is incorrectly displayed on the dashboard
Posted: Mon Mar 07, 2016 4:22 pm
by tmcdonald
If you expand out the entry for a log in the screenshot you sent, does it properly parse out the message? I believe that table will always show the full message, and then there are options to use grok to provide context/labels.
Re: Log data is incorrectly displayed on the dashboard
Posted: Mon Mar 07, 2016 4:23 pm
by jolson
Give this filter a try:
Code: Select all
%{IP:client_ip_address} - - \[%{MONTHDAY:day}/%{MONTH:month}/%{YEAR:year}:%{TIME} %{NOTSPACE:class} \"%{NOTSPACE:get} /%{WORD:app}/%{GREEDYDATA:info}\" %{WORD:error_code}
I am wondering if the " characters need escaping.
Re: Log data is incorrectly displayed on the dashboard
Posted: Mon Mar 07, 2016 5:11 pm
by gimeb
I have tried the filter you have provided. I have even enhanced it to display "misc" on the end
%{IP:client_ip_address} - - \[%{MONTHDAY:day}/%{MONTH:month}/%{YEAR:year}:%{TIME} %{NOTSPACE:class} \"%{NOTSPACE:get} /%{WORD:app}/%{GREEDYDATA:info}\" %{WORD:error_code} %{GREEDYDATA:misc}
The results are still the same - some URLs are parsed and displayed correctly. However, majority are not. The grok debugger is always shows correct parsing. I have attached another document with screen prints and URLs. Please let me know if you need additional information
Re: Log data is incorrectly displayed on the dashboard
Posted: Mon Mar 07, 2016 5:57 pm
by jolson
It is worth noting that only logs that enter Nagios Log Server after the filter has been applied will be impacted by the filter. Looking at the document you provided, the failing logs are from 03-04, are you certain that new logs are still being impacted by this problem? I ran a few test simulations and could not find a single problem with the filter you've defined - it should parse properly in all cases.
Re: Log data is incorrectly displayed on the dashboard
Posted: Mon Mar 07, 2016 8:56 pm
by gimeb
Thank for the clarification. I was not aware that the filter can only be applied to new logs and previous logs will still be incorrectly parsed. Please confirm. It looks like they are being parsed and displayed correctly now. Thank you again.
I have one question: what is the importance of the escaping and explain the reason it started to work when you updated the filter with " characters needed escaping. As I have indicated the grok debugger was parsing message correctly with and without the escape character and only when it went through the NLS filter it was not parsed as it should.
Re: Log data is incorrectly displayed on the dashboard
Posted: Tue Mar 08, 2016 2:33 pm
by tmcdonald
gimeb wrote:I was not aware that the filter can only be applied to new logs and previous logs will still be incorrectly parsed. Please confirm.
This is correct, the filters are not retroactive.
gimeb wrote:what is the importance of the escaping and explain the reason it started to work when you updated the filter with " characters needed escaping.
Escaping is used to tell the software that a cretain character should not be treated as special, and instead be treated as plain text. This is useful in programming, so that you can put a quotation mark inside of another set of quotation marks without causing the quote to end early:
Code: Select all
"Hello, world"
"Hello, "world""
"Hello, \"world\""
In the example above, the first line would render as
Hello, world, the second would render as
Hello, and probably cause an error in a program that tried to do so, while the third, because of the escaping, would render as
Hello, "world". Does this make more sense?
gimeb wrote:As I have indicated the grok debugger was parsing message correctly with and without the escape character and only when it went through the NLS filter it was not parsed as it should.
It is worth noting that the various online grok debuggers (such as
https://grokdebug.herokuapp.com/) may not be running the same filter plugin versions that Logserver uses, so the results may differ.
Re: Log data is incorrectly displayed on the dashboard
Posted: Tue Mar 08, 2016 4:18 pm
by jolson
I have one question: what is the importance of the escaping and explain the reason it started to work when you updated the filter with " characters needed escaping. As I have indicated the grok debugger was parsing message correctly with and without the escape character and only when it went through the NLS filter it was not parsed as it should.
The debugger may be using a different interpretor than Nagios Log Server, so the results are not always 100% accurate (but I'd wager that they're close to 95% accurate). Double-quotes need to be escaped, here is a list of characters that I believe need escaping for future reference:
Re: Log data is incorrectly displayed on the dashboard
Posted: Tue Mar 08, 2016 6:10 pm
by gimeb
Thank you for the reply and explanation. I have additional questions.
1. Please let me know if I can paste screen prints directly into support forum page or I still need to create a document to be attached
2. On my dashboard I have successfully separated the application name from the URL with your assistance (%{WORD:app}). The dashboard has row with multiple panels. I would like to display different application on each. Please let me know how to do it. Do I need to run a separate queries and if I do - then how to associate them with individual panel
Re: Log data is incorrectly displayed on the dashboard
Posted: Wed Mar 09, 2016 2:41 pm
by jolson
Please let me know if I can paste screen prints directly into support forum page or I still need to create a document to be attached
You can attach pictures via the 'Upload attachment' button - like so:
2016-03-09 12_48_56-Post a reply • Nagios Support Forum - Firefox Developer Edition.png
On my dashboard I have successfully separated the application name from the URL with your assistance (%{WORD:app}). The dashboard has row with multiple panels. I would like to display different application on each. Please let me know how to do it. Do I need to run a separate queries and if I do - then how to associate them with individual panel
What do you mean by a 'different application' on each panel? Typically applications are separated on a per-dashboard basis. Is there any chance that you could separate your applications by dashboard instead of by panel? Let me know if I'm misunderstanding anything.