How to process Websphere Application server/HTTP server logs

[gimeb]

I would like to implement the stacktrace parsing for Webpshere Application server logs since I am getting a lot of “at” in the view. See below. We have tried codec during my 1hr session with Jesse and it did not work. He recommended to post this on support site. The attached is detailed document with screen prints and stacktrace.

tcp {
port => 5556
type => 'websphere-app-logs'
codec => multiline {
pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
what => "previous"
}

[jolson]

To get to this point, there were several adjustments that needed to be made. First of all, the workers for logstash needed to be turned down to 1, as previously mentioned:

Code: Select all

# cat /etc/sysconfig/logstash | grep LS_OPTS
LS_OPTS=" -w 1"

Be sure to include the 'space' before the '-w', to be safe. I was able to restart logstash with no problems.

Code: Select all

service logstash restart

Next, I had to make adjustments to rsyslog on my client machine so that it read multiline files properly:

Code: Select all

cat /etc/rsyslog.d/90-nagioslogserver_root_test.txt.conf
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for tomcat
$InputFileName /root/test.txt
$InputFileTag tomcat:
$InputFileStateFile nls-state-root_test.txt # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFileReadMode 2
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'tomcat' then @@x.x.x.x:9002
if $programname == 'tomcat' then ~

Changing the ReadMode to '2' will enable multiple-line logging - rsyslog will assume that any line beginning with 'space' is part of the previous line. I restarted rsyslog here just to be safe.

Code: Select all

service rsyslog restart

Note: The above rsyslog config assumes that you are sending log data from a file.


This is what my input/filter chain looks like:

Code: Select all

#cat /usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf

tcp {
         port => 9002
         type => "tomcatlogs"
}


#cat /usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf

if [type] == "tomcatlogs" {
      multiline {
        pattern => "^\s+at .+"
        what => "previous"
   }
}

Also, here's some discussion about the subject: http://stackoverflow.com/questions/3352 ... -java-logs


Let me know if the above works out for you - my example uses tomcat logs, but you could easily make adjustments to the above for websphere logs. Let me know!

Jesse

[gimeb]

Jesse,
Thank you the reply.

Currently none of the logs including Websphere logs are getting into NLS from Linux box we have worked during our session. I have send you an e-mail about it with screen prints.

The Websphere Application server logs that I am referring are processed from Windows production servers. Therefore, I do not know what should be done to process them correctly. See the previously attached document "How to process Websphere Application server 2.docx".

I also read the link you have provided and I am not sure if I should replace the filter you have created during our session with the one from stackoverflow site below since I do not know what these parameters mean. Please let me know how to proceed

Code: Select all

input {
  file {    
    path => "D:\Log\Logstash\testlog.log"
    type => "LOG"
    start_position => "beginning"
    codec => plain { charset => "ISO-8859-1" }
  } 
}
filter {
    multiline {
        pattern => "^\A%{SYSLOG5424SD}"
        negate => true
        what => previous
    }
    # ... all other filters 
}

[jolson]

Please let me know how to proceed

If your Linux and Windows websphere filters are separate, this is what your input/filter combination should look like for websphere:

Code: Select all

tcp {
   port => some_port
   type => 'websphere-app-logs'
   codec => multiline {
   pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
   what => "previous"
}

if [type] == 'websphere-app-logs' {
  multiline {
      pattern => "^\s+at .+"
      what => previous
  }
  grok {
    match => [ 'message', '%{custom_filter_here}']
    }
}

Note that 'some_port' and 'custom_filter_here' will need to be replaced by your Windows Websphere logs and filters you have generated, respectively.

Remember that you'll need to make the worker change I mentioned above as well. Thanks!

[gimeb]

Jesse,
1. I have attached the document where I have included the current configuration and the changes I am planning to make for Inputs and Filters. Please review and let me know if these are ok to implement.

2. My current configuration for logstash is below. Please let me know the reason I will need to change it to LS_OPTS=" -w 1" and how would it impact my current logs processing since I am currently have 10 hosts sending data to NLS

[root@X4LOGW01 ~]# cat /etc/sysconfig/logstash | grep LS_OPTS
LS_OPTS=" -w 4"
[root@X4LOGW01 ~]#

3. The Linux websphere logs should be in the same format as on windows. I have executed the following command to send websphere logs. Do I still need to make adjustments to rsyslog and which ones (as you can see I do not have ReadMode

bash setup-linux.sh -s 136.133.236.12 -p 5556 -f "/wsapps/logs_musa/SystemErr.log /wsapps/logs_musa/SystemOut.log" -t websphere-logs

[root@x3musaa80 rsyslog.d]# ls
90-nagioslogserver_wsapps_logs_musa_SystemErr.log.conf
90-nagioslogserver_wsapps_logs_musa_SystemOut.log.conf
[root@x3musaa80 rsyslog.d]#

[root@x3musaa80 rsyslog.d]# cat 90-nagioslogserver_wsapps_logs_musa_SystemErr.log.conf
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for websphere-logs
$InputFileName /wsapps/logs_musa/SystemErr.log
$InputFileTag websphere-logs:
$InputFileStateFile nls-state-wsapps_logs_musa_SystemErr.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'websphere-logs' then @@136.133.236.12:5556
if $programname == 'websphere-logs' then ~
[root@x3musaa80 rsyslog.d]#

[hsmith]

Jesse is out of the office for the day. I'll send him this link, but I want to warn you there will be a delay in response(tomorrow, most likely.) I'll see if I can revisit this one later today, but as Jesse is the one that's been working closely with you on this issue, he'll most likely have to look at this.

[gimeb]

Please let me know if there any updates

[jolson]

1. I have attached the document where I have included the current configuration and the changes I am planning to make for Inputs and Filters. Please review and let me know if these are ok to implement.

I reviewed your attached document, and I've made an adjustment - please see the bottom of this post.

2. My current configuration for logstash is below. Please let me know the reason I will need to change it to LS_OPTS=" -w 1" and how would it impact my current logs processing since I am currently have 10 hosts sending data to NLS

The reason that you need to turn the worker number down to 1 is due to the fact that the 'multiline' filter does not work among multiple workers just yet.

3. The Linux websphere logs should be in the same format as on windows. I have executed the following command to send websphere logs. Do I still need to make adjustments to rsyslog and which ones (as you can see I do not have ReadMode

You will not need to make adjustments to rsyslog.


Current Inputs:

Code: Select all

tcp {
   port => 5556
   type => 'websphere-app-logs'
}

New Inputs:
None


Current Filter:

Code: Select all

if [type] == 'websphere-app-logs' {
    grok {
        match => [ 'message', '\[%{DATESTAMP} \w+\] %{NOTSPACE:status} %{NOTSPACE:status} %{GREEDYDATA:info}: %{GREEDYDATA:appname}' ]
        match => [ 'message', '\[%{DATESTAMP} \w+\] %{NOTSPACE:status} %{NOTSPACE:status} +\w \d+-\d+-\d+ \d+:\d+:\d+,\d+ \[%{WORD:appname}\] %{GREEDYDATA:info} \[%{WORD:error_code}\]%{GREEDYDATA:additional_info}' ]
  }
}

New Filter:

Code: Select all

if [type] == 'websphere-app-logs' {
  multiline {
      pattern => "^\s+at .+"
      what => previous
  }
  grok {
   match => [ 'message', '\[%{DATESTAMP} \w+\] %{NOTSPACE:status} %{NOTSPACE:status} %{GREEDYDATA:info}: %{GREEDYDATA:appname}' ]
        match => [ 'message', '\[%{DATESTAMP} \w+\] %{NOTSPACE:status} %{NOTSPACE:status} +\w \d+-\d+-\d+ \d+:\d+:\d+,\d+ \[%{WORD:appname}\] %{GREEDYDATA:info} \[%{WORD:error_code}\]%{GREEDYDATA:additional_info}' ]
  }
}

[gimeb]

I have implemented the filter. I have also placed the match ==> entries on the same column as it was in the current filter. The filter was implemented on 3:12PM and I do not see logs coming into the dashboard after 3:09PM. Do I have to wait?

2016-03-22T15:09:34.401-07:00 136.133.230.180 websphere-app-logs [3/22/16 15:09:33:523 PDT] 0000017a WSChannelFram CHFW0019I: The Transport Channel Service has started chain HttpOutboundChain:m1group101.mnao.net:8080.
2016-03-22T15:09:22.406-07:00 136.133.230.182 websphere-app-logs [3/22/16 15:09:22:026 PDT] 000001e3 SystemOut O 2016-03-22 15:09:22,026 [VehLocator] com.mazdausa.dealer.vehiclelocator.action.RetrieveDealerCommand [ERROR] - ******* Error in sortArrayListContainsLabelValueBeans ******* ...
2016-03-22T15:08:57.586-07:00 136.133.230.182 websphere-app-logs [3/22/16 15:08:57:018 PDT] 00000158 SystemOut O 2016-03-22 15:08:57,018 [VehLocator] com.mazdausa.dealer.vehiclelocator.action.RetrieveDealerCommand [ERROR] - ******* Error in sortArrayListContainsLabelValueBeans ******* ...
2016-03-22T15:08:03.389-07:00 136.133.230.180 websphere-app-logs [3/22/16 15:08:02:869 PDT] 0000014d WSChannelFram A CHFW0019I: The Transport Channel Service has started chain HttpOutboundChain:m1group101.mnao.net:8080

[gimeb]

I do see exceptions in the actual logs after 15:09PM and for some reason they are not getting displayed in the NLS dashboard

[3/22/16 15:19:00:909 PDT] 000051fd ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: An exception was thrown by one of the service methods of the servlet [/emazda/corporate/vehicles/fleet/rdr/eligiblevehicleslist.jsp] in application [FleetMassRDREAR]. Exception created : [java.lang.NumberFormatException: For input string: ""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:76)
at java.lang.Integer.parseInt(Integer.java:515)
at java.lang.Integer.parseInt(Integer.java:538)
at com.ibm._jsp._eligiblevehicleslist._jspService(_eligiblevehicleslist.java:442)
at com.ibm.ws.jsp.runtime.HttpJspBase.service(HttpJspBase.java:99)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.wsspi.webcontainer.servlet.GenericServletWrapper.handleRequest(GenericServletWrapper.java:122)
at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:219)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:1385)
at com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:194)
at com.mazdausa.common.adapter.actions.ActionMultiplexer.doPerform(ActionMultiplexer.java:75)
at com.mazdausa.common.application.actions.EMDCSInitializationServlet.forwardToBase(EMDCSInitializationServlet.java:153)
at com.mazdausa.common.application.actions.EMDCSInitializationServlet.performTask(EMDCSInitializationServlet.java:140)
at com.mazdausa.common.application.actions.EMDCSInitializationServlet.doPost(EMDCSInitializationServlet.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at com.mazdausa.corporate.vehicles.fleet.rdr.application.servlets.FleetRDRActionMultiplexer.service(FleetRDRActionMultiplexer.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3837)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:981)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:88)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1864)
]