How to process Websphere Application server/HTTP server logs

[gimeb]

I have an update.

I have reviewed the System Status and Logstash Collector was down. When I started it up and went down again. All dashboards are have created were showing no data!!! I had to revert the filter changes and everything went back to normal. Therefore, I do not know if the filter changes contributed to Logstash Collector crashing

[hsmith]

I've never seen a filter crash Logstash. Can you try to replicate it happening?

[gimeb]

I have not changed the workers by LS_OPTS=" -w 1". Could that be an issue?

You have also mentioned that on Linux you had make adjustments to rsyslog on so that it read multiline files properly ($InputFileReadMode 2). Does the same hold true for nxlog.conf on Windows and I need to update it with additional property.

[hsmith]

You changed the filter but didn't change the workers? That could be why it is happening. NXLog should handle multiline just fine as far as I understand it.

[gimeb]

Please let me if the changes to workers "LS_OPTS=" -w 1" might impact my current logs processing since I am currently have 12 hosts sending data to NLS. I have demonstration to our CEO and I do not want to jeopardize it

[jolson]

If you are attempting to use the multiline filter with more than one worker, Logstash will crash - that is the limitation of the multiline filter currently.

Please let me if the changes to workers "LS_OPTS=" -w 1" might impact my current logs processing since I am currently have 12 hosts sending data to NLS.

You should be fine to move down to a single worker, I don't expect that you'll notice any performance degradation by doing so. You are always capable of moving back to 4 workers if necessary.

[gimeb]

I have made changes to the workers and filter. However, I do not see exception lines in the dashboard any longer, i.e. all lines starting with "at" are not there only the first line is displayed. See the document attached.

[jolson]

Log on to your client machine and give the following rsyslog adjustment a try:

Code: Select all

vi /etc/rsyslog.d/90-nagioslogserver_root_test.conf

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for websphere-app-logs
$InputFileName /root/test.txt
$InputFileTag websphere-app-logs:
$InputFileStateFile nls-state-root_test # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFileReadMode 2
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'websphere-app-logs' then @@x.x.x.x:5555
if $programname == 'websphere-app-logs' then ~

Changing the ReadMode to '2' will enable multiple-line logging - rsyslog will assume that any line beginning with 'space' is part of the previous line. I restarted rsyslog here just to be safe.

Code: Select all

service rsyslog restart

Perhaps if we get this handled on the remote machine we won't have to use the heavy multiline filter moving forward. Can you let me know if this works for you? You can try disabling the 'multiline' filter we have in place currently to see whether or not it makes a difference. Thanks!

[gimeb]

The remote machine is Windows and not Linux. I have followed your suggestions in the post dated "Tue Mar 22, 2016 1:51 pm".

[gimeb]

Correction.

The client machine is on Windows and not on Linux. I have followed your recommendation in the post:

Tue Mar 22, 2016 10:51 am

1. I have attached the document where I have included the current configuration and the changes I am planning to make for Inputs and Filters. Please review and let me know if these are ok to implement.

I reviewed your attached document, and I've made an adjustment - please see the bottom of this post