Multi-line log entries to show up as one line in NagiosLog?

[gsl_ops_practice]

Hello,

We are adding some application log files to NagiosLog that are of the following format:

Code: Select all

INFO  [2017-05-04 14:30:25,044][ListenerContainer-10][ID:NNNNN-NNNNN-NNNNNN-NNNNN] - Forward.COMPONENT- Message sent to [QUEUE_NAME] Variant [RAW_TEXT] Message body [<PreScanResponse>
    <PreScanID>111111</PreScanID>
    <CreatedDateTime>2017-05-04T14:30:25</CreatedDateTime>
    <StatusCode>YYY</StatusCode>
</PreScanResponse>] Size [178] Send Time [12]

Each of these lines show up as a separate entry right now, but I would like to see just one entry, INFO etc etc that contains all 5 lines in one, so when I search for the prescan ID, I will see all 5 lines and see all the other related lines at the same time

Is there a way to do this? We would want to do the same for Java stack traces as well, as right now it pollutes our logs with excessive lines.

Thank you,
Alex

[mcapra]

There's quite a few different options each with their own benefits depending on the specific logs. Logstash has a native multiline codec which I typically recommend for environments using rsyslog. nxlog has it's own multiline plugin which works better (in my opinion) than the one Logstash has, but running nxlog on a Linux system is a pain in the butt.

Here's our docs for how to do it in nxlog:
https://support.nagios.com/kb/article.php?id=546

We don't have a proper document for rsyslog (yet), but I did a write-up recently for another customer on this topic. I'll attach it below (compressed zip with a pdf).

[gsl_ops_practice]

Thank you, I will try to follow the same for logstash and let you know if it works or not.

Alex

[gsl_ops_practice]

Based on what I am reading in the PDF, I tried to create the following input filter and modify the rsyslog config, but I am no longer seeing any events at all coming into NagiosLog at all. Can you please advise what I'm doing incorrectly?

Code: Select all

tcp {
    codec => multiline
    { pattern => 'INFO [0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate => true
    what => previous
}
    type => 'log_forwarder'
    port => 5545
}

Code: Select all

 cat 90-nagioslogserver_log_forwarder1.conf
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/lib/rsyslog

# Input for log_forwarder1
$InputFileName /mnt/host005p/logs/Forwarder.log
$InputFileTag log_forwarder1:
$InputFileStateFile nls-state-mnt_host005p_logs_Forwarder.log # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
$template clean,"%rawmsg%"

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'log_forwarder1' then @@10.10.10.10:5545;clean
if $programname == 'log_forwarder1' then ~

[mcapra]

If iptables or firewalld is running on this machine, did you create a rule to allow traffic on port 5545?

Is there anything in the Logstash log indicating parse failures? Was the configuration applied after the rule was created?

[dwhitfield]

Did @mcapra's post help you? Please let us know if you have any additional questions.