NLS remote syslog sending data works intermittently

[1aauto]

Hello,

I have Nagios Log Server 1.4.4 running. I'm sending data to it from 44 different Linux hosts via rsyslog via the main rsyslog.conf. That itself seems to work totally fine.

On a few additional particular servers, I'm sending a tomcat error log. The problem is, NLS seems to stop receiving the log at seemingly random times. For instance, if I look at a 24 hour dashboard view and filter out a particular host and program, I can see it stopped at 6:58pm EST. Restarting rsyslog on the server that's sending the data doesn't seem to help. Sometimes after a reboot of the NagiosLS server, it works fine.

I installed nagiosLS via the vmware 64 bit .ova package. I'm wondering if maybe something isn't tuned correctly in there. I don't yet know how to reproduce the issue, so I don't know exactly where the problem is yet.


Here is the rsyslog config for the file I'm sending from the client. This file was generated via the "Script Setup" at https://<server>/nagioslogserver/source-setup/linux-files:

[root@webserv2 ~]# cat /etc/rsyslog.d/90-nagioslogserver_opt_tomcat_logs_catalina.out.conf
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# Input for TOMCAT_CATALINA
$InputFileName /opt/tomcat/logs/catalina.out
$InputFileTag TOMCAT_CATALINA:
$InputFileStateFile nls-state-opt_tomcat_logs_catalina.out # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'TOMCAT_CATALINA' then @@nagiosls.example.com:5544
if $programname == 'TOMCAT_CATALINA' then ~
[root@webserv2 ~]#

I'm able to telnet to port 5544 on nagiosls.example.com from the client, so I know it's not a network problem. I'm thinking it could either be a problem with the main rsyslog.conf file on the client, or perhaps a config issue on the nagiosls server.

Any suggestions on troubleshooting this?


Thanks

[mcapra]

I would suggest sending your Tomcat logs to a port other than 5544. 5544 is by default an input dedicated to syslog data and has some default filters that get applied (via the logstash-input-syslog plugin) to the message once it's received.

A generic tcp/udp rule like so should be sufficient, using whatever port you like:

Code: Select all

tcp {
    type => 'tomcat'
    port => 5543
}
udp {
    type => 'tomcat'
    port => 5543
}

Be sure to apply configuration after making this input rule.

And then altering the rsyslog configuration for your Tomcat logs to ship to the configured port (5543 in this case). If all that doesn't fix anything, can you share the complete contents (all files) of your /var/log/logstash path? Feel free to PM it if you'd rather it not be in the public eye.