Are you able to run a tcpdump on the Nagios Log Server machine to verify traffic on port 2056 is being received from the host with the catalina logs? It might look something like this:
If there's absolutely nothing in the Logstash logs, and you can't see the events via the GUI, I question whether or not the traffic is making it to the Nagios Log Server machine. You might also check the Elasticsearch logs
(/var/log/elasticsearch/*.log) to make sure the data isn't having issues being inserted into the database.
# Input for CatalinaOut
$InputFileName /opt/tomcat/logs/catalina.out
$InputFileTag CatalinaOut:
$InputFileStateFile nls-state-opt_tomcat_logs_catalina.out # Must be unique for each file being polled
# Uncomment the folowing line to override the default severity for messages
# from this file.
#$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
# Forward to Nagios Log Server and then discard, otherwise these messages
# will end up in the syslog file (/var/log/messages) unless there are other
# overriding rules.
if $programname == 'CatalinaOut' then @@logs.isonasnet.com:2056
if $programname == 'CatalinaOut' then ~
And the last line of the rsyslog.conf does have:
$IncludeConfig /etc/rsyslog.d/*.conf
No firewall on the host blocking outbound and the NLS has the port open.
When you install tomcat and configure it according to the Apache docs part of that process is to give ownership of the tomcat directory to the tomcat user..... well that means that the rsyslog service can't get in there. chown the tomcat/logs dir recursively so it's owned by user syslog group tomcat and the entries started to flood right in.