Apache Filter stopped working when adding TLS

[tke395]

I set up a CentOS 7 server with apache to forward access log and error log files to Nagios Log Server using rsyslog. I used the scripts from NLS to configure rsyslog on the client. Everything worked as expected. Log entries are showing up as apache_access and apache_error.

When I set up TLS, I added another input for syslog over tcp:

Code: Select all

tcp {
    port => 8888
    type => 'syslog'
    ssl_extra_chain_certs => ['/etc/pki/tls/certs/ca.pem']
    ssl_cert => '/etc/pki/tls/certs/device-nls.crt'
    ssl_key =>  '/etc/pki/tls/private/device-nls.key'
    ssl_enable => true
    ssl_verify => false
}

I added the following to the client config:

Code: Select all

#tls setup
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/ca.pem
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name

Now, the log messages are still loaded into NLS, but they don't appear to be running through the Apache filter. They are all identified as "syslog" and the program value for apache_access or apache_error is not present.

As far as I can tell, I didn't do anything to change the format of the messages, only the transport. However, something must be different. Any ideas?

Thanks,
Eric

[cdienger]

We can enable debug logging on logstash to hopefully determine why it isn't triggering filters the way we'd expect. To do this edit /etc/init.d/logstash and change line 64 from:

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and then run:

systemctl daemon-reload
service logstash restart

/var/log/logstash/logstash.log will grow pretty quickly after this so only have it enabled just long enough to have some logs be sent over and then revert the change and restart again. Feel free to PM me the logstash.log if needed.

[tke395]

Thank you for your suggestion. I enabled debug logging and was able to verify that the filter was not triggered. However, I was not able to figure out why.

We stripped the TLS configuration out to eliminate that as a possible cause. At this point the only difference is that the working input uses syslog:

Code: Select all

syslog {
    type => 'syslog'
    port => 5544
}

The non-working input uses syslog over TCP:

Code: Select all

tcp {
    port => 7778
    type => 'syslog'
}

Is there any documentation on how the syslog message handling for these inputs differs? It appears that the "<%{POSINT:priority}>%{SYSLOGLINE}" pattern is not matched on the tcp syslog input. Then the [program] == 'apache_access' match fails on the Apache filter because program is empty.

[cdienger]

Ahh, somehow I missed the input type as being tcp. In this case then you'll want to account for the syslog formatting. The syslog input expects data following rfc3164: https://www.ietf.org/rfc/rfc3164.txt.

[tke395]

Thanks. We got it working by adding a filter. For some reason I assumed that adding type=>'syslog' to the tcp input would process the inputs as syslog messages.

[scottwilkerson]

Thanks. We got it working by adding a filter. For some reason I assumed that adding type=>'syslog' to the tcp input would process the inputs as syslog messages.

Glad to hear you got it working