TLS syslog messages not seen on Dashboard

[acwong]

I've recently configured TLS between the nagios log server and a client linux box. TCP syslog messages sent from client are seen on the nagios dashboard. However, TLS syslog messages are not being received on the dashboard of nagios log server. I've used tcpdump and confirmed that TLS traffic is being received on the nagios log server, but not seen on the dashboard. Where should I start troubleshooting?

[cdienger]

I'd enable logstash debugging to troubleshoot this.

Edit /etc/init.d/logstash and change line 64 from:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and restart the service with:

Code: Select all

systemctl daemon-reload
service logstash restart

Let this run just long enough to allow NLS to process some TLS events from the host and then collect the /var/log/logstash/logstash.log file before reverting the config back.

[acwong]

Looks like a CA issue. I've configured nagios syslog server as the CA. I followed the "Sending syslog With SSL/TLS" nagios documentation.
https://assets.nagios.com/downloads/nag ... th-SSL.pdf

Code: Select all

{:timestamp=>"2020-05-21T13:27:50.820000-0700", :message=>"SSL Error", :exception=>#<OpenSSL::SSL::SSLError: Received fatal alert: unknown_ca>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:272:in `accept'", "/usr/local/nagioslogserver/logstash/vendor/jruby/lib/ruby/shared/jopenssl19/openssl/ssl-internal.rb:106:in `accept'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:112:in `run_server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:84:in `run'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:342:in `inputworker'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:336:in `start_input'"], :level=>:debug, :file=>"logstash/inputs/tcp.rb", :line=>"117", :method=>"run_server"}

[cdienger]

Double check the configuration on the sending device to make sure it is pointing to the correct syslog-ca.pem and also that the syslog-ca.pem on the sending device matches the one in /etc/pki/tls/certs/syslog-ca.pem on the NLS machine.

[acwong]

After double checking the certs, there was a mismatch on a cert credentials which I fixed. Now I I have another issue.

Code: Select all

{:timestamp=>"2020-05-27T17:13:04.723000-0700", :message=>"SSL Error", :exception=>#<OpenSSL::SSL::SSLError: Received fatal alert: decrypt_error>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:272:in `accept'", "/usr/local/nagioslogserver/logstash/vendor/jruby/lib/ruby/shared/jopenssl19/openssl/ssl-internal.rb:106:in `accept'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:112:in `run_server'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.6/lib/logstash/inputs/tcp.rb:84:in `run'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:342:in `inputworker'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:336:in `start_input'"], :level=>:debug, :file=>"logstash/inputs/tcp.rb", :line=>"117", :method=>"run_server"}

[cdienger]

Please attach a copy of the syslog config as well as the NLS input's config.

I'd also like to see a tcpdump taken on the NLS machine:

Code: Select all

yum -y install tcpdump
tcpdump -s 0 -i any host w.x.y.z -w output.pcap

where w.x.y.z is the IP of the sending machine. Let this run just long enough to reproduce the 'SSL Error' message that you see in the logs then use CTRL+C to stop it. Please PM me the output.pcap that this creates.