Logserver creating multiple sessions via FW to AD

[james.liew]

Hi all,

NLS is creating multiple sessions to our LDAP/AD server.

I have a firewall in-between that's throwing alerts due to the amount of sessions being opened. How do I stop this from happening?

I can provide logs etc.

Screenshot attached.

L_7372.tmp.PNG

[avandemore]

If the LS OS is Cent/RHEL 7, what is the output of:

Code: Select all

# ss -tuna

If the LS OS is Cent/RHEL 6, what is the output of:

Code: Select all

netstat -tuna

[james.liew]

I filtered the results... see below.

[root@hs1-log-01 ~]# ss -tuna | grep 172.16.17.1

tcp ESTAB 0 0 ::ffff:172.16.21.37:3515 ::ffff:172.16.17.1:64854

[scottwilkerson]

Is your AD server also your DNS server? Looks like the Application listed in your report in the OP is DNS.

[james.liew]

Is your AD server also your DNS server? Looks like the Application listed in your report in the OP is DNS.

it is, yes.

Is this normal behaviour?

[mcapra]

The version of Logstash we are currently using in Nagios Log Server tends to keep connections open until they're very stale, so that sort of behavior isn't terrible surprising to me.

Can you share the Logstash configurations you're using? This should provide them:

Code: Select all

grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*

[james.liew]

Amended some ports to hide them. Sorry 'bout that.

Code: Select all

/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Logstash Configuration File
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Dynamically created by Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Created Tue, 03 Jan 2017 13:16:12 +0100
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Global inputs
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:input {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    syslog {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        type => 'syslog'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        port => *
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    syslog {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        type => 'switches'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        port => *
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    tcp {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        type => 'eventlog'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        port => *
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        codec => json {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:            charset => 'CP1252'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    tcp {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        type => 'import_raw'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        tags => 'import_raw'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        port => *
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    tcp {
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        type => 'import_json'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        tags => 'import_json'
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        port => *
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:        codec => json
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:}
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:# Local inputs
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/000_inputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Logstash Configuration File
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Dynamically created by Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Created Tue, 03 Jan 2017 13:16:12 +0100
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Global filters
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:filter {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:    if [program] == 'apache_access' {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        grok {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:            match => [ 'message', '%{COMBINEDAPACHELOG}']
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        date {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        mutate {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:            replace => [ 'type', 'apache_access' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:             convert => [ 'bytes', 'integer' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:             convert => [ 'response', 'integer' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:    if [program] == 'apache_error' {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        grok {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:            match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        mutate {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:            replace => [ 'type', 'apache_error' ]
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:    dns {
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        reverse => ['host']
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        action => ['replace']
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:        add_tag => ['dns']
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:}
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:# Local filters
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/500_filters.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Logstash Configuration File
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Dynamically created by Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Created Tue, 03 Jan 2017 13:16:12 +0100
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Required output for Nagios Log Server
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:output {
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:    elasticsearch {
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:        cluster => '1d9ff623-2ffd-4407-b023-ef71f56139d2'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:        host => 'localhost'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:        document_type => '%{type}'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:        node_name => '791cc6c8-f646-495e-9e58-1ec21a24b61c'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:        protocol => 'transport'
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:        workers => 4
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:    }
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:}
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Global outputs
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:# Local outputs
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:#
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:
/usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf:

[mcapra]

I don't have the equipment to properly test/reproduce this, but i'm betting the dns filter is what's causing those sessions :

Code: Select all

dns {
    reverse => ['host']
    action => ['replace']
    add_tag => ['dns']
}

Basically, every time Logstash receives an event, it's going to run it through that filter which is going to attempt to query the DNS server for the reverse lookup. For every event, a brand-new reverse DNS request.

[james.liew]

Is it possible to do without this?

Thanks!

[tacolover101]

well your filter kind of depends on DNS at that point so probably a better question to ask yourself - can you deal without translation?

you might be able to add ALL static entries to /etc/hosts