Need help with mulitiline grok match

[lars.eik]

Fresh install of 2.1.4. We need to get list of clients that authenticat to AD without encryption before march patches from MS arrives. I have tried grokdebug heroku and constructor sites, and my patterns work there but not in NLS... The problem is the fields dont get created and we dont get grokparsefailure.

The windows eventlog in 'message' field is like this:

The following client performed a SASL (Negotiate/Kerberos/NTLM/Digest) LDAP bind without requesting signing (integrity verification), or performed a simple bind over a cleartext (non-SSL/TLS-encrypted) LDAP connection.

Client IP address:
10.6.78.9:57348
Identity the client attempted to authenticate as:
DOM\username
Binding Type:
1

Example pattern I have tried, but have tried so many now it getting hazy:

if [EventID] == "2889" {

grok {

match => { 'message' => '(?m)%{IPV4:clientip}:%{POSINT:clientport}.*Identity the client attempted to authenticate as:%{DATA:user}Binding%{GREEDYDATA:resten}'

}
}
}

[mcapra]

Unfortuantely Elastic doesn't produce docker images going back to 2.x so I can't quickly test against a 2.x instance, but the grok filter hasn't changed much over the years.

This "just works" for me on Logstash 7.5.2 (ignore the input/output sections):

Code: Select all

input {
  http {
      port => 25565
    }
}

filter {
  grok {
    match => { 'message' => '(?m)%{IPV4:clientip}:%{POSINT:clientport}.*Identity the client attempted to authenticate as:%{DATA:user}Binding%{GREEDYDATA:resten}' }
  }
}

output {
  stdout {
    codec => json
  }
}

I put your sample message in:

Code: Select all

curl -XPUT 'http://127.0.0.1:25565' -d 'The following client performed a SASL (Negotiate/Kerberos/NTLM/Digest) LDAP bind without requesting signing (integrity verification), or performed a simple bind over a cleartext on-SSL/TLS-encrypted) LDAP connection.

Client IP address:
10.6.78.9:57348
Identity the client attempted to authenticate as:
DOM\\username\
Binding Type:
1'

I get well-formatted fields out:

Code: Select all

{
  "headers": {
    "request_method": "PUT",
    "request_path": "/",
    "http_user_agent": "curl/7.58.0",
    "http_version": "HTTP/1.1",
    "content_type": "application/x-www-form-urlencoded",
    "http_accept": "*/*",
    "http_host": "127.0.0.1:25565",
    "content_length": "335"
  },
  "resten": " Type:\n1",
  "host": "172.17.0.1",
  "@version": "1",
  "clientport": "57348",
  "user": "\nDOM\\\\username\\\n",
  "@timestamp": "2020-02-05T13:39:33.692Z",
  "message": "The following client performed a SASL (Negotiate/Kerberos/NTLM/Digest) LDAP bind without requesting signing (integrity verification), or performed a simple bind over a cleartext (non-SSL/TLS-encrypted) LDAP connection.\n\nClient IP address:\n10.6.78.9:57348\nIdentity the client attempted to authenticate as:\nDOM\\\\username\\\nBinding Type:\n1",
  "clientip": "10.6.78.9"
}

Granted there's some sketchy new-line stuff in a few of the fields which is fixable in a variety of different ways. Are you sure there aren't other filter rules that may be picking your messages up?

[lars.eik]

Thanks for replying but I cannot reproduce your commands. I think the problem is with multiline because I do have other filters who splits up the message field just fine. I just recently installed this two node cluster so there are not many filters. I do have a strange issue with some input tcp/udp filter though, maybe there is something odd here? We used to have this setup earlier on centos 6.10 but this is now from scratch on centos7, filters used to work.
edit: found out about this 'cisco', there was another syslog entry with the same port.
If I enable this input filter:

Code: Select all

tcp {
          type => 'cisco'
          port => 5544
   }

udp {
         type => 'cisco'
         port => 5544
   }

logstash stop taking new logs...no more logs in the web gui. Config analyze etc is green and OK.

I cannot find anything about this in logstash.log but I assume I should be searching elsewhere, but where? Or is it possible to enable more logging/debug?

So how to troubleshoot with some reasonable linux skills?

[cdienger]

So you see these logs make it to the dashboard, but the additional fields are not created? Can you provide us with a screenshot of one of these expanded events so we can see all the fields that are getting parsed?

You can enable additional logging for logstash:

Edit /etc/init.d/logstash and change line 64 from:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

to:

Code: Select all

DAEMON_OPTS="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS} --debug"

and restart the service with:

Code: Select all

systemctl daemon-reload
service logstash restart

Let this run just long enough to allow NLS to accept some of these events and then collect the /var/log/logstash/logstash.log file before reverting the config back.

[lars.eik]

Solved , I'm sorry and a bit embarrassed but also glad I found the mistake. The EventID must not be quoted in any way, not single or double quotes, just the number. I guess I saw other examples where the digits were quoted and automatically used quoting for both fieldname and value. But I learned some more about logstash. Thanks for help.

[cdienger]

Glad to help! Coming up with the right filter isn't always obvious. I've definitely done my fair share of trial and error!