Page 1 of 1

Nagios Log Server - useragent logstash plugin

Posted: Wed Aug 30, 2017 12:14 pm
by bpizzutiWHI
I'm attempting to use this logstash plugin, I'm assuming it's enabled. I've got my Apache logs coming in in varios ways, my IIS logs are coming in via EventLogging. I've got the following filters set up but I don't see any fields or data being generated by the useragent plugin. Any ideas on how to get this working?

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 30 Aug 2017 13:12:03 -0400
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMMONAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
    
    if [program] == 'apache_custom' {
        grok {
            match => [ 'message', '%{CUSTOMAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_custom' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
    if [program] == 'apache_combined' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_combined' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '%{HTTPD20_ERRORLOG}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [SourceName] == 'Microsoft-Windows-IIS-Logging' {
        useragent {
            source => 'csUser-Agent'
        }
    }
    if [program] == 'apache_access' {
        useragent {
            source => 'agent'
        }
    }
    if [program] == 'apache_combined' {
        useragent {
            source => 'agent'
        }
    }
    if [program] == 'apache_custom' {
        useragent {
            source => 'agent'
        }
    }
    if [program] == 'apache_error' {
        useragent {
            source => 'agent'
        }
    }
}

#
# Local filters
#



Re: Nagios Log Server - useragent logstash plugin

Posted: Wed Aug 30, 2017 2:53 pm
by dwhitfield
Can you attach all the files located in the following paths:

Code: Select all

    /var/log/logstash
    /var/log/elasticsearch
If you could compress them into a zip or tarball, that would be ideal.

I will not be in the next couple of days, so if you do not want them on the forum you can PM them to @cdienger.

Re: Nagios Log Server - useragent logstash plugin

Posted: Wed Aug 30, 2017 3:17 pm
by bpizzutiWHI
I don't mind, here you go

Re: Nagios Log Server - useragent logstash plugin

Posted: Thu Aug 31, 2017 12:39 pm
by cdienger
I see a few messages indicating it's having a problem parsing them:

message=>"IP Field contained invalid IP address or hostname"

and later...

"clientip"=>"www.nexpartqa.com"

Looking at the logged line:

216.113.160.81 http://www.nexpartqa.com - - [30/Aug/2017:16:13:29 -0400] \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36\"

There seems to be a few odd things. First the domain after the IP second the "\"s to escape the quotes around the user-agent.

Re: Nagios Log Server - useragent logstash plugin

Posted: Thu Aug 31, 2017 1:00 pm
by bpizzutiWHI
Isn't that an Apache log? Sounds like that error is probably being caused by the geoip plugin if anything (I removed that from the config to make it easier to read) but that one's actually WORKING. ANd that doesn't explain why it's not working with IIS events either.

Re: Nagios Log Server - useragent logstash plugin

Posted: Fri Sep 01, 2017 8:30 am
by bpizzutiWHI
Never mind, I figured it out. managed to find an example of the useragent plugin config, and unlike the geoip plugin, useragent actually requires a target to be specified.

Re: Nagios Log Server - useragent logstash plugin

Posted: Fri Sep 01, 2017 9:50 am
by scottwilkerson
Excellent. Glad you were able to figure it out.