Nagios Log Server - useragent logstash plugin

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
bpizzutiWHI
Posts: 64
Joined: Thu Mar 02, 2017 10:15 am

Nagios Log Server - useragent logstash plugin

Post by bpizzutiWHI »

I'm attempting to use this logstash plugin, I'm assuming it's enabled. I've got my Apache logs coming in in varios ways, my IIS logs are coming in via EventLogging. I've got the following filters set up but I don't see any fields or data being generated by the useragent plugin. Any ideas on how to get this working?

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Wed, 30 Aug 2017 13:12:03 -0400
#

#
# Global filters
#

filter {
    if [program] == 'apache_access' {
        grok {
            match => [ 'message', '%{COMMONAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_access' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
    
    if [program] == 'apache_custom' {
        grok {
            match => [ 'message', '%{CUSTOMAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_custom' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
    if [program] == 'apache_combined' {
        grok {
            match => [ 'message', '%{COMBINEDAPACHELOG}']
        }
        date {
            match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z', 'MMM dd HH:mm:ss', 'ISO8601' ]
        }
        mutate {
            replace => [ 'type', 'apache_combined' ]
             convert => [ 'bytes', 'integer' ]
             convert => [ 'response', 'integer' ]
        }
    }
     
    if [program] == 'apache_error' {
        grok {
            match => [ 'message', '%{HTTPD20_ERRORLOG}']
        }
        mutate {
            replace => [ 'type', 'apache_error' ]
        }
    }
    if [SourceName] == 'Microsoft-Windows-IIS-Logging' {
        useragent {
            source => 'csUser-Agent'
        }
    }
    if [program] == 'apache_access' {
        useragent {
            source => 'agent'
        }
    }
    if [program] == 'apache_combined' {
        useragent {
            source => 'agent'
        }
    }
    if [program] == 'apache_custom' {
        useragent {
            source => 'agent'
        }
    }
    if [program] == 'apache_error' {
        useragent {
            source => 'agent'
        }
    }
}

#
# Local filters
#


dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Nagios Log Server - useragent logstash plugin

Post by dwhitfield »

Can you attach all the files located in the following paths:

Code: Select all

    /var/log/logstash
    /var/log/elasticsearch
If you could compress them into a zip or tarball, that would be ideal.

I will not be in the next couple of days, so if you do not want them on the forum you can PM them to @cdienger.
bpizzutiWHI
Posts: 64
Joined: Thu Mar 02, 2017 10:15 am

Re: Nagios Log Server - useragent logstash plugin

Post by bpizzutiWHI »

I don't mind, here you go
You do not have the required permissions to view the files attached to this post.
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios Log Server - useragent logstash plugin

Post by cdienger »

I see a few messages indicating it's having a problem parsing them:

message=>"IP Field contained invalid IP address or hostname"

and later...

"clientip"=>"www.nexpartqa.com"

Looking at the logged line:

216.113.160.81 http://www.nexpartqa.com - - [30/Aug/2017:16:13:29 -0400] \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36\"

There seems to be a few odd things. First the domain after the IP second the "\"s to escape the quotes around the user-agent.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
bpizzutiWHI
Posts: 64
Joined: Thu Mar 02, 2017 10:15 am

Re: Nagios Log Server - useragent logstash plugin

Post by bpizzutiWHI »

Isn't that an Apache log? Sounds like that error is probably being caused by the geoip plugin if anything (I removed that from the config to make it easier to read) but that one's actually WORKING. ANd that doesn't explain why it's not working with IIS events either.
bpizzutiWHI
Posts: 64
Joined: Thu Mar 02, 2017 10:15 am

Re: Nagios Log Server - useragent logstash plugin

Post by bpizzutiWHI »

Never mind, I figured it out. managed to find an example of the useragent plugin config, and unlike the geoip plugin, useragent actually requires a target to be specified.
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Nagios Log Server - useragent logstash plugin

Post by scottwilkerson »

Excellent. Glad you were able to figure it out.
Former Nagios employee
Creator:
Human Design Website
Get Your Human Design Chart
Locked