Page 1 of 1

http input plugin

Posted: Fri Feb 26, 2016 8:37 am
by WillemDH
Hello,

Is this plugin supported yet?

https://www.elastic.co/blog/introducing ... ttp-plugin

Has anyone tested this work over ssl?

I'm in fact thinking of a way to send data to my nls different then using this function I've been using over the year in Powershell:

Code: Select all

function Write-Log {
    [CmdletBinding()]
    param (
        [parameter(Mandatory=$true)][string]$Log,
        [parameter(Mandatory=$true)][ValidateSet('Debug', 'Info', 'Warning', 'Error')][string]$Severity,
        [parameter(Mandatory=$true)][string]$Message
    )
    $Now = Get-Date -Format 'yyyy-MM-dd HH:mm:ss,fff'
    $LocalScriptName = split-path $MyInvocation.PSCommandPath -Leaf
    if ($Log -eq 'Undefined') {
        Write-Debug "${Now}: ${LocalScriptName}: Info: LogServer is undefined."
    }
    elseif ($Log -eq 'Verbose') {
        Write-Verbose "${Now}: ${LocalScriptName}: ${Severity}: $Message"
    }
    elseif ($Log -eq 'Debug') {
        Write-Debug "${Now}: ${LocalScriptName}: ${Severity}: $Message"
    }
    elseif ($Log -eq 'Output') {
        Write-Host "${Now}: ${LocalScriptName}: ${Severity}: $Message"
    }
    elseif ($Log -match '^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])(?::(?<port>\d+))$' -or $Log -match "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$") {
        $IpOrHost = $log.Split(':')[0]
        $Port = $log.Split(':')[1]
        if  ($IpOrHost -match '^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$') {
            $Ip = $IpOrHost
        }
        else {
            $Ip = ([System.Net.Dns]::GetHostAddresses($IpOrHost)).IPAddressToString
        }
        Try {
            $LocalHostname = ([System.Net.Dns]::GetHostByName((hostname.exe)).HostName).tolower()
            $JsonObject = (New-Object PSObject | 
                Add-Member -PassThru NoteProperty logsource $LocalHostname | 
                Add-Member -PassThru NoteProperty hostname $LocalHostname | 
                Add-Member -PassThru NoteProperty scriptname $LocalScriptName | 
                Add-Member -PassThru NoteProperty logtime $Now | 
                Add-Member -PassThru NoteProperty severity_label $Severity | 
                Add-Member -PassThru NoteProperty message $Message ) | 
                ConvertTo-Json
            $JsonString = $JsonObject -replace "`n",' ' -replace "`r",' '
            $Socket = New-Object System.Net.Sockets.TCPClient($Ip,$Port) 
            $Stream = $Socket.GetStream() 
            $Writer = New-Object System.IO.StreamWriter($Stream)
            $Writer.WriteLine($JsonString)
            $Writer.Flush()
            $Stream.Close()
            $Socket.Close()
        }
        catch {
            Write-Host "${Now}: ${LocalScriptName}: Error: Something went wrong while trying to send message to Logstash server `"$Log`"."
        }
        Write-Verbose "${Now}: ${LocalScriptName}: ${Severity}: Ip: $Ip Port: $Port JsonString: $JsonString"
    }
    elseif ($Log -match '^((([a-zA-Z]:)|(\\{2}\w+)|(\\{2}(?:(?:25[0-5]|2[0-4]\d|[01]\d\d|\d?\d)(?(?=\.?\d)\.)){4}))(\\(\w[\w ]*))*)') {
        if (Test-Path -Path $Log -pathType container){
            Write-Host "${Now}: ${LocalScriptName}: Error: Passed Path is a directory. Please provide a file."
            exit 1
        }
        elseif (!(Test-Path -Path $Log)) {
            try {
                New-Item -Path $Log -Type file -Force | Out-null	
            } 
            catch { 
                $Now = Get-Date -Format 'yyyy-MM-dd HH:mm:ss,fff'
                Write-Host "${Now}: ${LocalScriptName}: Error: Write-Log was unable to find or create the path `"$Log`". Please debug.."
                exit 1
            }
        }
        try {
            "${Now}: ${LocalScriptName}: ${Severity}: $Message" | Out-File -filepath $Log -Append   
        }
        catch {
            Write-Host "${Now}: ${LocalScriptName}: Error: Something went wrong while writing to file `"$Log`". It might be locked."
        }
    }
}
This only works with an ip and a port however and is not encrypted. This is very low prio thread, but it would be nice to get some advice so we can come to the best way to send data to nls encrypted with Powershell. We have a pki, so I can generate the required certificates SHA256. I've read through the pdf with the documentation how to do this with NxLog. There tcp input is also used with certificates.

Code: Select all

tcp {
port => 7777
type => "nxlogs"
ssl_cacert => "/etc/pki/tls/certs/rootCA.pem"
Page 1
Copyright © 2010-2014 Nagios Enterprises, LLC
Revision 1.0 – February, 2016
Nagios Log Server – Sending nxlogs with SSL
Nagios Enterprises, LLC US: 1-888-NAGIOS-1 Web: www.nagios.com
P.O. Box 8154
Saint Paul, MN 55108
USA
Int'l: +1 651-204-9102 Email: [email protected]
Fax: +1 651-204-9103
ssl_cert => "/etc/pki/tls/certs/device-nls.crt"
ssl_key => "/etc/pki/tls/private/device-nls.key"
ssl_enable => true
format => 'json'
}
I'd like to find out what input is the most secure and easy to use.

Thanks already and grtz

Willem

Re: http input plugin

Posted: Fri Feb 26, 2016 1:46 pm
by rkennedy
I originally wrote that document for NX Log with SSL and was surprised at the simplicity behind it. There really wasn't much to it.

It looks to be installing fine on NLS -

Code: Select all

[root@localhost logstash]# bin/plugin install logstash-input-http
Validating logstash-input-http
Installing logstash-input-http
Installation successful
Here's some testing for you. It looks to be working fine over HTTP, but I ran into issues with SSL and using the java keystore (this is new to me). I don't think it's a NLS error, but rather mine. I don't have much more time to poke at the SSL today, so here's what I've concluded.

As for over HTTP, it seems to be working fine (I know you're looking for SSL) -

Input:

Code: Select all

http {
   port => 8080
}
CLI:

Code: Select all

curl -XPUT 'http://127.0.0.1:8080/test' -d 'testing...'
nls-http-input.PNG
For SSL, I had to alter the configuration a little bit for the input side (ssl on -> true) -

Code: Select all

http {
   port => 4040
   user => username
   password => "inputpassword"
   ssl => true
   keystore => "/usr/local/nagioslogserver/ssl/keystore.jks"
   keystore_password => "keystore"
}
I haven't used java keystore files before so I may have done something wrong, but I imported the rootCA.pem + client certificate. At this point, it should work but, I'm hitting a wall with SSL right now though (I presume not NLS related) -

Code: Select all

[root@localhost ssl]# curl -XPUT 'https://127.0.0.1:4040/test/' -d 'ssl testing'
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
I ran out of time for now Willem, but you should be able to input both TCP / HTTP over SSL (provided the certificates are correct :D). I am willing to look more into this as time permits though, so this won't be it.

Re: http input plugin

Posted: Sat Feb 27, 2016 5:26 am
by WillemDH
Give me some time to find some time to test this at my end. I also have no experience with java Keystore certificates. I'll have to see if I can convert my pfx ( as our pki is from MS) to it. so for using the http input I'm supposed to add a username and password? This is an nls user?

It might take some time before I can test this further. But as I said before, this is very low prio. Tx!

Re: http input plugin

Posted: Mon Feb 29, 2016 10:24 am
by tmcdonald
We'll keep this open, feel free to post back once you have more!

Re: http input plugin

Posted: Mon Feb 29, 2016 10:33 am
by jolson
so for using the http input I'm supposed to add a username and password? This is an nls user?
Those two settings are responsible for controlling access to the HTTP server that you're setting up with this plugin - this would especially make sense if you were concerned about the security of this web server.