Nagios Support Forum • Powershell log directly to NLS

Page 1 of 2

Powershell log directly to NLS

Posted: Wed May 13, 2015 9:29 am

by WillemDH

Hello,

Would it be possible to send logs from a Powershell script immediately to NLS? If I could create a json file and send it to one of the NLS node on some port where I have an input defined, should this work?

Grtz

WIllem

Re: Powershell log directly to NLS

Posted: Wed May 13, 2015 9:57 am

by jolson

Logstash will take any old raw data, so there should be no problem with what you've described. Of course I recommend using the json codec if you're sending json formatted logs - besides that, set up a generic tcp/udp input and you should be good to go.

Re: Powershell log directly to NLS

Posted: Fri May 15, 2015 9:56 am

by WillemDH

Jesse,

I'm almost there. NLS catches the Powershell object I'm sending over TCP.

So I made this inpu:

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
    codec => json {
        charset => 'CP1252'
    }
}

Then made this script:

Code: Select all

Function Send-TCPMessage { 
    param ( [ValidateNotNullOrEmpty()] 
        [string] $EndPoint, 
        [int] $Port, 
        $Message
    ) 
     
    $UTF8 = [System.Text.Encoding]::UTF8
    $IP = [System.Net.Dns]::GetHostAddresses($EndPoint) 
    $Address = [System.Net.IPAddress]::Parse($IP) 
    $Socket = New-Object System.Net.Sockets.TCPClient($Address,$Port) 
    $data = $UTF8.GetBytes($Message)
    $Stream = $Socket.GetStream() 
    $Writer = New-Object System.IO.StreamWriter($Stream)
<#    $Message | %{
        $Writer.WriteLine($_)
        $Writer.Flush()
    }#>
    $Writer.WriteLine($Message)
    $Writer.Flush()
    $Stream.Close()
    $Socket.Close()
}

$Test = (New-Object PSObject |
   Add-Member -PassThru NoteProperty Name 'Json Smurf' |
   Add-Member -PassThru NoteProperty Age 34        |
   Add-Member -PassThru NoteProperty Weight 64     |
   Add-Member -PassThru NoteProperty Hobbiess ("Adventure","Reading","Monitoring")
) | ConvertTo-JSON

Write-Host $Test

Send-TCPMessage NLSServer 5551 "$Test"

But NLS treats my json object as a series of strings ap)parently. I tried

Code: Select all

$Message | %{
        $Writer.WriteLine($_)
        $Writer.Flush()

Code: Select all

 $Writer.WriteLine($Message)

Code: Select all

$Writer.Write($Message)

But result seems always the same, see screenshot. You happen to have an idea what I need to do to make it treat my json as one logline?

Re: Powershell log directly to NLS

Posted: Fri May 15, 2015 10:37 am

by jolson

I am glad to hear that we're almost there.

Logstash is the service running on NLS that's catching these logs - and logstash is a relatively dumb in terms of operation. It does exactly what you tell it to - in this case, it's listening on TCP for any data, and it's processing it using the JSON codec (formatting). Based on this, we know that the problem is not with logstash doing something strange. The problem very likely lies with powershell sending its output line-by-line instead of all together - is there a way that you know of to get powershell to concatenate its results and send them all together?

Re: Powershell log directly to NLS

Posted: Fri May 15, 2015 11:17 am

by WillemDH

Jesse,

Yes I can with for example

Code: Select all

$JsonString = $Test -replace "`n"," " -replace "`r"," " -replace " ",""

But from the moment I send it like that it doesn't show up in Logstash.

Code: Select all

Function Send-TCPMessage { 
    param ( [ValidateNotNullOrEmpty()] 
        [string] $EndPoint, 
        [int] $Port, 
        $Message
    ) 
     
    $UTF8 = [System.Text.Encoding]::UTF8
    $IP = [System.Net.Dns]::GetHostAddresses($EndPoint) 
    $Address = [System.Net.IPAddress]::Parse($IP) 
    $Socket = New-Object System.Net.Sockets.TCPClient($Address,$Port) 
    $data = $UTF8.GetBytes($Message)
    $Stream = $Socket.GetStream() 
    $Writer = New-Object System.IO.StreamWriter($Stream)
<#    $Message | %{
        $Writer.WriteLine($_)
        $Writer.Flush()
    }#>
    $Writer.Write($Message)
    $Writer.Flush()
    $Stream.Close()
    $Socket.Close()
}

$Object = (New-Object PSObject |
   Add-Member -PassThru NoteProperty Name 'Json Smurf' |
   Add-Member -PassThru NoteProperty Age 34        |
   Add-Member -PassThru NoteProperty Weight 64     |
   Add-Member -PassThru NoteProperty Hobbiess ("Adventure","Reading","Monitoring")
) 

Write-Host "Object: $Object"

$Json = $Object | ConvertTo-Json

Write-Host "Json: $Test"

$JsonString = $Test -replace "`n"," " -replace "`r"," " -replace " ",""

Write-Host "Sending `"$JsonString`""

Send-TCPMessage srvnaglog01.gentgrp.gent.be 5551 "$JsonString"

Sending this now: {"Name":"JsonSmurf","Age":34,"Weight":64,"Hobbiess":["Adventure","Reading","Monitoring"]}

When I put it in json validator it is valid.

Still using this filter:

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
    codec => json {
        charset => 'CP1252'
    }
}

When I do a tcpdump on the port I can see the message coming in.... Any idea why it is not visible in NLS?

Grtz

Re: Powershell log directly to NLS

Posted: Fri May 15, 2015 11:30 am

by jolson

What happens if you remove the codec from your input? Or maybe just the "charset => 'CP1252'" definition?

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
}

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
    codec => json {
    }
}

Re: Powershell log directly to NLS

Posted: Fri May 15, 2015 11:39 am

by WillemDH

Seems it works with

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
}

Althought as it is no json object, everything is in the message, see screenshot. If it is json, the fields should be populated automatically right?

But it does not work with

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
    codec => json {
    }
}

or

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
    codec => json
}

Re: Powershell log directly to NLS

Posted: Fri May 15, 2015 2:14 pm

by ssax

Your script looks like it's sending in UTF-8, have you tried it with?

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
    codec => json {
        charset => 'UTF-8'
    }
}

Re: Powershell log directly to NLS

Posted: Fri May 15, 2015 2:34 pm

by jolson

I am performing some testing in-house. When I run your script via powershell, it does not appear to work properly. However, if I send the line directly using linux, it's picked up and works properly. This makes me think that Windows is adding something to the log-line when processing the powershell script - perhaps a hidden character or similar. I took a tcpdump of either capture and diff'd them:

Code: Select all

[root@localhost ~]# diff /root/json1 /root/json2
1,27c1,30
< 17:42:07.198090 IP x.x.x.x.61737 > x.x.x.x.5551: Flags [S], seq 4279265083, win 64240, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
<         0x0000:  4500 0034 7c69 4000 7f06 f356 c0a8 05e9  E..4|[email protected]....
<         0x0010:  c0a8 04ca f129 15af ff10 673b 0000 0000  .....)....g;....
<         0x0020:  8002 faf0 7afc 0000 0204 05b4 0103 0302  ....z...........
<         0x0030:  0101 0402                                ....
< 17:42:07.198638 IP x.x.x.x.61737 > x.x.x.x.5551: Flags [.], ack 1830534608, win 16060, length 0
<         0x0000:  4500 0028 7c6a 4000 7f06 f361 c0a8 05e9  E..(|[email protected]....
<         0x0010:  c0a8 04ca f129 15af ff10 673c 6d1b bdd0  .....)....g<m...
<         0x0020:  5010 3ebc 4d02 0000 0000 0000 0000       P.>.M.........
< 17:42:07.199567 IP x.x.x.x.61737 > x.x.x.x.5551: Flags [P.], seq 0:89, ack 1, win 16060, length 89
<         0x0000:  4500 0081 7c6b 4000 7f06 f307 c0a8 05e9  E...|k@.........
<         0x0010:  c0a8 04ca f129 15af ff10 673c 6d1b bdd0  .....)....g<m...
<         0x0020:  5018 3ebc a170 0000 7b22 4e61 6d65 223a  P.>..p..{"Name":
<         0x0030:  224a 736f 6e53 6d75 7266 222c 2241 6765  "JsonSmurf","Age
<         0x0040:  223a 3334 2c22 5765 6967 6874 223a 3634  ":34,"Weight":64
<         0x0050:  2c22 486f 6262 6965 7373 223a 5b22 4164  ,"Hobbiess":["Ad
<         0x0060:  7665 6e74 7572 6522 2c22 5265 6164 696e  venture","Readin
<         0x0070:  6722 2c22 4d6f 6e69 746f 7269 6e67 225d  g","Monitoring"]
<         0x0080:  7d                                       }
< 17:42:07.200559 IP x.x.x.x.61737 > x.x.x.x.5551: Flags [F.], seq 89, ack 1, win 16060, length 0
<         0x0000:  4500 0028 7c6d 4000 7f06 f35e c0a8 05e9  E..(|m@....^....
<         0x0010:  c0a8 04ca f129 15af ff10 6795 6d1b bdd0  .....)....g.m...
<         0x0020:  5011 3ebc 4ca8 0000 0000 0000 0000       P.>.L.........
< 17:42:07.203033 IP x.x.x.x.61737 > x.x.x.x.5551: Flags [.], ack 2, win 16060, length 0
<         0x0000:  4500 0028 7c6e 4000 7f06 f35d c0a8 05e9  E..(|n@....]....
<         0x0010:  c0a8 04ca f129 15af ff10 6796 6d1b bdd1  .....)....g.m...
<         0x0020:  5010 3ebc 4ca7 0000 0000 0000 0000       P.>.L.........
---
> 17:42:31.926129 IP x.x.x.x.38954 > x.x.x.x.5551: Flags [S], seq 2652321108, win 14600, options [mss 1460,sackOK,TS val 185267070 ecr 0,nop,wscale 6], length 0
>         0x0000:  4500 003c c62a 4000 4006 e9e3 c0a8 0492  E..<.*@.@.......
>         0x0010:  c0a8 04cb 982a 15af 9e17 3554 0000 0000  .....*....5T....
>         0x0020:  a002 3908 047d 0000 0204 05b4 0402 080a  ..9..}..........
>         0x0030:  0b0a f37e 0000 0000 0103 0306            ...~........
> 17:42:31.926314 IP x.x.x.x.38954 > x.x.x.x.5551: Flags [.], ack 2577196732, win 229, options [nop,nop,TS val 185267070 ecr 2583056353], length 0
>         0x0000:  4500 0034 c62b 4000 4006 e9ea c0a8 0492  E..4.+@.@.......
>         0x0010:  c0a8 04cb 982a 15af 9e17 3555 999c e6bc  .....*....5U....
>         0x0020:  8010 00e5 012a 0000 0101 080a 0b0a f37e  .....*.........~
>         0x0030:  99f6 4fe1                                ..O.
> 17:42:31.926339 IP x.x.x.x.38954 > x.x.x.x.5551: Flags [P.], seq 0:90, ack 1, win 229, options [nop,nop,TS val 185267070 ecr 2583056353], length 90
>         0x0000:  4500 008e c62c 4000 4006 e98f c0a8 0492  E....,@.@.......
>         0x0010:  c0a8 04cb 982a 15af 9e17 3555 999c e6bc  .....*....5U....
>         0x0020:  8018 00e5 558d 0000 0101 080a 0b0a f37e  ....U..........~
>         0x0030:  99f6 4fe1 7b22 4e61 6d65 223a 224a 736f  ..O.{"Name":"Jso
>         0x0040:  6e53 6d75 7266 222c 2241 6765 223a 3334  nSmurf","Age":34
>         0x0050:  2c22 5765 6967 6874 223a 3634 2c22 486f  ,"Weight":64,"Ho
>         0x0060:  6262 6965 7373 223a 5b22 4164 7665 6e74  bbiess":["Advent
>         0x0070:  7572 6522 2c22 5265 6164 696e 6722 2c22  ure","Reading","
>         0x0080:  4d6f 6e69 746f 7269 6e67 225d 7d0a       Monitoring"]}.
> 17:42:31.926447 IP x.x.x.x.38954 > x.x.x.x.5551: Flags [F.], seq 90, ack 1, win 229, options [nop,nop,TS val 185267070 ecr 2583056353], length 0
>         0x0000:  4500 0034 c62d 4000 4006 e9e8 c0a8 0492  E..4.-@.@.......
>         0x0010:  c0a8 04cb 982a 15af 9e17 35af 999c e6bc  .....*....5.....
>         0x0020:  8011 00e5 00cf 0000 0101 080a 0b0a f37e  ...............~
>         0x0030:  99f6 4fe1                                ..O.
> 17:42:31.930697 IP x.x.x.x.38954 > x.x.x.x.5551: Flags [.], ack 2, win 229, options [nop,nop,TS val 185267074 ecr 2583056357], length 0
>         0x0000:  4500 0034 c62e 4000 4006 e9e7 c0a8 0492  E..4..@.@.......
>         0x0010:  c0a8 04cb 982a 15af 9e17 35b0 999c e6bd  .....*....5.....
>         0x0020:  8010 00e5 00c6 0000 0101 080a 0b0a f382  ................
>         0x0030:  99f6 4fe5                                ..O.

json1 is being sent from windows using your powershell script, while json2 is being sent via a different linux box directly. I can't see any distinct differences between the two at the moment. I will do further testing, but I figured I would provide you with what I have so far.

Re: Powershell log directly to NLS

Posted: Sat May 16, 2015 5:52 am

by WillemDH

Hey Jesse,

Well,

Code: Select all

$UTF8 = [System.Text.Encoding]::UTF8

and

Code: Select all

$data = $UTF8.GetBytes($Message)

seemed I did never really use $data, so I left it all out.

I tried with UTF-8

Code: Select all

tcp {
    type => 'powershell'
    port => 5551
    codec => json {
        charset => 'UTF-8'
    }
}

And did some tests. FIrst tests failed (with $Writer.Write($Message))

But then I tried again with $Writer.WriteLine($Message) and NLS seemd to be able to detect the json and assign the correct fields.

I made a blogpost about this function. http://outsideit.net/json-over-tcp-with-powershell/

Grtz

WIllem