Nagios LS - Create fields with Windows DHCP files

[jameshanguyen]

Hi there,
I change the nxlog.conf like this https://gist.github.com/Eagle6705/3d91b2270bf60b7cff12 and can load the lines in Windows DHCP files to Nagios Log Server.
So each message I receive like this:
11,12/20/19,21:33:44,Renew,10.10.106.54,Johnmacbook.sapa.net,18602487D86B,,2360084275,0,,,,0x4D53465420352E30,MSFT 5.0,,,,0

I would like to assign the values of each message to the fields, so I create a filter like this (10.10.10.10 is the ip address of the dhcp server which has the DHCP files):
if [host] == '10.10.10.10' {
dissect {
mapping => { 'message' => '%{ID} %{Date} %{Time} %{Description} %{IPAddress} %{HostName} %{MACAddress} %{UserName} %{TransactionID} %{QResult} %{Probationtime} %{CorrelationID} %{Dhcid} %{VendorClassHex} %{VendorClassASCII} %{UserClassHex} %{UserClassASCII} %{RelayAgentInformation} %{DnsRegError}'
}
}
}

It doesn't pass the Verify Configuration because "Reason: Couldn't find any filter plugin named 'dissect'".
So, what filter can I use and how ?
Thanks.

[scottwilkerson]

No filter should be necessary

So each message I receive like this:
11,12/20/19,21:33:44,Renew,10.10.106.54,Johnmacbook.sapa.net,18602487D86B,,2360084275,0,,,,0x4D53465420352E30,MSFT 5.0,,,,0

Does your nxlog.conf have this section

Code: Select all

#Fields obtained from DHCP Server logs
<Extension ParseDHCP>
        Module  xm_csv
                
		Fields $ID ,$Date ,$Time ,$Description ,$IPAddress ,$ReportedHostname ,$MACAddress ,$UserName ,$TransactionID ,$QResult ,$Probationtime ,$CorrelationID ,$Dhcid ,$VendorClass(Hex) ,$VendorClass(ASCII) ,$UserClass(Hex) ,$UserClass(ASCII) ,$RelayAgentInformation ,$DnsRegError
        Delimiter       ','
</Extension>

and this line in the Input?

Code: Select all

ParseDHCP->parse_csv();	

[jameshanguyen]

Yes, of course because I copied the file in the link I gave.
I just deleted all things about the IIS.
Here is my nxlog.conf (10.10.10.100 is the Nagios server):

## See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-re ... anual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

## Config file structure
##
## nxLog Directory Locations
## Extensions
## IIS Log Parsing Modules (If IIS is detected)
## Input Modules
## Dedupe for Windows Logs
## Output Modules
## Route Modules
##
# Tested on Server 2008, Server 2008 R2
# Adjust Out modules based on your own logstash configurations

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

#Extensions----------------------------------------------------------------------------------

<Extension gelf>
Module xm_gelf
</Extension>

<Extension json>
Module xm_json
</Extension>
#Uncomment this and the file out in the DHCP OUT to check output.
#<Extension fileop>
# Module xm_fileop
#</Extension>

#Extensions----------------------------------------------------------------------------------

# Select the input folder where logs will be scanned
# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
# Uncomment Extension w3c for IIS logging

# Window Event Log
<Input in>
Module im_msvistalog

</Input>

#Fields obtained from DHCP Server logs
<Extension ParseDHCP>
Module xm_csv

Fields $ID ,$Date ,$Time ,$Description ,$IPAddress ,$ReportedHostname ,$MACAddress ,$UserName ,$TransactionID ,$QResult ,$Probationtime ,$CorrelationID ,$Dhcid ,$VendorClass(Hex) ,$VendorClass(ASCII) ,$UserClass(Hex) ,$UserClass(ASCII) ,$RelayAgentInformation ,$DnsRegError
Delimiter ','
</Extension>

#DHCP logs assumed they are located in default location
#Use "sysnative" for DHCP Log location for 32-bit applications to access the SYSTEM32 directory on a 64 Bit System
#Use "system32" for DHCP Log location on 32 Bit systems
<Input DHCP_IN>
Module im_file
File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log"
SavePos TRUE
InputType LineBased
Exec $Message = $raw_event;

#Exec if $raw_event =~ /^30/ \
# log_info($raw_event); \
# $IDdef = "DNSUpdateRequest";

Exec if $raw_event =~ /^[0-9][0-9],/ \
{ \
ParseDHCP->parse_csv(); \
if $raw_event =~ /^00/ $IDdef = "The log was started."; \
if $raw_event =~ /^01/ $IDdef = "The log was stopped."; \
if $raw_event =~ /^02/ $IDdef = "The log was temporarily paused due to low disk space."; \
if $raw_event =~ /^10/ $IDdef = "A new IP address was leased to a client."; \
if $raw_event =~ /^11/ $IDdef = "A lease was renewed by a client."; \
if $raw_event =~ /^12/ $IDdef = "A lease was released by a client."; \
if $raw_event =~ /^13/ $IDdef = "An IP address was found to be in use on the network."; \
if $raw_event =~ /^14/ $IDdef = "A lease request could not be satisfied because the scope's address pool was exhausted."; \
if $raw_event =~ /^15/ $IDdef = "A lease was denied."; \
if $raw_event =~ /^16/ $IDdef = "A lease was deleted."; \
if $raw_event =~ /^17/ $IDdef = "A lease was expired and DNS records for an expired leases have not been deleted."; \
if $raw_event =~ /^18/ $IDdef = "A lease was expired and DNS records were deleted."; \
if $raw_event =~ /^20/ $IDdef = "A BOOTP address was leased to a client."; \
if $raw_event =~ /^21/ $IDdef = "A dynamic BOOTP address was leased to a client."; \
if $raw_event =~ /^22/ $IDdef = "A BOOTP request could not be satisfied because the scope's address pool for BOOTP was exhausted."; \
if $raw_event =~ /^23/ $IDdef = "A BOOTP IP address was deleted after checking to see it was not in use."; \
if $raw_event =~ /^24/ $IDdef = "IP address cleanup operation has began."; \
if $raw_event =~ /^25/ $IDdef = "IP address cleanup statistics."; \
if $raw_event =~ /^30/ $IDdef = "DNS update request to the named DNS server."; \
if $raw_event =~ /^31/ $IDdef = "DNS update failed."; \
if $raw_event =~ /^32/ $IDdef = "DNS update successful."; \
if $raw_event =~ /^33/ $IDdef = "Packet dropped due to NAP policy."; \
if $raw_event =~ /^34/ $IDdef = "DNS update request failed.as the DNS update request queue limit exceeded."; \
if $raw_event =~ /^35/ $IDdef = "DNS update request failed."; \
if $raw_event =~ /^36/ $IDdef = "Packet dropped because the server is in failover standby role or the hash of the client ID does not match."; \
if $raw_event =~ /^[5-9][0-9]/ $IDdef = "Codes above 50 are used for Rogue Server Detection information."; \
if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,0,/ $QResultDef = "NoQuarantine"; \
if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,1,/ $QResultDef = "Quarantine"; \
if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,2,/ $QResultDef = "Drop Packet"; \
if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,3,/ $QResultDef = "Probation"; \
if $raw_event =~ /^.+,.+,.+,.+,.+,.+,.+,.+,6,/ $QResultDef = "No Quarantine Information ProbationTime:Year-Month-Day Hour:Minute:Second:MilliSecond."; \
$host = hostname_fqdn(); \
$EventTime = parsedate($Date + " " + $Time); \
$SourceName = "DHCPEvents"; \
$Message = to_json(); \
} \
else \
drop();

</Input>

<Output out>
Module om_tcp
Host 10.10.10.100
Port 2057
</Output>

<Route 1>
Path DHCP_IN => out
</Route>

[jameshanguyen]

And here is the result the nagios:

{
"_index": "logstash-2019.12.21",
"_type": "import_json",
"_id": "AW8mF-hiDNDgkq0MWPuN",
"_score": null,
"_source": {
"message": "32,12/21/19,08:34:58,DNS Update Successful,10.10.10.49,JanePC.sapa.net,,,0,6,,,,,,,,,0\r",
"tags": [
"_jsonparsefailure",
"import_json"
],
"@version": "1",
"@timestamp": "2019-12-21T01:34:58.585Z",
"host": "10.10.10.10",
"port": 49744,
"type": "import_json",
"alert_names": ""
},
"highlight": {
"host.raw": [
"@start-highlight@10.10.10.10@end-highlight@"
],
"host": [
"@start-highlight@10.10.10.10@end-highlight@"
]
},
"sort": [
1576892098585,
1576892098585
]
}

[scottwilkerson]

The typical out block in our recommended files would look like this

Code: Select all

<Output out>
    Module om_tcp
    Host  10.10.10.100
    Port 3515
	
    Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec  $raw_event = to_json();
	
	# Uncomment for debug output
	# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

[jameshanguyen]

Thank you scottwilkerson very much.
It works.

[scottwilkerson]

Thank you scottwilkerson very much.
It works.

Great!

Locking