EVentIDs not showing

[CFT6Server]

We are collecting Windows Event Logs and the field EventID is there, but looks like when I setup a table, it isn't able to search for the field. I am not sure why....

[CFT6Server]

Here's the micro analysis showing the eventIDs.

micro.JPG

[jolson]

My suspicion is that this has to do with your EventID logs showing up at your system as both strings and long integers - see here:

2016-01-07 13_45_57-micro.JPG (JPEG Image, 553 × 235 pixels).png

Do you know why this might be? Is it possible to narrow down your time window to such a point where it is only either a string or a long integer? Maybe set a custom date and go back in time until the field only shows one or the other.

If you can find a point in time where the EventID field is _either_ a string _or_ a long, try generating a panel with that timeperiod selected. Does the behavior change?

[CFT6Server]

EventIDs are coming in via nxlog only so I am not sure why they are coming in as different data types. I will have to try to narrow down to see what is setting different types.

[jolson]

Sounds good to me - did you try finding a date during which EventID only had one field type? I've tried to reproduce this in my lab, but can't get a field to accept two different types simultaneously - I think I need to modify the mapping or similar. Let me know what you find out!

[CFT6Server]

So I thought about two scenarios where this might have caused this. I have two main sources of event logs from windows.

1. Event Collector that collects and sends events to LS (one source in LS, for many hosts)
2. Direct logs using nxlogs from certain servers that cannot connect to the event collector

I have filtered and separated these two and doesn't seem to make any difference. They are still showing as string,long.....
So I am not sure what could be setting the EventIDs.

I have another 2 seperate LS that collects only from Windows Event Collector at different sites and looks like those are showing up as long only. So I think it most be one of the servers that are sending in logs directly from nxlog that is causing this.... however, going to find out which might take a bit of effort.

Also now that the field is set to string,long I am not sure how to 'reset' that as it is already set in the index isn't it? so essentially I have to wait until the new index is created before seeing any changes to the field?

[jolson]

Also now that the field is set to string,long I am not sure how to 'reset' that as it is already set in the index isn't it? so essentially I have to wait until the new index is created before seeing any changes to the field?

That is an accurate statement - after a field has types assigned to it, changing those types is a process. Typically it's easiest to wait until the next index is generated (normally at midnight on the next day).

however, going to find out which might take a bit of effort.

It will take some hunting. Is there any way you know of that would allow me to reproduce this problem? I could try and resolve it on my end if I got two types assigned to the same field. Perhaps a few of your nxlog configs would do?

[CFT6Server]

I will need to re-address this again as the EventID issue was never resolved and we are not able to filter via EventIDs. It is still showing as (string, long).

[mcapra]

Can we see the following for one of the problem servers:

• grok filter being applied
• nxlog configuration file
• a sample of the input being sent

[CFT6Server]

Grok Filter - we only manipulate hostname.

Code: Select all

if [type] == "eventlog" {
   grok {
      match => ["Hostname", "%{WORD:Hostname}"]
      overwrite => ["Hostname"]
    }
    date {
       match => ["EventTime", "YYYY-MM-dd HH:mm:ss"]
      }
}
if [type] == "dmz-eventlog" {
   grok {
      match => ["Hostname", "%{WORD:Hostname}"]
      overwrite => ["Hostname"]
      add_tag => ["Event Log"]
    }
    date {
       match => ["EventTime", "YYYY-MM-dd HH:mm:ss"]
      }
}

Input

Code: Select all

tcp {
    type => 'eventlog'
    port => 3515
    tags => 'Event Log'
    codec => json {
        charset => 'CP1252'
    }
}

nxlog configuration

Code: Select all

#This DEFINES the ROOT folder nxlog is installed into
define ROOT D:\nxlog
define CERT D:\nxlog\cert

Moduledir D:\nxlog\modules
CacheDir D:\nxlog\cachedata
Pidfile D:\nxlog\data\nxlog.pid
SpoolDir D:\nxlog\spooldata
LogFile D:\nxlog\data\nxlog.log

<Extension json>
    Module      xm_json
</Extension>

# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog

Query <QueryList> \
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
<Select Path="Setup">*</Select>\
<Select Path="ForwardedEvents">*</Select>\
</Query>\
</QueryList>

# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
</Input>

<Processor buffer>
    Module      pm_buffer
    MaxSize	1900000
    Type	Mem
    WarnLimit	1572864
</Processor>

<Processor diskbuffer>
    Module  pm_buffer
    MaxSize 3145728
    Type    Disk
    File    "D:\nxlog\buffer"
    WarnLimit   2097152
</Processor>

<Output out>
    Module om_tcp
    Host 10.242.102.108
    Port 3515

    Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec  $raw_event = to_json();

	# Uncomment for debug output
	# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Route 1>
#  Path eventlog => diskbuffer => buffer => out
  Path eventlog => out
</Route>

Events are coming into a windows event collector and we have a single nxlog sending to NLS.