EVentIDs not showing

[mcapra]

Can we also see a few sample log entries? This will help narrow down issues that might exist within the grok filter.

[CFT6Server]

I believe Windows event logs are automatically filtered? Or the json input will automatically handle event logs that are coming in. I do not see the entire event logs for some entries. I don't think there's a grok filter in place, unless it is in a pattern file? For example:

Code: Select all

The computer attempted to validate the credentials for an account.

Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:	<accountname>
Source Workstation:	<Computername>
Error Code:	0x0

This is the message, but of course everything is parsed to the fields already.

Here are some others:

Code: Select all

An account was successfully logged on.

Subject:
	Security ID:		S-1-0-0
	Account Name:		-
	Account Domain:		-
	Logon ID:		0x0

Logon Type:			3

Impersonation Level:		Impersonation

New Logon:
	Security ID:		S-1-5-21-39997874-886523497-1299147156-88363
	Account Name:		<accountname>
	Account Domain:		<DOMAIN>
	Logon ID:		0x5B64539F
	Logon GUID:		{2413BCF9-9F53-DA70-E100-05EE153CCCAD}

Process Information:
	Process ID:		0x0
	Process Name:		-

Network Information:
	Workstation Name:	
	Source Network Address:	<IP>
	Source Port:		63017

Detailed Authentication Information:
	Logon Process:		Kerberos
	Authentication Package:	Kerberos
	Transited Services:	-
	Package Name (NTLM only):	-
	Key Length:		0

This event is generated when a logon session is created. It is generated on the computer that was accessed.

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The impersonation level field indicates the extent to which a process in the logon session can impersonate.

The authentication information fields provide detailed information about this specific logon request.
	- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
	- Transited services indicate which intermediate services have participated in this logon request.
	- Package name indicates which sub-protocol was used among the NTLM protocols.
	- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

Code: Select all

An account was logged off.

Subject:
	Security ID:		S-1-5-21-39997874-886523497-1299147156-143786
	Account Name:		<Computername>$
	Account Domain:		DOMAIN
	Logon ID:		0x44f51a889

Logon Type:			3

This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.

[mcapra]

Just to clarify, the nxlog configuration you posted earlier was from one of the sources producing the (string,long) fields? Going to try and replicate this internally with the samples you've provided.

Does the logstash log (/var/log/logstash/logstash.log) contain any sort of warnings about charsets? The json codec can sometimes have a bit of trouble juggling those.

[CFT6Server]

Yes. All event logs are forwarded to an Event Collector and then from there a single nxlog source is going to NLS.

I am watching the logs for a bit, but I am not seeing any complains of the windows events that are coming in so far.

[mcapra]

This block in the nxlog config isn't working for me.

Code: Select all

Query <QueryList> \
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
<Select Path="Setup">*</Select>\
<Select Path="ForwardedEvents">*</Select>\
</Query>\
</QueryList>

Says "invalid keyword: Query". Probably not the source of the issues though since it just invalidates the input completely in my case. I wasn't able to replicate the EventID field being multi-typed with any of that configuration though.

You could try forcibly casting the field's type, but i'm not sure what this will do to existing indices in the interim. All the documentation indicates that based on how multi fields work this should be fine, but if you start missing eventlog entries you'll know something is up.

Adding this to the filter might work to force a long type for future indices:

Code: Select all

mutate {
      convert => { "EventID" => "long" }
    }

We can certainly continue to troubleshoot this over the forums, but Is it possible for you to submit a ticket and schedule a remote assistance session? Without being able to replicate/observe the specific issue the problem becomes progressively more difficult to troubleshoot.

[CFT6Server]

So "long" does not work and I get an error.

Code: Select all

{:timestamp=>"2016-08-24T15:44:45.766000-0700", :message=>"Error: Cannot register filter mutate plugin. The error reported is: \n  Invalid conversion type 'long', expected one of 'string,integer,float,boolean'"}

So I changed it to Integer just to test, but the field is not updating

eventid.JPG

[mcapra]

The schema elasticsearch is using for these entries cannot (easily) be updated on the fly. The field type shouldn't be updating until the next index is started.

Since I assume this was changed on Friday, can you check the latest index to see if the mutate correctly altered the EventID field?

[CFT6Server]

I know the new fields takes effect once the new index is created, but the new indexes since does not reflect any change to the type in this case, which is a bit baffling.

[mcapra]

Can you show us your elasticsearch log? Feel free to PM it to me if you have security concerns. It should be located at /var/log/elasticsearch/<cluster id>.log

Is moving this to a ticket and scheduling a remote assistance session an option for you?

[CFT6Server]

Remote assistance to ticket would work for me. I will PM you the logs.