nxlog memory leak?

[rkennedy]

Thanks for that info @WillemDH. I have a feeling it's a configured setting in nxlog that we'll hopefully be able to isolate.

[CFT6Server]

Here's the configuration of nxlog.

Code: Select all

## See the nxlog reference manual at
## http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

#This DEFINES the ROOT folder nxlog is installed into
define ROOT D:\nxlog
define CERT D:\nxlog\cert

Moduledir D:\nxlog\modules
CacheDir D:\nxlog\cachedata
Pidfile D:\nxlog\data\nxlog.pid
SpoolDir D:\nxlog\spooldata
LogFile D:\nxlog\data\nxlog.log

# Include fileop while debugging, also enable in the output module below
#<Extension fileop>
#    Module xm_fileop
#</Extension>

<Extension json>
    Module      xm_json
</Extension>

#<Input internal>
#    Module im_internal
#</Input>

# Watch your own files
#<Input file1>
#    Module   im_file
#    File     '%ROOT%\data\nxlog.log'
#    SavePos  TRUE
#</Input>

# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog

Query <QueryList> \
<Query Id="0">\
<Select Path="Application">*</Select>\
<Select Path="System">*</Select>\
<Select Path="Security">*</Select>\
<Select Path="Setup">*</Select>\
<Select Path="ForwardedEvents">*</Select>\
</Query>\
</QueryList>

# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
</Input>

<Processor buffer>
    Module      pm_buffer
    MaxSize	1900000
    Type	Mem
    WarnLimit	1572864
</Processor>

<Processor diskbuffer>
    Module  pm_buffer
    MaxSize 3145728
    Type    Disk
    File    "D:\nxlog\buffer"
    WarnLimit   2097152
</Processor>

<Output out>
    Module om_tcp
    Host 10.242.102.108
    Port 3515

    Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec  $raw_event = to_json();

	# Uncomment for debug output
	# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>

<Route 1>
  Path eventlog => diskbuffer => buffer => out
</Route>

Our setup sends logs from a single windows event collector source which is where nxlog is located. I've been adjust the mem and diskbuffer settings to find out what I can tweak. Watching the peak memory of the nxlog process, it reaches just below 2GB,

I am still seeing odd errors with jsonparsefailures, but percentage wise, it isn't much, just some log entries. Since the logs all come from the same source, I am not sure why it only happens on certain ones.
here's an example of a single server where majority is fine, but I am see parse failures.

parsefailures.JPG

I am currently trying to lower the mem buffer actually and see if it hits the disk buffer due to peak mem usage observed on the windows event collector for the nxlog process. So far, it has lasted more than 12 hours.

[hsmith]

Let us know what the results of the test are. I wish I could provide more support for this particular issue, but it's difficult when it's not a product we made, just one we use heavily.

[CFT6Server]

I might need some help or guidance on how to troubleshoot the jsonparsefailures, as I am seeing these on multiple log servers

[jolson]

As far as the JSONfailure is concerned, could you please pick one of the 'messages' fields out of a log and send it over to me? I'd like the raw format so I can use it to troubleshoot with on my end.

Have you upgraded or downgraded any of the built-in plugins? I imagine not, but I figured I would ask.

I'm also interested in seeing the configuration of the input/filter that is responsible for processing those Windows logs.

Thanks CFT6Server!