Page 1 of 2
nxlog memory leak?
Posted: Mon Jan 18, 2016 2:00 pm
by CFT6Server
I understand Nxlog is not under Nagios support, but we are using the version provided by the log server and looks like there could be memory leak issues? I am getting the following messaging in Nxlog resulting in events not coming in.
Code: Select all
2016-01-14 17:35:04 ERROR EvtUpdateBookmark failed: The handle is invalid.
2016-01-14 17:35:04 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.
2016-01-14 17:35:04 ERROR EvtUpdateBookmark failed: The handle is invalid.
2016-01-14 17:35:04 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.
2016-01-14 17:35:04 ERROR EvtCreateBookmark failed: Not enough storage is available to complete this operation.
2016-01-14 17:35:04 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.
2016-01-14 17:35:04 ERROR EvtCreateBookmark failed: Not enough storage is available to complete this operation.
2016-01-14 17:35:04 ERROR EvtCreateRenderContext failed; Not enough storage is available to complete this operation.
The server has plenty of space, and I am not seeing warnings or errors on the buffer which usually gets logged. Wondering if other nxlog/Nagios log server users have reported seeing this?
Re: nxlog memory leak?
Posted: Mon Jan 18, 2016 2:11 pm
by tmcdonald
I was able to find more reports of memory leaks from mid-2015:
https://nxlog.co/question/802/nxlog-ce-memory-leak
They said they have identified the cause and it will be fixed in the next release. I am not able to find a changelog, but if you download the latest version it may contain the fix.
Re: nxlog memory leak?
Posted: Mon Jan 18, 2016 2:24 pm
by CFT6Server
That's what I found as well and cannot find changelogs. I have updated to CE 2.9.1504, hopefully it has the fix.
Re: nxlog memory leak?
Posted: Mon Jan 18, 2016 3:09 pm
by rkennedy
I'll leave this thread open - will you post back and let us know the result once it's ran for a little bit?
Re: nxlog memory leak?
Posted: Fri Jan 29, 2016 1:18 pm
by CFT6Server
I am continuing to have issues daily with nxlog and having to restart it. I've given the server more RAM and increased the buffer in the configuration, but no luck.
Looks like it dies under half a day.
Code: Select all
2016-01-28 17:15:06 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.
2016-01-28 17:15:06 ERROR EvtUpdateBookmark failed: The handle is invalid.
2016-01-28 17:15:06 ERROR EvtNext failed with error 14: Not enough storage is available to complete this operation.
2016-01-28 17:15:06 ERROR EvtUpdateBookmark failed: The handle is invalid.
2016-01-28 17:15:06 ERROR EvtCreateRenderContext failed; Not enough storage is available to complete this operation.
and then eventually just stops all together. I am thinking nxlog cannot keep up with the amount of incoming logs on this windows event collector.
Re: nxlog memory leak?
Posted: Fri Jan 29, 2016 2:33 pm
by hsmith
How much information is coming in? Have you reported the latest finding on their website?
Re: nxlog memory leak?
Posted: Mon Feb 01, 2016 1:08 pm
by CFT6Server
There's a lot of events coming in via the collector. Over 400 to 500 servers that are forwarded here and then nxlog is picking this up.
I have no put up a post of the community forum on their site...
Looks like I cannot get more than about half a day's worth until it crashes. Today, it just refused to work all together even after restarting nxlog. I started seeing 404 in the logstash logs as well. So I restarted logstash and now the logs are all coming in like the following...
logs.JPG
Re: nxlog memory leak?
Posted: Mon Feb 01, 2016 1:42 pm
by CFT6Server
Update: Looks like this is most likely nxlog, as I rebooted the Windows Event Collector and it is fine. Restart the nxlog service did not help.
Re: nxlog memory leak?
Posted: Mon Feb 01, 2016 3:07 pm
by rkennedy
I'd like to try and replicate this over here. Can you post your nxlog configuration, and exact version of Windows you're running?
Re: nxlog memory leak?
Posted: Tue Feb 02, 2016 7:26 am
by WillemDH
I haven't seen any issues with my NxLog version 2.9.1347. Started updating to version 2.9.1504 today. Some of the servers where NxLog is running are sending a lot of logs to my NLS nodes. I have seen some domain controllers send more then 1000 logs per second. Check screenshot. This dc in the screenshot is physical and has 32 GB RAM. But other dc's which are virtual and only have 6 GB RAM also sent up to 1500 logs / sec and never had any memory related issue.
These are all 2008 R2 Windows servers. Hope this info helps you somehow. Grtz. Willem