OOM Killer Struck: before I had a chance to start using NLS

[chrisp]

Hi,

I deployed the Nagios Log Server yesterday. I got distracted and had to focus on another task, but when I came back to the VM, I could see that even without any actual usage (no syslog data being pointed at it and never even logged in to it), OOM Killer had started to kill processes (it chose java).







It's made be a bit shy of continuing to test it out. Is this "OK", or do I need to do something with my setup?

This VM will (hopefully) be replacing an existing SysLog server that doesn't get much love. The resources are minimal, as we're not exactly stressing it out with work.

Cheers,
--
ChrisP

[tmcdonald]

How was this set up initially? We have a pre-made VM appliance, or it can be installed from our source installer.

Also, what version of NLS are you running?

[jolson]

It's made be a bit shy of continuing to test it out. Is this "OK", or do I need to do something with my setup?

Are you certain that system logs weren't being sent to the IP address of Nagios Log Server before you stood it up? Typically 4GB is enough RAM to get started - but you might need to move to 8GB just to be safe. Nagios Log Server is by nature very RAM-heavy, and needs some room to breathe/build up a cache. Give 8GB a shot and let me know if you still have problems - 8 should be enough to add many hosts to and likely complete your evaluation of the product.

After you add the memory, run the following commands on the CLI:
service elasticsearch restart
service logstash restart

Best,

Jesse

[chrisp]

Thanks for your responses,

How was this set up initially? We have a pre-made VM appliance... what version of NLS are you running?

@tmcdonald, it's the VA I just deployed - latest version.

Are you certain that system logs weren't being sent to the IP address of Nagios Log Server before you stood it up?

@jolson, positive. Our systems send to UDP:514 and to our old SysLog server - the NLS was on a dynamic IP after deployment and had not been logged in to, or even completely configured hia HTTP.

I have now given the NLS 8GB RAM, shut down the old SysLog server, configured NLS with the IP of the old SysLog server, tweaked our Cisco routers to log to UDP:5544 and the packet/port redirection on our routers, to redirect U:514 => U:5544 (can't be bothered reconfiguring every device - some which cannot be configured away from U:514 - especially when I can just massage a port-redirection).

The NLS is now looking like this: -

Code: Select all

[root@hq-syslog ~]# top -n1 | head 
top - 13:28:06 up  2:58,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 184 total,   2 running, 182 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  0.2%sy,  0.2%ni, 99.1%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8193024k total,  5491352k used,  2701672k free,    39144k buffers
Swap:   262136k total,        0k used,   262136k free,   239220k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15853 nagios    39  19 5287m 396m  14m S  2.0  5.0   0:50.87 java
    1 root      20   0 19232 1524 1232 S  0.0  0.0   0:02.07 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd

No wonder it ran out of 4GB RAM, with "Mem: 8193024k total, 5491352k used"

I can continue on with my investigation, though the usual distractions will likely keep me from proper eval.

Cheers,
--
ChrisP

[hsmith]

Let us know if you have any further questions! I hope you like the product.