Nagios Log Server listening port abruptly halts

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:

Re: Nagios Log Server listening port abruptly halts

Post by hsmith »

dwhitfield wrote:What OS are the nodes running?
james.liew wrote:... stops listening on the designated port we use for Windows hosts, say port 3500 and then refuses to receive any log traffic on said port. The Windows boxes run the nxlog agent.
Looks like Windows.
Former Nagios Employee.
me.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Nagios Log Server listening port abruptly halts

Post by dwhitfield »

hsmith wrote: Looks like Windows.
Of course! I meant what version of, I'm assuming, Server. I hope https://technet.microsoft.com/en-us/lib ... s.10).aspx points the OP in the right direction.
james.liew
Posts: 59
Joined: Wed Feb 22, 2017 1:30 am

Re: Nagios Log Server listening port abruptly halts

Post by james.liew »

Windows Server 2012 R2

Sorry, haven't gotten to the traceroutes yet since I saw the newer posts.
james.liew
Posts: 59
Joined: Wed Feb 22, 2017 1:30 am

Re: Nagios Log Server listening port abruptly halts

Post by james.liew »

dwhitfield wrote:
hsmith wrote: Looks like Windows.
Of course! I meant what version of, I'm assuming, Server. I hope https://technet.microsoft.com/en-us/lib ... s.10).aspx points the OP in the right direction.
Getting a "page requested cannot be found" error.

So... do I need to open ports 9200 and 9300 on the Windows machines too? And on the firewall between NLS and my Windows hosts?

EDIT: Starting to think that Log Server has an issue somewhere, I've had to reboot logstash and elasticsearch just this past weekend(on Saturday)

Uploaded logs from my logserver for Sunday morning. It again started to refuse connections to port 3515 around 8:13am in the morning.

I have a cron job setup at 00:00:00 Sunday to reboot logstash and elasticsearch.

Based on my first post re-copied below, am I already running the latest versions of Elasticsearch and Log Server?

Current NLG version:
Nagios Log Server: 1.4.4
Elasticsearch: 1.6.0
Logstash: 1.5.1
Kibana: 3.1.1-nagios3
You do not have the required permissions to view the files attached to this post.
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Nagios Log Server listening port abruptly halts

Post by mcapra »

Can you send more recent Elasticsearch logs? The problem appears to be with Elasticsearch, but the Elasticsearch logs provided are a bit older than the Logstash logs provided so it's difficult to match the 2 up:

Logstash log start:

Code: Select all

2017-04-30T08:03:25.601000+0200

Elasticsearch log start:

Code: Select all

[2017-04-29 17:44:24,187]
Former Nagios employee
https://www.mcapra.com/
james.liew
Posts: 59
Joined: Wed Feb 22, 2017 1:30 am

Re: Nagios Log Server listening port abruptly halts

Post by james.liew »

Attachment #1
You do not have the required permissions to view the files attached to this post.
james.liew
Posts: 59
Joined: Wed Feb 22, 2017 1:30 am

Re: Nagios Log Server listening port abruptly halts

Post by james.liew »

Uploaded the last 3-4 days of logs from log server.
You do not have the required permissions to view the files attached to this post.
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios Log Server listening port abruptly halts

Post by cdienger »

The logs contain a lot of memory errors:

java.lang.OutOfMemoryError: unable to create new native thread

Elasticsearch doesn't have enough memory to function properly. Increase the amount of memory on the machine to avoid this problem.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
james.liew
Posts: 59
Joined: Wed Feb 22, 2017 1:30 am

Re: Nagios Log Server listening port abruptly halts

Post by james.liew »

Checked the past RAM usage and I'm hitting around 80%-81% at peak before I have to do a restart of logstash and elasticsearch.

My log server currently has 8GB of RAM. I would assume as I configure nxlog.conf to add more logs I would need to add more RAM too?
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios Log Server listening port abruptly halts

Post by cdienger »

Correct. Something to keep in mind as well is that Elasticsearch reserves half the total system memory for Java heap space so it effectively has only 4gigs on this system. You can increase system memory up to 64gigs giving Elasticsearch 32gigs to play with, but anything above 32gigs would actually hurt performance.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Locked