This support forum board is for support questions relating to
Nagios Log Server , our solution for managing and monitoring critical log data.
hsmith
Agent Smith
Posts: 3539 Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:
Post
by hsmith » Fri Apr 28, 2017 3:46 pm
dwhitfield wrote: What OS are the nodes running?
james.liew wrote: ... stops listening on the designated port we use for Windows hosts, say port 3500 and then refuses to receive any log traffic on said port. The Windows boxes run the nxlog agent.
Looks like Windows.
Former Nagios Employee.
me.
james.liew
Posts: 59 Joined: Wed Feb 22, 2017 1:30 am
Post
by james.liew » Mon May 01, 2017 9:46 pm
Windows Server 2012 R2
Sorry, haven't gotten to the traceroutes yet since I saw the newer posts.
james.liew
Posts: 59 Joined: Wed Feb 22, 2017 1:30 am
Post
by james.liew » Mon May 01, 2017 9:48 pm
Getting a "page requested cannot be found" error.
So... do I need to open ports 9200 and 9300 on the Windows machines too? And on the firewall between NLS and my Windows hosts?
EDIT: Starting to think that Log Server has an issue somewhere, I've had to reboot logstash and elasticsearch just this past weekend(on Saturday)
Uploaded logs from my logserver for Sunday morning. It again started to refuse connections to port 3515 around 8:13am in the morning.
I have a cron job setup at 00:00:00 Sunday to reboot logstash and elasticsearch.
Based on my first post re-copied below, am I already running the latest versions of Elasticsearch and Log Server?
Current NLG version:
Nagios Log Server: 1.4.4
Elasticsearch: 1.6.0
Logstash: 1.5.1
Kibana: 3.1.1-nagios3
You do not have the required permissions to view the files attached to this post.
mcapra
Posts: 3739 Joined: Thu May 05, 2016 3:54 pm
Post
by mcapra » Tue May 02, 2017 9:27 am
Can you send more recent Elasticsearch logs? The problem appears to be with Elasticsearch, but the Elasticsearch logs provided are a bit older than the Logstash logs provided so it's difficult to match the 2 up:
Logstash log start:
Elasticsearch log start:
james.liew
Posts: 59 Joined: Wed Feb 22, 2017 1:30 am
Post
by james.liew » Tue May 02, 2017 9:50 pm
Attachment #1
You do not have the required permissions to view the files attached to this post.
james.liew
Posts: 59 Joined: Wed Feb 22, 2017 1:30 am
Post
by james.liew » Tue May 02, 2017 9:51 pm
Uploaded the last 3-4 days of logs from log server.
You do not have the required permissions to view the files attached to this post.
cdienger
Support Tech
Posts: 5045 Joined: Tue Feb 07, 2017 11:26 am
Post
by cdienger » Wed May 03, 2017 10:49 am
The logs contain a lot of memory errors:
java.lang.OutOfMemoryError: unable to create new native thread
Elasticsearch doesn't have enough memory to function properly. Increase the amount of memory on the machine to avoid this problem.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new
Privacy Policy .
james.liew
Posts: 59 Joined: Wed Feb 22, 2017 1:30 am
Post
by james.liew » Thu May 04, 2017 10:24 pm
Checked the past RAM usage and I'm hitting around 80%-81% at peak before I have to do a restart of logstash and elasticsearch.
My log server currently has 8GB of RAM. I would assume as I configure nxlog.conf to add more logs I would need to add more RAM too?
cdienger
Support Tech
Posts: 5045 Joined: Tue Feb 07, 2017 11:26 am
Post
by cdienger » Fri May 05, 2017 9:52 am
Correct. Something to keep in mind as well is that Elasticsearch reserves half the total system memory for Java heap space so it effectively has only 4gigs on this system. You can increase system memory up to 64gigs giving Elasticsearch 32gigs to play with, but anything above 32gigs would actually hurt performance.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new
Privacy Policy .