No new indices being created

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
CameronWP
Posts: 134
Joined: Fri Apr 17, 2015 2:17 pm

No new indices being created

Post by CameronWP »

Hello:

I recently migrated my Nagios Log server to a new server and had some issues but thought I had everything working. My ports are listening, my services are started and seemingly happy, my logstash log is empty, I have 5TB of space for logs and I can see that all of my hosts are reporting in based on the number of hosts indication on the home screen. Despite all of this, there are no events actually being captured and there are no new indices. I am at a loss as to what to check next.

Thanks in advance!
CameronWP
Posts: 134
Joined: Fri Apr 17, 2015 2:17 pm

Re: No new indices being created

Post by CameronWP »

Ok, I am seeing this on a number of my hosts in the nxlog.log:

ERROR couldn't connect to tcp socket on 10.1.151.34:3515; No connection could be made because the target machine actively refused it.

Thanks!
CameronWP
Posts: 134
Joined: Fri Apr 17, 2015 2:17 pm

Re: No new indices being created

Post by CameronWP »

FYI also:

sestatus
SELinux status: disabled
service firewalld status
Redirecting to /bin/systemctl status firewalld.service
● firewalld.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: No new indices being created

Post by rkennedy »

This sounds like a firewall issue somewhere. From another machine, what is the output of nmap 10.1.151.34?

Is iptables running by chance? Depending on the host on where you transferred, there could be another firewall in between that needs to be opened up.

Lastly, from the NLS machine, what is the full output of netstat -anp
Former Nagios Employee
CameronWP
Posts: 134
Joined: Fri Apr 17, 2015 2:17 pm

Re: No new indices being created

Post by CameronWP »

Thanks for the reply! I think I got to the bottom of it, I was looking in the wrong place. My heap size was an issue for the logstash apparently. I set it to 1024M and restarted the service and it stopped crashing (that was the root cause of the other issues IO was having).

Thanks!
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: No new indices being created

Post by mcapra »

Awesome! If a Logstash instance is receiving a lot of traffic on startup, I can see where the heap might be overwhelmed.

Did you have additional questions regarding this issue, or can we close the thread and mark it as resolved?
Former Nagios employee
https://www.mcapra.com/
Locked