Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Manually Installing Log Server Issue

https://support.nagios.com/forum/viewtopic.php?t=33866

Page 2 of 3

Re: Manually Installing Log Server Issue

Posted: Thu Jul 23, 2015 9:50 am
by swilsongresh
I have just been through that, my node id was: b80d81b4-6d79-4a0c-ba18-9fe18da640e8 and as you suggested my name_name was blank in /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf. I have updated so that it now looks like:

output {
elasticsearch {
cluster => '3b70f83b-d36f-48ae-ba17-9f94d1d65244'
host => 'localhost'
document_type => '%{type}'
node_name => 'b80d81b4-6d79-4a0c-ba18-9fe18da640e8'
protocol => 'transport'
workers => 4
}
}

I restarted logstash but there were still no additional logs (which I assume I should start seeing?). To confirm though I re-ran the following on one of the target machines to monitor the syslogs:

curl -s -O http://192.168.150.209/nagioslogserver/ ... p-linux.sh
bash setup-linux.sh -s 192.168.150.209 -p 5544

Again this completed successfully but I am not seeing anything further in the UI, so I still can't see any new logs :(

Re: Manually Installing Log Server Issue

Posted: Thu Jul 23, 2015 10:00 am
by jolson
What concerns me the most is that when we showed your indices, we don't see any daily indices (which logs are stored in). Try accessing the 'Administration -> Index Status' screen. Are there any indices listed on this screen?

I'd like to see your logstash logs as well:

Code: Select all

tar zcf ~/logstashlogs.tar.gz /var/log/logstash/*

Re: Manually Installing Log Server Issue

Posted: Thu Jul 23, 2015 10:12 am
by swilsongresh
No there do not seem to be any indices showing:
Indices.JPG
That doesn't change whether I choose Open, Closed or Deleted.

Please see the requested logstash log files attached.

Re: Manually Installing Log Server Issue

Posted: Thu Jul 23, 2015 11:44 am
by jolson
From your Logstash logs:
Failed to install template: None of the configured nodes are available
This means that logstash cannot contact Elasticsearch appropriately. Let's ensure that ES is listening on port 9200/9300:

Code: Select all

netstat -na | egrep '9200|9300'
What does localhost resolve to?

Code: Select all

ping localhost
I am thinking that there could be some IPV6 binding which could be interfering with things. Either that or something might be blocking logstash from contacting localhost - but I doubt that.

Re: Manually Installing Log Server Issue

Posted: Fri Jul 24, 2015 4:09 am
by swilsongresh
A ping of localhost rseponds with:

64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.044 ms

The netstat command returns (to 192.168.150.209 is the servers address):

tcp 0 0 127.0.0.1:45667 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45659 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45687 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45681 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45684 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45680 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45672 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45669 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45679 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45689 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45692 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45662 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45696 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45660 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45674 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45695 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45677 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45697 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45682 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45663 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45683 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45688 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45673 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45676 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45665 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45664 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45700 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45694 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45698 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45670 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45686 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45671 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45691 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45678 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45675 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45685 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45658 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45666 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45661 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45690 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45701 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45699 127.0.0.1:9200 TIME_WAIT
tcp6 0 0 127.0.0.1:9200 :::* LISTEN
tcp6 0 0 192.168.150.209:9300 :::* LISTEN
tcp6 0 0 192.168.150.209:55967 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55968 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55965 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55973 ESTABLISHED
tcp6 0 0 192.168.150.209:55966 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55976 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55968 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55974 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55966 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55969 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55962 ESTABLISHED
tcp6 0 0 192.168.150.209:55969 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55976 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55970 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55967 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55965 ESTABLISHED
tcp6 0 0 192.168.150.209:55970 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55971 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55972 ESTABLISHED
tcp6 0 0 192.168.150.209:55962 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55975 ESTABLISHED
tcp6 0 0 192.168.150.209:55972 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55973 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55971 ESTABLISHED
tcp6 0 0 192.168.150.209:55974 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55975 192.168.150.209:9300 ESTABLISHED

On a side note, and in an attempt to get the logserver up and running I have run through the same process on CentOS 6.5 and all works, the install went through perfectly and I can monitor the same log files I was attempting to monitor on the Centos7 installation. Ideally I would like to stick with the centos7 install if I can though as going with 6.5 does feel a bit of a step backwards. So happy to work with both in parallel in order to get the centos7 version working.

Re: Manually Installing Log Server Issue

Posted: Fri Jul 24, 2015 9:11 am
by jolson
tcp6 0 0 127.0.0.1:9200 :::* LISTEN
tcp6 0 0 192.168.150.209:9300 :::* LISTEN
I have a suspicion that this is causing our problems.

Let's try disabling IPV6.

Put the following in /etc/sysctl.conf:

Code: Select all

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
Reload sysctl:

Code: Select all

sysctl -p
Restart relevant processes:

Code: Select all

service elasticsearch restart
service logstash restart
Let me know if that helps. Thanks!

Re: Manually Installing Log Server Issue

Posted: Fri Jul 24, 2015 9:58 am
by swilsongresh
Sadly that made no difference at all:

cat /etc/sysctl.conf:

# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
#

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

The netstat now returns:

tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN
tcp 0 0 192.168.150.209:9300 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:33815 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38380 ESTABLISHED
tcp 0 0 127.0.0.1:33817 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33829 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33793 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33798 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33830 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38379 192.168.150.209:9300 ESTABLISHED
tcp 0 0 192.168.150.209:9300 192.168.150.209:38386 ESTABLISHED
tcp 0 0 127.0.0.1:33820 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33791 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38379 ESTABLISHED
tcp 0 0 127.0.0.1:33807 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33806 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38382 ESTABLISHED
tcp 0 0 127.0.0.1:33813 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33812 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38374 ESTABLISHED
tcp 0 0 192.168.150.209:38377 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33795 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33797 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38382 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33839 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33810 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33796 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38378 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33811 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33833 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38381 ESTABLISHED
tcp 0 0 127.0.0.1:33821 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33804 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38376 ESTABLISHED
tcp 0 0 192.168.150.209:38386 192.168.150.209:9300 ESTABLISHED
tcp 0 0 192.168.150.209:38384 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33826 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38378 ESTABLISHED
tcp 0 0 127.0.0.1:33825 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38381 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33801 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38376 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33832 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33790 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38383 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33803 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38377 ESTABLISHED
tcp 0 0 127.0.0.1:33824 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33818 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33822 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33792 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38385 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33800 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33837 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38383 ESTABLISHED
tcp 0 0 127.0.0.1:33836 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33831 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38375 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33808 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38375 ESTABLISHED
tcp 0 0 127.0.0.1:33814 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33805 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33819 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33828 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33823 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38384 ESTABLISHED
tcp 0 0 127.0.0.1:33834 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38374 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33827 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33835 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33816 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38385 ESTABLISHED
tcp 0 0 127.0.0.1:33799 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33794 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33802 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33789 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33838 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38380 192.168.150.209:9300 ESTABLISHED

Re: Manually Installing Log Server Issue

Posted: Fri Jul 24, 2015 11:41 am
by jolson
When you performed your fresh CentOS 7 install, is it possible that you did not run a yum update prior to installing Nagios Log Server? I ask because the yum update is responsible for providing proper tools to map ipv4 -> ipv6. Let's try the following.

First, let's remove those ipv6 blocking rules from your sysctl.conf file.
remove:

Code: Select all

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
At this point, reboot your server.

Code: Select all

reboot
Now, let's run a yum update, rebooting afterwards so that we can boot into any potentially new kernel.

Code: Select all

yum update
reboot
Let me know if the above helps - this procedure worked for me on a test NLS node using CentOS 7.

Re: Manually Installing Log Server Issue

Posted: Mon Jul 27, 2015 7:54 am
by swilsongresh
It is very possible, I have just removed the the IPv6 blocking rules, run the system update and rebooted. To confirm:

[support@nagioslog ~]$ sudo yum update
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: centos.openitc.uk
* extras: mirrors.coreix.net
* updates: centos.hyve.com
No packages marked for update

Unfortunately I am still seeing the same error.

jolson - Thank you for all of your assistance with this, I am happy to keep working on this one if you would like to get to the bottom of this but due to this https://support.nagios.com/forum/viewto ... 38&t=33913 I do not think that Nagios Log monitor is going to quite be able to do what we require unfortunately. As such I am happy to put this down to bad OS configuration on my part initially. As I said when using CentOS 6.5 all worked as expected.

Re: Manually Installing Log Server Issue

Posted: Mon Jul 27, 2015 11:08 am
by jolson
Thank you for all of your input - below is my command history of my CentOS 7 box if you would like to install NLS on a CentOS 7 box moving forward. Ensure that SELinux is off before running through this procedure. (SELinux is disabled by the install script, but I like to ensure that it's off before using the fullinstall script).

Code: Select all

yum install -y wget
wget assets.nagios.com/downloads/nagios-log-server/nagioslogserver-latest.tar.gz
tar zxf nagioslogserver-latest.tar.gz
cd nagioslogserver
./fullinstall
yum upgrade
reboot
All times are UTC-05:00
Page 2 of 3

Powered by phpBB® Forum Software © phpBB Limited