Manually Installing Log Server Issue

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
swilsongresh
Posts: 14
Joined: Tue Jul 21, 2015 1:22 pm

Re: Manually Installing Log Server Issue

Post by swilsongresh »

I have just been through that, my node id was: b80d81b4-6d79-4a0c-ba18-9fe18da640e8 and as you suggested my name_name was blank in /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf. I have updated so that it now looks like:

output {
elasticsearch {
cluster => '3b70f83b-d36f-48ae-ba17-9f94d1d65244'
host => 'localhost'
document_type => '%{type}'
node_name => 'b80d81b4-6d79-4a0c-ba18-9fe18da640e8'
protocol => 'transport'
workers => 4
}
}

I restarted logstash but there were still no additional logs (which I assume I should start seeing?). To confirm though I re-ran the following on one of the target machines to monitor the syslogs:

curl -s -O http://192.168.150.209/nagioslogserver/ ... p-linux.sh
bash setup-linux.sh -s 192.168.150.209 -p 5544

Again this completed successfully but I am not seeing anything further in the UI, so I still can't see any new logs :(
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Manually Installing Log Server Issue

Post by jolson »

What concerns me the most is that when we showed your indices, we don't see any daily indices (which logs are stored in). Try accessing the 'Administration -> Index Status' screen. Are there any indices listed on this screen?

I'd like to see your logstash logs as well:

Code: Select all

tar zcf ~/logstashlogs.tar.gz /var/log/logstash/*
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
swilsongresh
Posts: 14
Joined: Tue Jul 21, 2015 1:22 pm

Re: Manually Installing Log Server Issue

Post by swilsongresh »

No there do not seem to be any indices showing:
Indices.JPG
That doesn't change whether I choose Open, Closed or Deleted.

Please see the requested logstash log files attached.
You do not have the required permissions to view the files attached to this post.
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Manually Installing Log Server Issue

Post by jolson »

From your Logstash logs:
Failed to install template: None of the configured nodes are available
This means that logstash cannot contact Elasticsearch appropriately. Let's ensure that ES is listening on port 9200/9300:

Code: Select all

netstat -na | egrep '9200|9300'
What does localhost resolve to?

Code: Select all

ping localhost
I am thinking that there could be some IPV6 binding which could be interfering with things. Either that or something might be blocking logstash from contacting localhost - but I doubt that.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
swilsongresh
Posts: 14
Joined: Tue Jul 21, 2015 1:22 pm

Re: Manually Installing Log Server Issue

Post by swilsongresh »

A ping of localhost rseponds with:

64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.044 ms

The netstat command returns (to 192.168.150.209 is the servers address):

tcp 0 0 127.0.0.1:45667 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45659 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45687 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45681 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45684 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45680 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45672 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45669 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45679 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45689 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45692 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45662 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45696 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45660 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45674 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45695 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45677 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45697 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45682 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45663 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45683 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45688 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45673 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45676 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45665 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45664 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45700 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45694 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45698 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45670 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45686 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45671 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45691 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45678 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45675 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45685 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45658 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45666 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45661 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45690 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45701 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:45699 127.0.0.1:9200 TIME_WAIT
tcp6 0 0 127.0.0.1:9200 :::* LISTEN
tcp6 0 0 192.168.150.209:9300 :::* LISTEN
tcp6 0 0 192.168.150.209:55967 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55968 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55965 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55973 ESTABLISHED
tcp6 0 0 192.168.150.209:55966 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55976 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55968 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55974 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55966 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55969 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55962 ESTABLISHED
tcp6 0 0 192.168.150.209:55969 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55976 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55970 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55967 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55965 ESTABLISHED
tcp6 0 0 192.168.150.209:55970 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55971 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55972 ESTABLISHED
tcp6 0 0 192.168.150.209:55962 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55975 ESTABLISHED
tcp6 0 0 192.168.150.209:55972 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55973 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:9300 192.168.150.209:55971 ESTABLISHED
tcp6 0 0 192.168.150.209:55974 192.168.150.209:9300 ESTABLISHED
tcp6 0 0 192.168.150.209:55975 192.168.150.209:9300 ESTABLISHED

On a side note, and in an attempt to get the logserver up and running I have run through the same process on CentOS 6.5 and all works, the install went through perfectly and I can monitor the same log files I was attempting to monitor on the Centos7 installation. Ideally I would like to stick with the centos7 install if I can though as going with 6.5 does feel a bit of a step backwards. So happy to work with both in parallel in order to get the centos7 version working.
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Manually Installing Log Server Issue

Post by jolson »

tcp6 0 0 127.0.0.1:9200 :::* LISTEN
tcp6 0 0 192.168.150.209:9300 :::* LISTEN
I have a suspicion that this is causing our problems.

Let's try disabling IPV6.

Put the following in /etc/sysctl.conf:

Code: Select all

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
Reload sysctl:

Code: Select all

sysctl -p
Restart relevant processes:

Code: Select all

service elasticsearch restart
service logstash restart
Let me know if that helps. Thanks!
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
swilsongresh
Posts: 14
Joined: Tue Jul 21, 2015 1:22 pm

Re: Manually Installing Log Server Issue

Post by swilsongresh »

Sadly that made no difference at all:

cat /etc/sysctl.conf:

# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
#

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

The netstat now returns:

tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN
tcp 0 0 192.168.150.209:9300 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:33815 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38380 ESTABLISHED
tcp 0 0 127.0.0.1:33817 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33829 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33793 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33798 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33830 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38379 192.168.150.209:9300 ESTABLISHED
tcp 0 0 192.168.150.209:9300 192.168.150.209:38386 ESTABLISHED
tcp 0 0 127.0.0.1:33820 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33791 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38379 ESTABLISHED
tcp 0 0 127.0.0.1:33807 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33806 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38382 ESTABLISHED
tcp 0 0 127.0.0.1:33813 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33812 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38374 ESTABLISHED
tcp 0 0 192.168.150.209:38377 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33795 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33797 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38382 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33839 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33810 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33796 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38378 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33811 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33833 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38381 ESTABLISHED
tcp 0 0 127.0.0.1:33821 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33804 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38376 ESTABLISHED
tcp 0 0 192.168.150.209:38386 192.168.150.209:9300 ESTABLISHED
tcp 0 0 192.168.150.209:38384 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33826 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38378 ESTABLISHED
tcp 0 0 127.0.0.1:33825 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38381 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33801 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38376 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33832 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33790 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38383 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33803 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38377 ESTABLISHED
tcp 0 0 127.0.0.1:33824 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33818 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33822 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33792 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38385 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33800 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33837 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38383 ESTABLISHED
tcp 0 0 127.0.0.1:33836 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33831 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38375 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33808 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38375 ESTABLISHED
tcp 0 0 127.0.0.1:33814 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33805 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33819 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33828 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33823 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38384 ESTABLISHED
tcp 0 0 127.0.0.1:33834 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38374 192.168.150.209:9300 ESTABLISHED
tcp 0 0 127.0.0.1:33827 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33835 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33816 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:9300 192.168.150.209:38385 ESTABLISHED
tcp 0 0 127.0.0.1:33799 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33794 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33802 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33789 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:33838 127.0.0.1:9200 TIME_WAIT
tcp 0 0 192.168.150.209:38380 192.168.150.209:9300 ESTABLISHED
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Manually Installing Log Server Issue

Post by jolson »

When you performed your fresh CentOS 7 install, is it possible that you did not run a yum update prior to installing Nagios Log Server? I ask because the yum update is responsible for providing proper tools to map ipv4 -> ipv6. Let's try the following.

First, let's remove those ipv6 blocking rules from your sysctl.conf file.
remove:

Code: Select all

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
At this point, reboot your server.

Code: Select all

reboot
Now, let's run a yum update, rebooting afterwards so that we can boot into any potentially new kernel.

Code: Select all

yum update
reboot
Let me know if the above helps - this procedure worked for me on a test NLS node using CentOS 7.
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
swilsongresh
Posts: 14
Joined: Tue Jul 21, 2015 1:22 pm

Re: Manually Installing Log Server Issue

Post by swilsongresh »

It is very possible, I have just removed the the IPv6 blocking rules, run the system update and rebooted. To confirm:

[support@nagioslog ~]$ sudo yum update
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: centos.openitc.uk
* extras: mirrors.coreix.net
* updates: centos.hyve.com
No packages marked for update

Unfortunately I am still seeing the same error.

jolson - Thank you for all of your assistance with this, I am happy to keep working on this one if you would like to get to the bottom of this but due to this https://support.nagios.com/forum/viewto ... 38&t=33913 I do not think that Nagios Log monitor is going to quite be able to do what we require unfortunately. As such I am happy to put this down to bad OS configuration on my part initially. As I said when using CentOS 6.5 all worked as expected.
Top
jolson
Attack Rabbit
Posts: 2560
Joined: Thu Feb 12, 2015 12:40 pm

Re: Manually Installing Log Server Issue

Post by jolson »

Thank you for all of your input - below is my command history of my CentOS 7 box if you would like to install NLS on a CentOS 7 box moving forward. Ensure that SELinux is off before running through this procedure. (SELinux is disabled by the install script, but I like to ensure that it's off before using the fullinstall script).

Code: Select all

yum install -y wget
wget assets.nagios.com/downloads/nagios-log-server/nagioslogserver-latest.tar.gz
tar zxf nagioslogserver-latest.tar.gz
cd nagioslogserver
./fullinstall
yum upgrade
reboot
Twits Blog
Show me a man who lives alone and has a perpetually clean kitchen, and 8 times out of 9 I'll show you a man with detestable spiritual qualities.
Top
Locked

Return to “Nagios Log Server”