Log Server stopped receiving logs

[GacoAdmin110]

I am having a problem with the Log Server test I am running in my cloud environment. All of the servers including the LS are on the same subnet.

1. It was running totally normal without any issues, and then yesterday the web console was stating that it could not start the Elasticsearch. I logged into the server console and ran the command to start that up, but it kept failing. I checked the disk usage, and there was still plenty of HDD left. I upgraded the CPU/RAM from 1/2GB to 2/4GB...still no luck. Eventually I just blew it away, and downloaded the updated version.

2. No I have built a new server with the updated OVA file. I configured it to use the same IP address as it was before. I copied the nxlog conf settihngs for adding Windows machines, and overwrote the old config. I made sure to have the nxlog service stopped while I did this, and started it up after. Still not getting any logs at the LS, and it doesnt even see that the servers are reporting to it.

3. So I tried changing the IP on the LS, recopying the nxlog conf settings, and testing that on one of my servers. I am still not getting any logs delivered.

This was running totally fine, and I still have many days left on my trial. Any help would be greatly appreciated!

Thnx

[jolson]

Are any logs showing up in the Nagios Log Server GUI?

If none are showing up, let's run a tcpdump to verify that logs are reaching Nagios Log Server at all:
On one of your instances that is receiving Windows logs via nxlog, run the following:
yum install tcpdump

Code: Select all

tcpdump -n dst port 3515 and host XXXX

Replace XXXX with the IP address of one of your Windows hosts that is supposed to be sending logs. Is any data arriving?

Ensure that core processes are running on Nagios Log Server:

Code: Select all

service elasticsearch status
service logstash status

Check the following log files for any errors:

Code: Select all

cat /var/log/logstash/logstash.log
tail -n200 /var/log/elasticsearch/*.log

Are your firewall ports open appropriately/allowed through AWS?

Code: Select all

iptables -L -n

[GacoAdmin110]

9-1-2015 1-37-33 PM.jpg

ElasticSearch & logstash are both running normal...attached IPtable image for ports

Do I need to add port 3515 to this since it isn't listed, and that's what nxlog is configured to use by default??

Logstash & elasticsearch both running normal.

[GacoAdmin110]

It definitely is receiving logs from the 1 server that I tested using the TCPdump util, but the web gui shows nothing showing up nor does it show any machines reporting to it.

IPtable is posted in this reply...not sure what it is supposed to look like to be honest. First timer here.

There is no firewall between these machines as they are all VM's in the same disk array/rack.

[jolson]

Do I need to add port 3515 to this since it isn't listed, and that's what nxlog is configured to use by default??

Unfortunately for us, port 3515 is listed - it's just tough to catch since it's wrapped around:

2015-09-01 14_00_24-9-1-2015 1-37-33 PM.jpg (JPEG Image, 884 × 641 pixels) - Firefox Developer Editi.png

Let's have a look at your logs:

Code: Select all

cat /var/log/logstash/logstash.log
tail -n200 /var/log/elasticsearch/*.log

[GacoAdmin110]

This thing is toying with me now...

Logs are showing up in the Dashboard of the web gui, but on the home screen of the gui it still only says that 1 host is reporting???

[GacoAdmin110]

Do I need to add port 3515 to this since it isn't listed, and that's what nxlog is configured to use by default??

Unfortunately for us, port 3515 is listed - it's just tough to catch since it's wrapped around:

2015-09-01 14_00_24-9-1-2015 1-37-33 PM.jpg (JPEG Image, 884 × 641 pixels) - Firefox Developer Editi.png

Let's have a look at your logs:

Code: Select all

cat /var/log/logstash/logstash.log
tail -n200 /var/log/elasticsearch/*.log

I ran both of these, but again...first time user so how do I get the file out of the LS server? Pardon me as I travel into the unknown.

[jolson]

Logs are showing up in the Dashboard of the web gui, but on the home screen of the gui it still only says that 1 host is reporting???

The host report isn't 100% accurate and may take some time to update - no need to be concerned in this regard.

I ran both of these, but again...first time user so how do I get the file out of the LS server? Pardon me as I travel into the unknown.

Assuming you are on Windows, I recommend downloading an SSH client - putty is the industry standard: http://the.earth.li/~sgtatham/putty/lat ... /putty.exe

After you download Putty, use it to remotely connect to your Nagios Log Server like so:

2015-09-01 14_48_25-Post a reply • Nagios Support Forum - Firefox Developer Edition.png

Replace '192.168.1.1' with the IP address of your Nagios Log Server instance.

Once connected, you will have a terminal open similar to the VMWare terminal that you've been taking screenshots of. The primary difference here is that you will be capable of copy+pasting text from Putty to notepad or similar.

Let me know!

[GacoAdmin110]

Logs are showing up in the Dashboard of the web gui, but on the home screen of the gui it still only says that 1 host is reporting???

The host report isn't 100% accurate and may take some time to update - no need to be concerned in this regard.

I ran both of these, but again...first time user so how do I get the file out of the LS server? Pardon me as I travel into the unknown.

Assuming you are on Windows, I recommend downloading an SSH client - putty is the industry standard: http://the.earth.li/~sgtatham/putty/lat ... /putty.exe

After you download Putty, use it to remotely connect to your Nagios Log Server like so:

2015-09-01 14_48_25-Post a reply • Nagios Support Forum - Firefox Developer Edition.png

Replace '192.168.1.1' with the IP address of your Nagios Log Server instance.

Once connected, you will have a terminal open similar to the VMWare terminal that you've been taking screenshots of. The primary difference here is that you will be capable of copy+pasting text from Putty to notepad or similar.

Let me know!

(facepalm...duh...sigh) Forgot about that way...winscp to the rescue worked good too! Had to install sshclients though on the server.

logstash.log

a70c9f7d-aa07-49ba-91d7-6be49acbf5c6.log

Here are the logs...

[jolson]

Does everything appear to be working from the Web GUI? I don't see any errors in your logs that are particularly alarming. The one I'm most concerned with is:

{:timestamp=>"2015-09-01T13:01:52.258000-0500", :message=>"An error occurred. Closing connection", :client=>"172.16.176.213:61931", :exception=>#<LogStash::ShutdownSignal: LogStash::ShutdownSignal>, :backtrace=>["org/jruby/RubyIO.javain `sysread'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:164:in `read'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:112:in `handle_socket'", "/usr/local/nagioslogserver/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/lib/logstash/inputs/tcp.rb:147:in `client_thread'"], :level=>:error}

However from the sounds of it Nagios Log Server is up and working now. Is that a true statement?