Hosts dropped to 1

[itbilling]

After increasing the Virtual Hard Drive size on the Log Server, my hosts have dropped to one and it's not picking any hosts up. I wasn't sure what logs you need to troubleshoot the issue, or if this is a known issue.

[cmerchant]

Just to check what source you are still getting?

From the home page, under the Global dashboards, click on the Top sources and Types link to find what host is still being collected. (I'm guessing just the local host is being collected).

Also, go to the Administration page, click on System Status, you should see green checks on Elasticsearch Database, and Logstash Collector. Restart the services if necessary.

[itbilling]

The Top Alert Producers is blank, events over time shows 0 of 0.

I have restarted the services with the same issue. One thing I did find is Cluster Health Status says "Red". Number of documents under indices also shows 0. I have deleted a couple of the previous days just to make sure the space wasn't still an issue, and no change. It shows correctly the space we have added now and is at 97% free. Anything else I can check to see why it is not collecting any logs?

[cmerchant]

You should delete the index for the one that shows zero. It's possible that it failed to allocate the newest one that was from being out of space. Logserver will automatically recreate an index for today. Did you increase the size on each server in the cluster?

[itbilling]

We are running only one server, I deleted every one that stated 0, it created a new one, but it is still at 0, and has 0 hosts showing up. the count is still at 1, but nothing shows up in the top alerts list.

[itbilling]

Any other ideas? It has yet to add any more documents or show any hosts in the log server.

[scottwilkerson]

Can we forcibly restart logstash

Code: Select all

service logstash restart

Then, can we tail the logstash log file for a minute and return any results

Code: Select all

tail -f /var/log/logstash/logstash.log

[itbilling]

here is the result:

{:timestamp=>"2015-01-05T15:48:05.917000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000Z\\u0000\\u0000\\u0001\\u0000\\u0000\\u0000\\u00016\\u0001,\\u0000\\u0000\\b\\u0000\\u007F\\xFF\\u007F\\b\\u0000\\u0000\\u0000\\u0001\\u0000 \\u0000:\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u00004\\xE6\\u0000\\u0000\\u0000\\u0001\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000(CONNECT_DATA=(COMMAND=version))", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:48:16.086000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x94\\u0000\\xCD\\xEF\\xD1a\\x91\\u0003", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:49:00.113000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x80\\u0000\\u0000(*c\\xF6\\xB8\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0002\\u0000\\u0001\\x86\\xA0\\u0000\\u0000\\u0000\\u0002\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:00.379000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x80\\u0000\\u0000(r\\xFE\\u001D\\u0013\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0002\\u0000\\u0001\\x86\\xA0\\u0000\\u0001\\x97|\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:18.368000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0016\\u0003\\u0000\\u0000S\\u0001\\u0000\\u0000O\\u0003\\u0000?G\\xD7\\xF7\\xBA,\\xEE\\xEA\\xB2`~\\xF3\\u0000\\xFD\\x82{\\xB9Ֆ\\xC8w\\x9B\\xE6\\xC4\\xDB<=\\xDBo\\xEF\\u0010n\\u0000\\u0000(\\u0000\\u0016\\u0000\\u0013\\u0000\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:23.409000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000\\u0000\\u0000qj\\x81n0\\x81k\\xA1\\u0003\\u0002\\u0001\\u0005\\xA2\\u0003\\u0002\\u0001\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:28.436000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\xA4\\x81^0\\\\\\xA0\\a\\u0003\\u0005\\u0000P\\x80\\u0000\\u0010\\xA2\\u0004\\e\\u0002NM\\xA3\\u00170\\u0015\\xA0\\u0003\\u0002\\u0001\\u0000\\xA1\\u000E0\\f\\e\\u0006krbtgt\\e\\u0002NM\\xA5\\u0011\\u0018\\u000F19700101000000Z\\xA7\\u0006\\u0002\\u0004\\u001F\\u001E\\xB9٨\\u00170\\u0015\\u0002\\u0001\\u0012\\u0002\\u0001\\u0011\\u0002\\u0001\\u0010\\u0002\\u0001\\u0017\\u0002\\u0001\\u0001\\u0002\\u0001\\u0003\\u0002\\u0001\\u0002", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:33.461000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000\\u0000\\u0000\\xA4\\xFFSMBr\\u0000\\u0000\\u0000\\u0000\\b\\u0001@\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000@\\u0006\\u0000\\u0000\\u0001\\u0000\\u0000\\x81\\u0000\\u0002PC NETWORK PROGRAM 1.0\\u0000\\u0002MICROSOFT NETWORKS 1.03\\u0000\\u0002MICROSOFT NETWORKS 3.0\\u0000\\u0002LANMAN1.0\\u0000\\u0002LM1.2X002\\u0000\\u0002Samba\\u0000\\u0002NT LANMAN 1.0\\u0000\\u0002NT LM 0.12\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:53.819000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"0\\f\\u0002\\u0001\\u0001`\\a\\u0002\\u0001\\u0002\\u0004\\u0000\\x80\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:51:11.345000-0500", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.javain `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}

[sreinhardt]

Looks like you are definitely getting some incorrect character encoding warnings, but that should not cause the crash. This guy, seems to be the culprit:

{:timestamp=>"2015-01-05T15:51:11.345000-0500", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.javain `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}

This is for inputs, so logstash should not be connecting to other hosts, but opening up listening sockets. Lets try checking the existing open ports, shutting down logstash, checking them again, and finally starting it back up:

Code: Select all

netstat -naop | grep 5544
ps -ef | grep logstash
service logstash stop
netstat -nao | grep 5544
ps -ef | grep logstash
service logstash start
netstat -naop | grep 5544
ps -ef | grep logstash

[itbilling]

(different person, Brian Richins (Security Architect))
It appears the issue is related more to the logstash itself. Running an NMap on the server we always see the 3515/TCP shutdown. This is the port that all the nxlog clients are sending on, correct?

Scanning atclogserver (172.16.16.128) [1000 ports]
Discovered open port 80/tcp on 172.16.16.128
Discovered open port 22/tcp on 172.16.16.128
Discovered open port 5544/tcp on 172.16.16.128

Restarting the Logstash services (or the server for that matter) does not appear to help.