Hosts dropped to 1
Hosts dropped to 1
After increasing the Virtual Hard Drive size on the Log Server, my hosts have dropped to one and it's not picking any hosts up. I wasn't sure what logs you need to troubleshoot the issue, or if this is a known issue.
Re: Hosts dropped to 1
Just to check what source you are still getting?
From the home page, under the Global dashboards, click on the Top sources and Types link to find what host is still being collected. (I'm guessing just the local host is being collected).
Also, go to the Administration page, click on System Status, you should see green checks on Elasticsearch Database, and Logstash Collector. Restart the services if necessary.
From the home page, under the Global dashboards, click on the Top sources and Types link to find what host is still being collected. (I'm guessing just the local host is being collected).
Also, go to the Administration page, click on System Status, you should see green checks on Elasticsearch Database, and Logstash Collector. Restart the services if necessary.
Re: Hosts dropped to 1
The Top Alert Producers is blank, events over time shows 0 of 0.
I have restarted the services with the same issue. One thing I did find is Cluster Health Status says "Red". Number of documents under indices also shows 0. I have deleted a couple of the previous days just to make sure the space wasn't still an issue, and no change. It shows correctly the space we have added now and is at 97% free. Anything else I can check to see why it is not collecting any logs?
I have restarted the services with the same issue. One thing I did find is Cluster Health Status says "Red". Number of documents under indices also shows 0. I have deleted a couple of the previous days just to make sure the space wasn't still an issue, and no change. It shows correctly the space we have added now and is at 97% free. Anything else I can check to see why it is not collecting any logs?
Re: Hosts dropped to 1
You should delete the index for the one that shows zero. It's possible that it failed to allocate the newest one that was from being out of space. Logserver will automatically recreate an index for today. Did you increase the size on each server in the cluster?
Re: Hosts dropped to 1
We are running only one server, I deleted every one that stated 0, it created a new one, but it is still at 0, and has 0 hosts showing up. the count is still at 1, but nothing shows up in the top alerts list.
Re: Hosts dropped to 1
Any other ideas? It has yet to add any more documents or show any hosts in the log server.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Hosts dropped to 1
Can we forcibly restart logstash
Then, can we tail the logstash log file for a minute and return any results
Code: Select all
service logstash restart
Code: Select all
tail -f /var/log/logstash/logstash.log
Re: Hosts dropped to 1
here is the result:
{:timestamp=>"2015-01-05T15:48:05.917000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000Z\\u0000\\u0000\\u0001\\u0000\\u0000\\u0000\\u00016\\u0001,\\u0000\\u0000\\b\\u0000\\u007F\\xFF\\u007F\\b\\u0000\\u0000\\u0000\\u0001\\u0000 \\u0000:\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u00004\\xE6\\u0000\\u0000\\u0000\\u0001\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000(CONNECT_DATA=(COMMAND=version))", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:48:16.086000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x94\\u0000\\xCD\\xEF\\xD1a\\x91\\u0003", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:49:00.113000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x80\\u0000\\u0000(*c\\xF6\\xB8\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0002\\u0000\\u0001\\x86\\xA0\\u0000\\u0000\\u0000\\u0002\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:00.379000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x80\\u0000\\u0000(r\\xFE\\u001D\\u0013\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0002\\u0000\\u0001\\x86\\xA0\\u0000\\u0001\\x97|\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:18.368000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0016\\u0003\\u0000\\u0000S\\u0001\\u0000\\u0000O\\u0003\\u0000?G\\xD7\\xF7\\xBA,\\xEE\\xEA\\xB2`~\\xF3\\u0000\\xFD\\x82{\\xB9Ֆ\\xC8w\\x9B\\xE6\\xC4\\xDB<=\\xDBo\\xEF\\u0010n\\u0000\\u0000(\\u0000\\u0016\\u0000\\u0013\\u0000\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:23.409000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000\\u0000\\u0000qj\\x81n0\\x81k\\xA1\\u0003\\u0002\\u0001\\u0005\\xA2\\u0003\\u0002\\u0001\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:28.436000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\xA4\\x81^0\\\\\\xA0\\a\\u0003\\u0005\\u0000P\\x80\\u0000\\u0010\\xA2\\u0004\\e\\u0002NM\\xA3\\u00170\\u0015\\xA0\\u0003\\u0002\\u0001\\u0000\\xA1\\u000E0\\f\\e\\u0006krbtgt\\e\\u0002NM\\xA5\\u0011\\u0018\\u000F19700101000000Z\\xA7\\u0006\\u0002\\u0004\\u001F\\u001E\\xB9٨\\u00170\\u0015\\u0002\\u0001\\u0012\\u0002\\u0001\\u0011\\u0002\\u0001\\u0010\\u0002\\u0001\\u0017\\u0002\\u0001\\u0001\\u0002\\u0001\\u0003\\u0002\\u0001\\u0002", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:33.461000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000\\u0000\\u0000\\xA4\\xFFSMBr\\u0000\\u0000\\u0000\\u0000\\b\\u0001@\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000@\\u0006\\u0000\\u0000\\u0001\\u0000\\u0000\\x81\\u0000\\u0002PC NETWORK PROGRAM 1.0\\u0000\\u0002MICROSOFT NETWORKS 1.03\\u0000\\u0002MICROSOFT NETWORKS 3.0\\u0000\\u0002LANMAN1.0\\u0000\\u0002LM1.2X002\\u0000\\u0002Samba\\u0000\\u0002NT LANMAN 1.0\\u0000\\u0002NT LM 0.12\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:53.819000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"0\\f\\u0002\\u0001\\u0001`\\a\\u0002\\u0001\\u0002\\u0004\\u0000\\x80\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:51:11.345000-0500", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.javain `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}
{:timestamp=>"2015-01-05T15:48:05.917000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000Z\\u0000\\u0000\\u0001\\u0000\\u0000\\u0000\\u00016\\u0001,\\u0000\\u0000\\b\\u0000\\u007F\\xFF\\u007F\\b\\u0000\\u0000\\u0000\\u0001\\u0000 \\u0000:\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u00004\\xE6\\u0000\\u0000\\u0000\\u0001\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000(CONNECT_DATA=(COMMAND=version))", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:48:16.086000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x94\\u0000\\xCD\\xEF\\xD1a\\x91\\u0003", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:49:00.113000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x80\\u0000\\u0000(*c\\xF6\\xB8\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0002\\u0000\\u0001\\x86\\xA0\\u0000\\u0000\\u0000\\u0002\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:00.379000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\x80\\u0000\\u0000(r\\xFE\\u001D\\u0013\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0002\\u0000\\u0001\\x86\\xA0\\u0000\\u0001\\x97|\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:18.368000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0016\\u0003\\u0000\\u0000S\\u0001\\u0000\\u0000O\\u0003\\u0000?G\\xD7\\xF7\\xBA,\\xEE\\xEA\\xB2`~\\xF3\\u0000\\xFD\\x82{\\xB9Ֆ\\xC8w\\x9B\\xE6\\xC4\\xDB<=\\xDBo\\xEF\\u0010n\\u0000\\u0000(\\u0000\\u0016\\u0000\\u0013\\u0000\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:23.409000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000\\u0000\\u0000qj\\x81n0\\x81k\\xA1\\u0003\\u0002\\u0001\\u0005\\xA2\\u0003\\u0002\\u0001\\n", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:28.436000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\xA4\\x81^0\\\\\\xA0\\a\\u0003\\u0005\\u0000P\\x80\\u0000\\u0010\\xA2\\u0004\\e\\u0002NM\\xA3\\u00170\\u0015\\xA0\\u0003\\u0002\\u0001\\u0000\\xA1\\u000E0\\f\\e\\u0006krbtgt\\e\\u0002NM\\xA5\\u0011\\u0018\\u000F19700101000000Z\\xA7\\u0006\\u0002\\u0004\\u001F\\u001E\\xB9٨\\u00170\\u0015\\u0002\\u0001\\u0012\\u0002\\u0001\\u0011\\u0002\\u0001\\u0010\\u0002\\u0001\\u0017\\u0002\\u0001\\u0001\\u0002\\u0001\\u0003\\u0002\\u0001\\u0002", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:33.461000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"\\u0000\\u0000\\u0000\\xA4\\xFFSMBr\\u0000\\u0000\\u0000\\u0000\\b\\u0001@\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000@\\u0006\\u0000\\u0000\\u0001\\u0000\\u0000\\x81\\u0000\\u0002PC NETWORK PROGRAM 1.0\\u0000\\u0002MICROSOFT NETWORKS 1.03\\u0000\\u0002MICROSOFT NETWORKS 3.0\\u0000\\u0002LANMAN1.0\\u0000\\u0002LM1.2X002\\u0000\\u0002Samba\\u0000\\u0002NT LANMAN 1.0\\u0000\\u0002NT LM 0.12\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:50:53.819000-0500", :message=>"Received an event that has a different character encoding than you configured.", :text=>"0\\f\\u0002\\u0001\\u0001`\\a\\u0002\\u0001\\u0002\\u0004\\u0000\\x80\\u0000", :expected_charset=>"UTF-8", :level=>:warn}
{:timestamp=>"2015-01-05T15:51:11.345000-0500", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.javain `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}
-
- -fno-stack-protector
- Posts: 4366
- Joined: Mon Nov 19, 2012 12:10 pm
Re: Hosts dropped to 1
Looks like you are definitely getting some incorrect character encoding warnings, but that should not cause the crash. This guy, seems to be the culprit:
This is for inputs, so logstash should not be connecting to other hosts, but opening up listening sockets. Lets try checking the existing open ports, shutting down logstash, checking them again, and finally starting it back up:{:timestamp=>"2015-01-05T15:51:11.345000-0500", :message=>"syslog udp listener died", :address=>"0.0.0.0:5544", :exception=>#<SocketError: recvfrom: name or service not known>, :backtrace=>["/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:119:in `udp_listener'", "org/jruby/RubyKernel.javain `loop'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:118:in `udp_listener'", "/usr/local/nagioslogserver/logstash/lib/logstash/inputs/syslog.rb:76:in `run'"], :level=>:warn}
Code: Select all
netstat -naop | grep 5544
ps -ef | grep logstash
service logstash stop
netstat -nao | grep 5544
ps -ef | grep logstash
service logstash start
netstat -naop | grep 5544
ps -ef | grep logstash
Nagios-Plugins maintainer exclusively, unless you have other C language bugs with open-source nagios projects, then I am happy to help! Please pm or use other communication to alert me to issues as I no longer track the forum.
Re: Hosts dropped to 1
(different person, Brian Richins (Security Architect))
It appears the issue is related more to the logstash itself. Running an NMap on the server we always see the 3515/TCP shutdown. This is the port that all the nxlog clients are sending on, correct?
Scanning atclogserver (172.16.16.128) [1000 ports]
Discovered open port 80/tcp on 172.16.16.128
Discovered open port 22/tcp on 172.16.16.128
Discovered open port 5544/tcp on 172.16.16.128
Restarting the Logstash services (or the server for that matter) does not appear to help.
It appears the issue is related more to the logstash itself. Running an NMap on the server we always see the 3515/TCP shutdown. This is the port that all the nxlog clients are sending on, correct?
Scanning atclogserver (172.16.16.128) [1000 ports]
Discovered open port 80/tcp on 172.16.16.128
Discovered open port 22/tcp on 172.16.16.128
Discovered open port 5544/tcp on 172.16.16.128
Restarting the Logstash services (or the server for that matter) does not appear to help.